Avoid using subscriptions only as app-state getters

Introduction.

Subscriptions in re-frame or re-om are query functions that extract data from the app-state and provide it to view functions in the right format. When we use subscriptions well, they provide a lot of value^[1], because they avoid having to keep derived state the app-state and they dumb down the views, that end up being simple “data in, screen out” functions.

However, things are not that easy. When you start working with subscriptions, it might happen that you end up using them as mere getters of app-state. This is a missed opportunity because using subscriptions in this way, we won’t take advantage of all the value they can provide, and we run the risk of leaking pieces of untested logic into our views.

An example.

We’ll illustrate this problem with a small real example written with re-om subscriptions (in an app using re-frame subscriptions the problem would look similar). Have a look at this piece of view code in which some details have been elided for brevity sake:

this code is using subscriptions written in the horizon.domain.reports.dialogs.edit namespace.

The misuse of subscriptions we’d like to show appears on the following piece of the view:

Notice how the only thing that we need to render this piece of view is next-generation. To compute its value the code is using several subscriptions to get some values from the app-state and binding them to local vars (delay-unit start-at, delay-number and date-modes). Those values are then fed to a couple of private functions also defined in the view (get-next-generation and delay->interval) to obtain the value of next-generation.

This is a bad use of subscriptions. Remember subscriptions are query functions on app-state that, used well, help to make views as dumb (with no logic) as possible. If you push as much logic as possible into subscriptions, you might achieve views that are so dumb you nearly don’t need to test, and decide to limit your unit tests to do only subcutaneous testing of your SPA.

Refactoring: placing the logic in the right place.

We can refactor the code shown above to remove all the leaked logic from the view by writing only one subscription called next-generation which will produce the only information that the view needs. As a result both get-next-generation and delay->interval functions will get pushed into the logic behind the new next-generation subscription and dissappear from the view.

This is the resulting view code after this refactoring:

and this is the resulting pure logic of the new subscription. Notice that, since get-next-generation function wasn’t pure, we had to change it a bit to make it pure:

After this refactoring the view is much dumber. The previously leaked (an untested) logic in the view (the get-next-generation and delay->interval functions) has been removed from it. Now that logic can be easyly tested through the new next-generation subscription. This design is also much better than the previous one because it hides how we obtain the data that the view needs: now both the view and the tests ignore, and so are not coupled, to how the data the view needs is obtained. We might refactor both the app-state and the logic now in get-next-generation and delay->interval functions without affecting the view. This is another example of how what is more stable than how.

Summary

The idea to remember is that subscriptions by themselves don’t make code more testable and maintainable. It’s the way we use subscriptions that produces better code. For that the logic must be in the right place which is not inside the view but behind the subscriptions that provide the data that the view needs. If we keep writing “getter” subscriptions” and placing logic in views, we won’t gain all the advantages the subscriptions concept provides and we’ll write poorly designed views coupled to leaked bunches of (very likely untested) logic.

Acknowledgements.

Many thanks to André Stylianos Ramos and Fran Reyes for giving us great feedback to improve this post and for all the interesting conversations.

Footnotes:

[1] Subscriptions also make it easier to share code between different views and, in the case of re-frame (and soon re-om as well), they are optimized to minimize unnecessary re-renderings of the views and de-duplicate computations.

Giving new life to existing Om legacy SPAs with re-om

Introduction.

We’re pleased to announce that our client GreenPowerMonitor has allowed us to open-source re-om, an event-driven functional framework which is giving new life to an existing legacy SPA that uses Om.

Why re-om?

1. The problem with our SPA: imperative programming everywhere.

We are working on a legacy ClojureScript SPA, called horizon, that uses Om.

This SPA might have had some kind of architecture in some moment in the past but technical debt, lack of documentation and design flaws had blurred it. Business logic (in this case, pure logic that decides how to interact with the user or data transformations) and effectful code were not clearly separated.

This lack of separation of concerns was making the SPA hard to evolve because its code was difficult to understand and to test. This resulted in a very low test coverage that amplified even more the problems to evolve the code safely and at a sustainable pace. This was generating a vicious circle.

Even more, conflating pure and effectful code destroys the advantages of functional programming. It doesn’t matter that we’re using a language like Clojure, without clear isolation between pure and effectful code, you’ll end up doing imperative programming.

2. A possible solution: effects and coeffects.

Using effects and coeffects is a way of getting the separation of concerns we were lacking. They help achieve a clear isolation of effectful code and business logic that makes the interesting logic pure and, as such, really easy to test and reason about. With them we can really enjoy the advantages of functional programming.

Any piece of logic using a design based on effects and coeffects is comprised of three parts:

1. Extracting all the needed data from “the world” (using coeffects for getting application state, getting component state, getting DOM state, etc).
2. Using pure functions to compute the description of the side effects to be performed (returning effects for updating application state, sending messages, etc) given what was extracted from “the world” in the previous step (the coeffects).
3. Performing the side effects described by the effects returned by the pure functions executed in the previous step.

At the beginning, when re-om wasn’t yet accepted by everyone in the team, we used coeffects and effects which were being manually created to improve some parts of the SPA, (have a look at Improving legacy Om code (I) and Improving legacy Om code (II)), but this can get cumbersome quickly.

3. An event-driven framework with effects and coeffects: re-frame.

Some of us had worked with effects and coeffects before while developing SPAs with re-frame and had experienced how good it is. After working with re-frame, when you come to horizon, you realize how a good architecture can make a dramatic difference in clarity, testability, understandability and easiness of change.

Having a framework like re-frame removes most of the boilerplate of working with effects and coeffects, creating clear boundaries and constraints that separate pure code from effectful code and gives you a very clear flow to add new functionality that’s very easy to test and protect from regressions. In that sense re-frame’s architecture can be considered what Jeff Atwood defined as a pit of success because it is:

“a design that makes it easy to do the right things and annoying (but not impossible) to do the wrong things.”

4. Why not use re-frame then?

In principle using re-frame in our SPA would have been wonderful, but in practice this was never really an option.

A very important premise for us was that a rewrite was out of the question because we would have been blocked from producing any new feature for too long. We needed to continue developing new features. So we decided we would follow the strangler application pattern, an approach which would allow us to progressively evolve our SPA to use an architecture like re-frame’s one, while being able to keep adding new features all the time. The idea is that all new code would use the new architecture, if it were pragmatically possible, and that we would only change bit by bit those legacy parts that needed to change. This means that during a, probably long, period of time inside the SPA, the new architecture would have to coexist with the old imperative way of coding.

Even though, following the strangler application pattern was not incompatible with introducing re-frame, there were more things to consider. Let’s examine more closely what starting to use re-frame would have meant to us:

4. 1. From Om to reagent.

re-frame uses reagent as its interface to React. Although I personally consider reagent to be much nicer than Om because it feels more ‘Clo­jur­ish’, as it is less verbose and hides React’s complexity better that Om (Om it’s a thinner abstraction over React that reagent), the amount of view code and components developed using Om during the nearly two years of life of the project made changing to reagent too huge of a change. GreenPowerMonitor had done a heavy investment on Om in our SPA for this change to be advisable.

If we had chosen to start using re-frame, we would have faced a huge amount of work. Even following a strangler application pattern, it would have taken quite some time to abandon Om, and in the meantime Om and reagent would have had to coexist in our code base. This coexistence would have been problematic because we’d have had to either rewrite some components or add complicated wrapping to reuse Om components from reagent ones. It would have also forced our developers to learn and develop with both technologies.

Those reasons made us abandon the idea of using re-frame, and chose a less risky and progressive way to get our real goal, which was having the advantages of re-frame’s architecture in our code.

5. re-om is born.

André and I decided to do a spike to write an event-driven framework using effects and coeffects. After having a look at re-frame’s code it turned out it wouldn’t be too big of an undertaking. Once we had it done, we called it re-om as a joke.

At the beginning we had only events with effects and coeffects and started to try it in our code. From the very beginning we saw great improvements in testability and understandability of the code. This original code that was independent of any view technology was improved during several months of use. Most of this code ended being part of reffectory.

Later our colleague Joel Sánchez added subscriptions to re-om. This radically changed the way we approach the development of components. They started to become dumb view code with nearly no logic inside, which started to make cumbersome component integration testing nearly unnecessary. Another surprising effect of using re-om was that we were also able to have less and less state inside controls which made things like validations or transformation of the state of controls comprised of other controls much easier.

A really important characteristic of re-om is that it’s not invasive. Since it was thought from the very beginning to retrofit a legacy SPA to start using an event-driven architecture with an effects and coeffects system, it’s ideal when you want to evolve a code base gradually following a strangler application pattern. The only thing we need to do is initialize re-om passing horizon’s app-state atom. From then on, re-om subscriptions will detect any changes made by the legacy imperative code to re-render the components subscribed to them, and it’ll also be able to use effect handlers we wrote on top of it to mutate the app-state using horizon’s lenses and do other effects that “talk” to the legacy part.

This way we could start carving islands of pure functional code inside our SPA’s imperative soup, and introduced some sanity to make its development more sustainable.

re-om & reffectory

We’ve been using re-om during the last 6 months and it has really made our lives much easier. Before open-sourcing it, we decided to extract from re-om the code that was independent of any view technology. This code is now part of reffectory and it might be used as the base for creating frameworks similar to re-om for other view technologies, like for example rum, or even for pure Clojure projects.

Acknowledgements.

We’d like to thank GreenPowerMonitor for open-sourcing re-om and reffectory, all our colleagues at GreenPowerMonitor, and the ones that are now in other companies like André and Joel, for using it, giving feedback and contributing to improve it during all this time. We’d also love to thank the re-frame project, which we think is a really wonderful way of writing SPAs and on which we’ve heavily inspired re-om.

Give it a try.

Please do have a look and try re-om and reffectory. We hope they might be as useful to you as they have been for us.

Originally published in Codesai's blog.

Native/Browser SPA versions using ClojureScript and re-frame talk at SCBCN17

Last month I had the pleasure of giving a talk at the Software Craftsmanship Barcelona 2017 conference about what my colleague Francesc and I have been learning lately while working on the mobile and browser versions of a SPA using ClojureScript and re-frame.

I talked mainly about how re-frame's effects and coeffects, re-frame's subscriptions and modelling a UI as FSMs place you in a pit of success, i. e., a design that, according to Jeff Atwood, "makes it easy to do the right things and annoying (but not impossible) to do the wrong things".

It was my first talk in a big conference and I had some technical problems, so I was very nervous at the beginning. After a while I relaxed and felt better. In the end I received more questions than I expected and very nice feedback.

I'd like to thank Modesto for lending me his computer when mine crashed just before the talk and helping me relax.

This is the video of the talk, (thanks Autentia for recording all the talks and for being so kind to me when I was having technical difficulties). I hope you'll find it interesting:



and these are the slides that I've improved after the talk using Modesto's feedback (thanks!!): .

Thanks to Francesc and all the people in Clojure Developers Barcelona Group, Green Power Monitor and Codesai that contributed to improve the talk with their feedback. Also thanks to my colleague Antonio for "persuading" me to give a talk in a conference.

Finally, I'd also like to thank Software Craftsmanship Barcelona 2017 conference organizers for giving me the opportunity to talk at such a great conference and for all the effort they put to make this conference better, edition after edition.

Interesting Talk: "re-frame your ClojureScript applications"

I've just watched this interesting talk by Shaun Mahood

• re-frame your ClojureScript applications

Creating custom effects in re-frame

In previous posts, we learned how to use coeffects and effects in re-frame to have effectful event handlers that are pure functions.

The example with effects we saw, used two of re-frame's built-in effect handlers (dispatch and dispatch-later). Since the set of possible side-effects is open-ended, re-frame gives you a way to define your own effects.

In this post, we'll show how to create your own effect by defining one that writes to the local storage.

Let's have a look at an event handler using that effect that writes to the local storage:

Notice the a key-value pair having the key :write-localstore in the effects map returned by the event handler. This key-value pair is telling re-frame to do a side-effect, which is uniquely identified by that key and requires the data in the corresponding value. But, how does re-frame know how to action on the :write-localstore effect?

We have to use re-frame's reg-fx function to associate the effect key, (:write-localstore), with an effect handler, (local-store/write function):

reg-fx receives two arguments: the key that identifies the effect and the function which actions the side-effect, the effect handler.

When an event handler returns an effects map which contains a given effect key, the effect handler associated with it using reg-fx will be called to action on the effect, passing it the value associated to the effect key in the effects map.

The local-store/write function is passed a map containing the key-value pairs it has to write to the local store.

Finally, this is how we'd test the event handler using it:

being check-writing-to-local-storage-contains and check-writing-to-local-storage:

And that's all we need to know in order to create a custom effect in re-frame.

Using effects in re-frame

In re-frame, we'd like to use pure event handlers because they provide some important advantages, (mentioned in a previous post about coeffects in re-frame): local reasoning, easier testing, and events replay-ability.

However, as we said, to build a program that does anything useful, it's inevitable to have some side-effects and/or side-causes. So, there will be many cases in which event handlers won't be pure functions.

We also saw how using coeffects in re-frame allows to have pure event handlers in the presence of side-causes.

In this post, we'll focus on side-effects:

If a function modifies some state or has an observable interaction with calling functions or the outside world, it no longer behaves as a mathematical (pure) function, and then it is said that it does side-effects.

Let's see some examples of event handlers that do side-effects (from a code animating the evolution of a cellular automaton):

These event handlers, registered using reg-event-db, are impure because they're doing a side-effect when they dispatch the :evolve event. With this dispatch, they are performing an action at a distance, which is in a way a hidden output of the function.

These impure event handlers are hard to test. In order to test them, we'll have to somehow spy the calls to the function that is doing the side-effect (the dispatch). Like in the case of side-causes from our previous post, there are many ways to do this in ClojureScript, (see Isolating external dependencies in Clojure), only that, in this case, the code required to test the impure handler will be a bit more complex, because we need to keep track of every call made to the side-effecting function.

In this example, we chose to make explicit the dependency that the event handler has on the side-effecting function, and inject it into the event handler which becomes a higher order function. Actually, we injected a wrapper of the side-effecting function in order to create an easier interface.

Notice how the event handlers, evolve and start-stop-evolution, now receive as its first parameter the function that does the side-effect, which are dispatch-later-fn and dispatch, respectively.

When the event handlers are registered with the events they handle, we partially apply them, in order to pass them their corresponding side-effecting functions, dispatch-later for evolve and dispatch for start-stop-evolution:

These are the wrapping functions:

Now when we need to test the event handlers, we use partial application again to inject the function that does the side-effect, except that, in this case, the injected functions are test doubles, concretely spies which record the parameters used each time they are called:

This is very similar to what we had to do to test event handlers with side-causes in re-frame before having effectful event handlers (see previous post). However, the code for spies is a bit more complex than the one for stubs.

Using test doubles makes the event handler testable again, but it's still impure, so we have not only introduced more complexity to test it, but also, we have lost the two other advantages cited before: local reasoning and events replay-ability.

Since re-frame's 0.8.0 (2016.08.19) release, this problem has been solved by introducing the concept of effects and coeffects.

Whereas, in our previous post, we saw how coeffects can be used to track what your program requires from the world (side-causes), in this post, we'll focus on how effects can represent what your program does to the world (side-effects). Using effects, we'll be able to write effectful event handlers that keep being pure functions.

Let's see how the previous event handlers look when we use effects:

Notice how the event handlers are not side-effecting anymore. Instead, each of the event handlers returns a map of effects which contains several key-value pairs. Each of these key-value pairs declaratively describes an effect using data. re-frame will use that description to actually do the described effects. The resulting event handlers are pure functions which return descriptions of the side-effects required.

In this particular case, when the automaton is evolving, the evolve event handler is returning a map of effects which contains two effects represented as key/value pairs. The one with the :db key describes the effect of resetting the application state to a new value. The other one, with the :dispatch-later key describes the effect of dispatching the :evolve event after waiting 100 microseconds. On the other hand, when the automaton is not evolving, the returned effect describes that the application state will be reset to its current value.

Something similar happens with the start-stop-evolution event handler. It returns a map of effects also containing two effects. The one with the :db key describes the effect of resetting the application state to a new value, whereas the one with the :dispatch key describes the effect of immediately dispatching the :evolve event.

The effectful event handlers are pure functions that accept two arguments, being the first one a map of coeffects, and return, after doing some computation, an effects map which is a description of all the side-effects that need to be done by re-frame.

As we saw in the previous post about coeffectts, re-frame's effectful event handlers are registered using the reg-event-fx function:

These are their tests:

Notice how by using effects we don't need to use tests doubles anymore in order to test the event handlers. These event handlers are pure functions, so, besides easier testing, we get back the advantages of local reasoning and events replay-ability.

:dispatch and :dispatch-later are builtin re-frame effect handlers already defined. It's possible to create your own effect handlers. We'll explain how and show an example in a future post.

Using coeffects in re-frame

Event handlers, collectively, provide the control logic in a re-frame application. Since the mutation of the application state is taken care of by re-frame, these event handlers can be pure functions that given the current value of the application state as first argument and the event (with its payload) as second argument, provide a new value for the application state.

This is important because the fact that event handlers are pure functions brings great advantages:

• Local reasoning, which decreases the cognitive load required to understand the code.
• Easier testing, because pure functions are much easier to test.
• Events replay-ability, you can imagine a re-frame application as a reduce (left fold) that proceeds step by step. Following this mental model, at any point in time, the value of the application state would be the result of performing a reduce over the entire collection of events dispatched in the application up until that time, being the combining function for this reduce the set of registered event handlers.

However, to build a program that does anything useful, it's inevitable to have some side-effects and/or side-causes. So, there would be cases in which event handlers won't be pure functions.

In this post, we'll focus on side-causes.

"Side-causes are data that a function, when called, needs but aren't in its argument list. They are hidden or implicit inputs."

Let's see an example:

The event handler for the :word-ready-to-check event, that we are registering using reg-event-db, is not pure because it's using the value returned by JavaScript's Date.now function instead of getting it through its arguments. To make matters worse, in this particular case, this side-cause also makes the event handler untestable because JavaScript's Date.now function returns a different value each time it's called.

To be able to test this event handler we'll have to somehow stub the function that produces the side-cause. In ClojureScript, there are many ways to do this (using with-redefs, with-bindings, a test doubles library like CircleCI's bond, injecting and stubbing the dependency, etc.).

In this example, we chose to make the dependency that the handler has on a function that returns the current timestamp explicit and inject it into the event handler which becomes a higher order function.

In the previous code example, notice that the event handler now receives as its first parameter a function that returns the current timestamp, time-fn.

And this would be how before registering that event handler with reg-event-db, we perform a partial application to inject JavaScript's Date.now function into it:

Using the same technique, we can now inject a stub of the time-fn function into the event handler in order to test it:

This is the code of the home-made factory to create stubs for time-fn that we used in the previous test:

We've seen how using a test double makes the event handler testable again, but at the price of introducing more complexity.

The bottom line problem is that the event handler is not a pure function. This makes us lose not only the easiness of testing, as we've seen, but also the rest of advantages cited before: local reasoning and events replay-ability. It would be great to have a way to keep event handlers pure, in the presence of side-effects and/or side-causes.

Since re-frame's 0.8.0 (2016.08.19) release, this problem has been solved by introducing the concept of effects and coeffects. Effects represent what your program does to the world (side-effects) while coeffects track what your program requires from the world (side-causes). Now we can write effectful event handlers that keep being pure functions.

Let's see how to use coeffects in the previous event handler example. As we said, coeffects track side-causes, (see for a more formal definition Coeffects The next big programming challenge).

At the beginning, we had this impure event handler:

then we wrote a testable but still impure version:

Thanks to coeffects, we can eliminate the side-cause, passing the timestamp input through the argument list of the event handler. The resulting event handler is a pure function:

This event handler receives, as its first parameter, a map of coeffects, cofx, which contains two coeffects, represented as key/value pairs. One of them is the current timestamp which is associated to the :timestamp key, and the other one is the application state associated to the :db key. The second argument is, as in previous examples, the event with its payload.

The map of coeffects, cofx, is the complete set of inputs required by the event handler to perform its computation. Notice how the application state is just another coeffect.

This is the same event handler but using destructuring (which is how I usually write them):

How does this work? How does the coeffects map get passed to the event handler?

We need to do two things previously:

First, we register the event handler using re-frame's reg-event-fx instead of reg-event-db.

When you use reg-event-db to associate an event id with the function that handles it, its event handler, that event handler gets as its first argument, the application state, db.

While event handlers registered via reg-event-fx also get two arguments, the first argument is a map of coeffects, cofx, instead of the application state. The application state is still passed in the cofx map as a coeffect associated to the :db key, it's just another coeffect. This is how the previous pure event handler gets registered:

Notice the second parameter passed to reg-event-fx. This is an optional parameter which is a vector of interceptors. Interceptors are functions that wrap event handlers implementing middleware by assembling functions, as data, in a collection. They "can look after cross-cutting concerns helping to "factor out commonality, hide complexity and introduce further steps into the 'Derived Data, Flowing' story promoted by re-frame".

In this example, we are passing an interceptor created using re-frame's inject-cofx function which returns an interceptor that will load a key/value pair (coeffect id/coeffect value) into the coeffects map just before the event handler is executed.

Second, we factor out the coeffect handler, and then register it using re-frame's reg-cofx. This function associates a coeffect id with the function that injects the corresponding key/value pair into the coeffects map. This function is known as the coeffect handler. For this example, we have:

Since the event handler is now a pure function, it becomes very easy to test it:

Notice how we don't need to use tests doubles anymore in order to test it. Thanks to the use of coeffects, the event handler is a pure function, and we can just pass any timestamp to it in its arguments.

More importantly, we've also regained the advantages of local reasoning and events replay-ability that comes with having pure event handlers.

In future posts, we'll see how we can do something similar using effects to keep events handlers pure in the presence of side-effects.

Interesting Talk: "Native mobile apps with ClojureScript"

I've just watched this very interesting talk by Viktor Eriksson

• Native mobile apps with ClojureScript

Kata: Variation on Cellular Automata, animating automaton evolution using re-frame

In the last Clojure Developers Barcelona meetup, we started to use re-frame to animate in the browser the code to evolve cellular automata that we wrote some weeks ago.

We managed to make a rough version but we didn't have time to test it and make it nice.

When I got home I redid the exercise from scratch using a mix of TDD and REPL Driven Development.

First, I coded a couple of evolution rules. These are their tests:

and this is their code:

Next, I coded the cellular automata evolution, these are its tests:

and the resulting code:

Once I had the evolution working, I started to work on the view, using Figwheel and a hard-coded list of cellular automaton states on the db. This way I focused only on rendering the list of cellular automaton states.

With that working, I played on the REPL to create a subscriber that reacted to changes on the cellular automaton states making the view render.

This is the resulting code of the view:

the code of the subscriber:

and the default db:

Then, I used TDD to write the event handlers.

The use of effects keeps re-frame handlers pure. They allow us to avoid making side effects. We just have to describe as data the computation that will be made instead of doing it. re-frame takes care of that part.

These are the handlers tests:

these are the handlers:

Notice, how in order to dispatch to another handler (a side effect) we just have to add a key-value pair to the map of effects returned by the handler. In this case, I used the :dispatch and :dispatch-later effects which are part of re-frame's built-in effects. You can also create and register your own effects (I will talk about it in some future post).

To see the whole picture this is the code that registers the handlers:

I think that effects and coeffects (I'll talk about them in a future post) are just great!

They make testing handlers logic very easy (since handlers keep being pure functions) and avoid having to use test doubles.

Finally, everything is put together in the core namespace:

You can see the cellular automaton evolving in this video (I used garden library to add some CSS to make it look nicer):

You can find the code on this GitHub repository.

In this post I've tried to, somehow, describe the process I followed to write this code, but the previous gists just reflect the final version of the code. If you want to follow how the code evolved, have a look to all the commits on Github.

I really love using ClojureScript and re-frame with Figwheel.