The wizard of HOP - How we built the web based HOP CLI Settings Editor using Babashka and Scittle

‍

In December we announced HOP, the open platform for Clojure(Script) developers. One of the most interesting features of the platform is that it offers a Command Line Interface Tool for bootstrapping new projects. For that the user has to first provide the configuration to create, configure and provision the platform: starting from things as simple as the project name, to more complex parameters like how a specific service (Grafana or Keycloak, if selected) will be deployed (using Docker or as external service).

‍

In the first version of the tool the configuration had to be provided as a Clojure data structure in a Extensible Data Notation (edn) file. Although the file is quite simple to understand and edit if you have been working on the HOP internals, it can be intimidating for newcomers. So we decided that we would build a graphical User Interface that would simplify the process. We had that idea since the early days of the project, but we thought it would require a lot of work and we decided to postpone it to a second version of HOP. We first wanted to make HOP work reliably, and then focus on improving the developer experience. 

‍

After the Christmas break we started working on it, and, to our surprise and delight, building the graphical wizard has been easier than expected. There were two key factors: choosing the right Clojure data structures, and choosing the right Clojure tools.

‍

‍

Choosing the right data structure

‍

Sometimes we, developers, tend to choose the technologies or tools we want to use before modeling the problem we want to solve. We first choose the tools we think might be good for the job, and then we design the data structures that make using that specific library or technology easier to use. That pragmatic approach can be valid in some cases, but in general at Magnet we think that it should be the other way around. A good data structure can work with any technology (using some boundary transformations if needed), and it should be the base of any program we build. For example, we are now building a web based wizard, but we might want to build a terminal based one in the future. Or change technologies for some reason. The same data should work for all use cases.

‍

That said, the first step was to design a solid data structure for the bootstrapping settings.

‍

The main idea behind the settings structure was to make it completely self-contained. Inspired by the Emacs customization sub-system we decided that each setting parameter should not only contain an identifier and the configured value itself, but also some extra fields that help on its configuration:

• Tag or human readable text field that shortly explains what each settings parameter is about.
• Docstring in Hiccup format for a longer explanation about the settings parameter. 
• Type to know whether it’s text, number, a password, etc. It’s used for building the UI but also for validating that the configured value has at least the correct shape. Validating some settings requires domain knowledge, but we can check that at least looks good. 
• List of choices if the user can select one or more values from between a set of preconfigured choices.

‍

We also wanted the settings structure to guide the UI through the conditional parameters that should be shown to the user depending on their choices. For example, we should only show AWS specific settings parameters if the user selects AWS as the Cloud Provider. We achieved that by using a tree-like nested structure by implementing group type configuration parameters. Instead of having simple type parameters (e.g., text,number) we included three group type parameters (plain-group, single-choice-group and multiple-choice-group). The value of those parameters is a list of more parameters. Following the previous example, AWS would be a simple-group that includes all the AWS-specific parameters. At the same type AWS would be a choice in a single choice group parameter for choosing the Cloud Provider.

And this is the Clojure data structure we came up with:

‍

All the metadata that we mentioned above makes the file very verbose, and the final file has currently around 2k lines. That makes the graphical User Interface even more important.

‍

‍

Choosing the right tools

‍

When we decided to build a Command Line Interface tool for HOP, using Babashka was our first choice for a simple reason: startup time. Nobody wants to wait several seconds between runs when using a CLI tool. The project is mature, stable and there are many libraries available, so we were almost sure it would be more than enough for everything that we needed. Our only concern was building the graphical User Interface for the Settings Editor. We were not sure if it was even possible to do it using Babashka, but we decided to go ahead and trust that we would manage to make it work somehow in the future.

‍

When time for building the UI arrived we started exploring options for providing a UI from our Babahska CLI.  These are some of the first options we considered:

• Develop a full backend+frontend Clojure project, build an uberjar and run it from within Babashka calling it as a shell process. 
• Develop a frontend Clojurescript project, compile static HTML and JS files and serve it from Babashka using http-kit server.

‍

Both options would allow us to stick to our current expertise area (building SPAs), but they had the big disadvantage of having to maintain two independent Clojure projects (the CLI and the Settings Editor application). Additionally, the first approach required for the user to have Java installed (which we have avoided until now using Babashka). So we were not completely happy about any of them.

‍

But then we found Scittle by the author of Babashka, and we realized it was the perfect solution. Scittle exposes the Small Clojure Interpreter to the browser, and it allows executing Clojure code with no compilation needed. By including the Scittle JS dependency the browser is able to interpret the ClojureScript at runtime. So in short we can just write some static HTML, CSS and CLJS files and serve them using http-kit server from Babashka. No more machinery is needed.

‍

As there is no real Clojurescript compiler involved we can’t just include any dependency on the wild, but Scittle already includes all the basic libraries that we needed. At Magnet we use Reagent and ReFrame for building our SPAs, and both are already available in Scittle. The other two available libraries are Cljs Ajax and Cljs Pprint. We also used both.

‍

Having those ingredients building the editor itself was business as usual. We just had to write some CLJS code to read the settings editor and draw some fancy UI based on its contents.

‍

‍

Result

The usage from the final user point of view is the following:

‍

First the user has to start the web server used to serve the static resources running the `bootstrap open-setings-editor`command.

‍

‍

Then the user has to open the URL prompted by the CLI. At this point the editor is ready for use. The user can use the form to edit the different fields and download the final settings file when it’s done.

‍

‍

‍