The default languages in the team are Clojure and NodeJS, these provide both the power and speed necessary for rapid prototyping and concept development.
Clojure (clj) has a sub project that allows you to write code in Clojure and then compile it into JavaScript that can be run in any normal browser. This is called ClojureScript (cljs) and it appears to give you all the benefits of Clojure combined with the benefits from Node.
A recent pure Clojure project provided an API with an in memory (with persistent file based storage) database. While this works perfectly, is stable and runs without any issues, the time to deploy it on our Kubernetes Cloud can be in excess of 10 minutes.
As an extended spike, we decided to see how easy it would be to migrate the app to NodeJS via ClojureScript.
The Pros
- Re-use of large chunks of the existing Clojure code base without change
- Ability to create CLJC (common) files that can be used in either pure Clojure or in ClojureScript
- Asynchronous application
- Fast deployment and start times
The Cons
Problem Number 1 – Data Storage
There are few reliable, in memory with persistence JS only database libraries. There are even fewer that also compile happily with ClojureScript. Most of the databases that JS utilises in Node are based on external server capabilities – MongoDB, CouchDB.
Given the lightweight nature of the data being stored, and the ability to run it without relying on additional servers, I initially used SQLite via the SQLite3 NPM library. While this appeared to work fine, run-time errors meant that the DB was never properly built or connected. Using naive NodeJS to create, write and read back the same data did not give us errors, so it appears that the library uses something that the compilation process doesn’t handle.
Swapping to the lighter-weight DBLite NPM library allowed us to build the new db, populate it and then use it via the app.
The app followed the usual path of being Dockerised and then run in Kubernetes. Here we hit a strange situation that is probably NOT related to ClojureScript, but is worth flagging. While the SQLite DB would initialise quite happily in a Docker that ran locally, deploying the same container to the Kubernetes platform consistently gave us a DB error that the file was locked.
Eventually, I removed DBLite and tried a third database system called NeDB. While this required re-writing sections of the DB system to use a MongoDB/Key-value style of record handling, the app now deploys and runs without issue.
Problem Number 2 – Platform Bias
ClojureScript is written to compile to browser JavaScript, not NodeJS. This means that things can fail with strange error messages. Initially I tried to use the Request NPM library to make HTTP GET and POST calls to an external REST API. Although the code didn’t throw any errors during the compilation process, when it was used in the actual program it would return an error:
“XMLHTTP Not found”
It appears that as ClojureScript is browser focused, it makes use of, or at least assumes the presence of, libraries built into the browser. Using NPM dependencies that also use them in NodeJS where they don’t exist naturally causes problems. As these issues don’t prevent compilation, errors will only appear during run-time.
Reverting to built-in NodeJS functionality – in this case replacing the ‘Request NPM’ with the native ‘request’, removes this as an issue, but does mean that additional work is required to write and then use HTTP calls.
Problem Number 3 – Logging frameworks might not run
Normal NodeJS development will often make use of a third party logging framework or tool, but these appear to suffer from having their output curtailed. For example ‘Sexylog’ NPM when used in TRACE mode did not seem to produce as much output as I would expect from previous experience.
However, the built in logging and use of the normal ‘console.log’ probably limit the effect this will have on development. As with platform bias above, it pays to keep it simple and utilise the native capabilities of NodeJS & ClojureScript rather than add complexity in the form of external libraries.
Problem Number 4 – Scaffolding
Clojure requires a fair amount of scaffolding to setup, compile and run the final code. While most of this is handled by Leiningen (or Boot), the number and variety of frameworks/task runners/ helpers than can be called to setup the initial project is overwhelming.
Granted most of these are targeted at specific outcomes – use Om if you want to build a React based app, but it is not always obvious which framework will be the best to use in a given situation.
This is more of a personal peeve than a technical one – as someone who is new to Clojure I seem to spend a lot of time at the beginning trying to remember what I should do to start off a new project. Do I use figwheel or not? What was figwheel? Is it an app or a library? What’s the difference?
Now add the additional complexity on top of that to ensure that we can use ClojureScript… I’ve still not figured out how I can scaffold a Clojure backend with a ClojureScript front end.
Problem Number 5 – Clojure Libraries
Like the Node NPM libraries discussed above in Platform Bias, not all Clojure libraries will run with ClojureScript. So this often leaves us with a NodeJS library we may know and use that we cannot use via ClojureScript.
The Gotchas
Other little things that have caught me out:
1) During compilation missing dependencies (NPM or Clojure) don’t get flagged.
So a missing NPM/Clojar dependency won’t show up until everything falls over at runtime
2) EDN -> JSON data conversion will break if EDN is using numbers as keywords.
The EDN {:1 “some data”, :2 “other thing”} won’t convert to JSON as JSON doesn’t accept numbers as labels.
{“1”:”some data”, “2”:”other thing”} is NOT valid.
3) REPL is missing
One of the best things about developing Clojure is the REPL. It allows you to immediately see the effect changes on the code will have. As the code compiles to ClojureScript, it appears that this level of immediate feedback isn’t possible – although this might just be in the Atom based photo-REPL add on.
4) The ’new’ keyword and callbacks
We’ve all dealt with callback hell in JavaScript… thankfully as ClojureScript is asynchronous callbacks are not an issue. However, when we try to use some NodeJS libraries or functions callbacks are essential. This leaves us with the only option but to ‘hack’ our way around the problem by using either Clojure channels or anonymous functions.
Eg:
JS: db.loadDatabase (console.log(error))
CLJS: (.loadDatabase db (fn [err] (println err)))
Of the two options channels is by far the more efficient as it allows the output to be piped directly to the calling function without additional processing, something that is more in line with functional programming.
The Outcome
Although the outcome was a working version of the API, the extra time spent trying to find solutions to the differences between Browser JS and NodeJS engines was considerable.
The ability to migrate a significant amount of the internal processes – data structures, validation and algorithms across without change initially made the re-write very easy. It was when issues started to appear at the two ‘ends’ of the program that things began to slow down.
Getting data in and out via the web interface and getting data in and out of the storage system both showed where the NodeJS compiled ClojureScript wasn’t quite working.
Based on this spike, we have decided to continue the development of the API as a Clojure project for now… but with a view to using more CLJC (common) files that will allow us to migrate to ClojureScript at a later date if we need to. It is our belief that the development of the Clojurescript compiler is moving forward and the differences highlighted between the browser and Node JavaScript output will soon be handled.
twistednix's 