Clojure Cup - the experience of participating in the hackathon on writing a web application on Clojure

This weekend was Clojure Cup - 48 hour hackathon on Clojure. It will be a relatively long post about how it went on in our team, with technical details and other things. Well, and at the end of the post, where already without this, we will tell quite a bit about our project. And those who are too lazy to read, I can immediately go to CodeNotes and even vote for us on the Clojure Cup page.

Work environment

There are two different “regimes” in participation in the hackathon: participation for the sake of pleasure and for the sake of victory. In the first case, we lazily code something cool for two days; in the second, everything is a little different, with hardcore and early training. We chose the second option.
There were four of us in the team - two, including me and si14 , are in Petersburg, one - in Tyumen ( zloy_alu ), another one - in Munich. But we didn’t experience any communication problems - more or less successful task isolation, excellent Trello service and simply divine FlowDock helped . The latter, in my opinion, is the ideal service for intra-team interaction. Judge for yourself - there is both chat and file sharing, and separate threads of discussions, and notifications about various events (deploy, push to the repository, new tasks in trello, mention on Twitter). It looks like this:


Since we were clearly planning to win, we started the preparation in advance. In particular:

• checked selected libraries and the structure of the application as a whole (for details, see below about Clony);
• checked the integration with third-party services like NewRelic;
• started a blog and Twitter;
• learned how to quickly and conveniently deploy (Makefiles and Ansible);
• prepared the JSON API documentation between the server and the web app, which allowed us to write these pieces more or less independently;
• prepared interface sketches on paper;
• bought a domain and issued an SSL certificate;
• learned how to configure nginx for normal work with websockets and static resources;
• we prepared contacts for the pool of beta testers (a certain file came out here - we were behind the schedule and sent out only at 11 Moscow time, so we managed to get only one review before the end of the hackathon).

Technical details

And now the details of what might be useful for web developers at Clojure.
')
The experience of creating a small Clony application was very useful. This is an extremely simple thing on Clojure using http-kit (http-server and client for Clojure) and Compojure (router), which communicates with its front-end with websockets. It uses Stuart Sierra's workflow , thanks to which the development becomes noticeably more pleasant: we completely restarted our entire application (or its pieces) from nrepl in a split second. During the hackathon, we made several improvements that are waiting for their backport in Clony.

The first day

We knew in advance that we would be deploying to a virtual machine, so we learned how to use Ansible . Thanks to him, we were online 4 hours after the start of the competition - with a fully developed production-configuration and a test application. This would have happened earlier if it were not for problems with DigitalOcean , which provided hosting for the participants (they banned our account due to the suspicion of fraud). We also had local virtuals ( Vagrant - super!) With a completely identical configuration, but in the end everyone scored on them and deployed immediately to production. This happened also because some things had to be changed on the fly on the server by hand — starting from the MariaDB configuration (damn these ANSI_QUOTES! We should have taken Postgres right away) and ending with the fact that we forgot to write logs to disk. Plus, we did a fairly complete integration with NewRelic . I cannot say that he critically rescued us, but the ability to easily see such a large number of parameters — server response time, DOM rendering time by clients, network delays, amount of free memory on the server, GC JVM statistics — soothed my nerves. The integration library turned out to be very small and we will definitely upload it in open-source.

On the first day, we spent a lot of time trying to start clang - integrating Angular.js with ClojureScript. The abundance of minor troubles eventually forced to abandon ClojureScript entirely and switch to bare JavaScript. ClojureScript is much better than a language, but in the conditions of a hackathon, there was simply no time to fight against its own lack of understanding of its subtleties.

Trying to do everything as correctly as possible, we decided to add a library for SQL migrations to the project. It was interesting to find that the very popular library for migrating Lobos does not work at all . As a result, we lost about an hour to search and process with a new one; stopped at ragtime .

As a result, at the end of the first day we had a beautiful main page and a semi-working list of repositories (a small description of our project is slightly lower). In the interval 22: 00–03: 00, the team dispersed to sleep (at this point no one slept from 4 am) and we gathered again around 8 am on Sunday.

Second day

On Sunday, things went better, but there were quite a lot of problems:

• I had to redo a small piece of OAuth authorization;
• at some point, we were banned by GitHub on the limit of API requests, which was extremely unpleasant, and we had to reduce the number of requests being sent;
• To generate images, we first tried to use something in pure Java, but in the end we took a binding to ImageMagick. There are two of them: one works through sh and pipes, the other through JNI. The first one slows down (30 + ms per image), it was not possible to get the second one;
• and with the slow generation of images, and with the number of requests to GitHub, we were rescued by core.cache - the wonderful “core” library of Clojure;
• we made a risky decision to take core.async to work with events on the server. As a result, this turned out to be a good solution: the first working version of the code with its use was born in an hour and a half and there were practically no problems with it, despite the fact that no one in the team had used it before. With the help of core.async, we made it possible from any place in the code to write to the user's specific web socket and complex logic to restart scanning repositories in the event of server overload: we process the queue of scan requests, discarding everything that overflows the queue queues choose from the database that thrown during reloading. The code is very compact and seems to work;
• bugs on the client, some of which, alas, we did not have time to fix until the end of the hackathon.

In general, the second day was more than productive. We tried to focus on the first impression, so we paid attention to the little things: redirect to our own domain, beautiful “How to Use”, clear text on the main, nice design, dies with “sorry, you don’t have TODO in the repository, here’s a bug report form, if this is not true". Of course, to speed up development, we used Bootstrap , but we tried to do everything as nicely and as “non-default”. How much this will help us get the best marks from the judges, we will find out on Friday.

The most interesting - what is it all about?

It would be strange to say nothing at all about our project.

It seems to us that humanity has not yet invented the perfect bugtracker. But ... Why do we need a bugtracker at all? The essence of our project is to remove a separate bugtracker from your life. And we have tried and will continue to try so that it can look like this:

def somefun ( ) :
# TODO: @ si14, please fix this before release
return true

And we will do the rest - we will send a notification to the user that you have created a task for him, display it in our interface, and also monitor your commits and scan the code for new TODO, FIXME and NOTE. In addition, you can generate a beautiful image for your repository on GitHub, indicating how much TODO is in your code and a link to information about it. Our CodeNotes project does not yet have all the desired functionality (plus there are still some rather annoying bugs, so what to hide), but when the voting is over, we will immediately continue to improve it and make it the best bug tracker for small projects.

In the meantime, by the way, you can vote for us and comment in the comments about what you think about the idea, about the hackathons, and about Clojure web programming in general.