Journey from Node to Crystal
Duo has used Node for many years as the main platform. However, recently they have experimented with a very new, not yet fully formed language Crystal. According to them, the more they did to them, the more they became attached to him.
Today we want to share with you a translation of their story about the strengths and weaknesses of the Node and Crystal platforms, and why in Duo more and more server projects are being transferred to Crystal.
We switched to Node a few years ago. We were then a small company with a handful of developers, and we needed our employees to be as versatile as possible. Allowing developers to specialize in the frontend or backend was an unattainable luxury for us. If we had a lot of work on the server or client parts of the applications, we needed people who were able, regardless of their specialization, to help deal with the affairs.
')
Node then seemed like an obvious choice to us. If we took a developer who knew JavaScript to the staff, it meant for us that he could work on both client and server projects. Tools, syntax and dependencies would overlap, and everyone would improve their skills, as any task would involve the use of JavaScript.
The server and client code have completely different goals; these types of code require knowledge of very different methods of work. Typically, client code is user interaction, interface updates, and data queries from the server. Our developers usually worked with webpack or browserify for packaging code, developed interfaces on React, and used CSS frameworks to simplify page layout.
On the server, the programmer deals with SQL queries to databases, with ORM, with reading and writing files, organizes interaction with third-party APIs. Data streams on the server are subject to the model "request - response". Between requests, all tasks must serve the answers, and all this needs to be done in a specific way. If a certain step relies on the results obtained in the previous step, the corresponding processes should be carried out in order; if not, they can be performed in parallel.
Node is designed so that it performs each task asynchronously. This means that if you ask Node to do 5 tasks, it will try to do it all at once. The last few years, the main means to support this model of work have been promises. In a nutshell, promises allow the programmer to chain sets of asynchronous tasks, which are sequences of steps that follow each other.
On the server, the standard use of parallel tasks may seem like a very effective idea. In reality, for most of the tasks we face, data from previous tasks are required. Even if we can perform tasks in parallel, system resources can be quite quickly depleted, that is, performing multiple parallel queries to the database can exhaust the pool of connections and reduce the number of users that can be served simultaneously.
Over the years of using Node, the creation of chains of promises has become the norm for us. Half of the written code was aimed at turning asynchronous tasks into tasks solved sequentially. These promise chains are hard to test, debug, the code is not particularly intelligible. It often happens that just looking at the code is hard to understand even the order in which tasks and subtasks are performed.
Roal Dahl, the creator of Node, quite successfully described this situation, comparing Node and Go in an interview :
But the interface that this system provides to the programmer is blocking, and I think, in fact, that this is a better programming model. Using the blocking approach allows, in many situations, to see the essence of the actions performed much better. Say, if there are a bunch of consecutive actions, it is quite useful to tell the system something like the following: “Solve task A, wait for an answer, perhaps give an error. Solve problem B, wait for an answer, or give an error. ” And in Node, because of the need to constantly “jump” between function calls, it is much more difficult to achieve this.
Anyone who regularly programs in JavaScript, sooner or later will get acquainted with the error "undefined is not an object". This error occurs when you try to access a method or property of a variable that you consider to be an object, but which contains the value null. It is not enough to control what data is transferred between asynchronous parts of the code, you also need to be aware of what is happening with the types anywhere in the application code. Each time an application receives data from one process and transfers it to another, it may fail. If you do not anticipate the possibility of processing all possible values, the server will generate an error, or, much worse, do something unexpected.
While working with Node, I explored many other languages ​​and platforms, including Python, PHP, Ruby and Go. They, as a rule, were either slower than Node, or not as convenient for development purposes. Speed ​​and syntax are two things in a language that can be optimized only up to a certain limit.
Then, last year, I read an article about the Crystal language. It is from a new generation of languages ​​that are compiled into machine code via LLVM. Its syntax is similar to Ruby (I like it), but it works as fast as Go (and this beast doesn’t take speed!).
Crystal is still very young, but I decided to redo some server parts of our content management system on it. It turned out just fine. Here is what I found out in the course of work:
I liked Crystal programming so much that we rewrote the whole backend of our CMS in this language. Its API is compatible with our Node-based CMS, as a result, websites can be transferred to a new system, or returned to the old system, with relatively little effort. This is important because Crystal is still a young project and we need insurance.
After our DuoCMS was completely rewritten to Crystal, I needed to test it in production. As a matter of fact, the original of this material is placed on the site that works on Crystal.
Below, for comparison, is a slightly simplified version of the controller code written in Crystal and Node.
Here is the controller on Node (using the Express framework).
Here is the Crystal controller (using the Kemal framework).
It is easy to see that the structure of these two examples is very similar. However, when there was no need for promises, the total amount of code decreased. When writing larger applications, this is noticeably stronger. The DuoCMS 5 server code consists of approximately 15,609 lines of JavaScript. The volume of the DuoCMS 6 code is close to 10186 lines. At the moment, DuoCMS 6 has more features, the implementation of which required 30% less code. At the same time, due to the absence of promises, this code is much easier to read and maintain.
Developers call the current release of the Crystal alpha version. Here I must say that I had to use much less developed frameworks intended for production. At worst, I would say that Crystal is now in beta. However, I can understand the caution of the developers. They talk about the alpha version, as it gives them room to maneuver, to make changes, even to break some API, and so on.
I have been using Crystal for about a year now and have only encountered a few changes, which are explained by the development of the project and the fact that this is still the alpha version. I had more problems updating the React on the front end. In addition, it is worth saying that Crystal is written in Crystal, that is, if something turns out to be non-working, you can make corrections to the language and the standard library (I did).
At the moment, the main missing capabilities of Crystal include the following:
For us, none of these shortcomings was the reason for abandoning Crystal. I needed to make a few additions that implement the missing features in the libraries we use, but this also happened when working with Node. In the end, I can say that I am very pleased with Crystal.
Should you try Crystal? Yes worth it! Stuck is really wonderful. Crystal is nice to program, just read and edit the code. And by the way, the more people will use Crystal and contribute to the development of this language - the better it will become. Want to see everything with your own eyes? Here are the installation instructions . Here is the project site . And this is Crystal chat , if anything - write me at @crisward.
If you ask if you should use Crystal in production, well, that's what you want. Personally, I think that the only way to make something suitable for practical use is to try it on those tasks that fail, and then gradually move to it in larger projects. We do not use Crystal everywhere, for example - on projects with very high traffic, or on critical areas. We monitor all of our sites and back them up regularly, in addition, Node is always on the pickup - in case something happens to Crystal.
Dear readers! Do you plan to try Crystal in your projects?
Today we want to share with you a translation of their story about the strengths and weaknesses of the Node and Crystal platforms, and why in Duo more and more server projects are being transferred to Crystal.
Node
â–Ť Waiting
We switched to Node a few years ago. We were then a small company with a handful of developers, and we needed our employees to be as versatile as possible. Allowing developers to specialize in the frontend or backend was an unattainable luxury for us. If we had a lot of work on the server or client parts of the applications, we needed people who were able, regardless of their specialization, to help deal with the affairs.
')
Node then seemed like an obvious choice to us. If we took a developer who knew JavaScript to the staff, it meant for us that he could work on both client and server projects. Tools, syntax and dependencies would overlap, and everyone would improve their skills, as any task would involve the use of JavaScript.
â–ŤReality
The server and client code have completely different goals; these types of code require knowledge of very different methods of work. Typically, client code is user interaction, interface updates, and data queries from the server. Our developers usually worked with webpack or browserify for packaging code, developed interfaces on React, and used CSS frameworks to simplify page layout.
On the server, the programmer deals with SQL queries to databases, with ORM, with reading and writing files, organizes interaction with third-party APIs. Data streams on the server are subject to the model "request - response". Between requests, all tasks must serve the answers, and all this needs to be done in a specific way. If a certain step relies on the results obtained in the previous step, the corresponding processes should be carried out in order; if not, they can be performed in parallel.
â–ŤStandard asynchrony
Node is designed so that it performs each task asynchronously. This means that if you ask Node to do 5 tasks, it will try to do it all at once. The last few years, the main means to support this model of work have been promises. In a nutshell, promises allow the programmer to chain sets of asynchronous tasks, which are sequences of steps that follow each other.
On the server, the standard use of parallel tasks may seem like a very effective idea. In reality, for most of the tasks we face, data from previous tasks are required. Even if we can perform tasks in parallel, system resources can be quite quickly depleted, that is, performing multiple parallel queries to the database can exhaust the pool of connections and reduce the number of users that can be served simultaneously.
Over the years of using Node, the creation of chains of promises has become the norm for us. Half of the written code was aimed at turning asynchronous tasks into tasks solved sequentially. These promise chains are hard to test, debug, the code is not particularly intelligible. It often happens that just looking at the code is hard to understand even the order in which tasks and subtasks are performed.
Roal Dahl, the creator of Node, quite successfully described this situation, comparing Node and Go in an interview :
But the interface that this system provides to the programmer is blocking, and I think, in fact, that this is a better programming model. Using the blocking approach allows, in many situations, to see the essence of the actions performed much better. Say, if there are a bunch of consecutive actions, it is quite useful to tell the system something like the following: “Solve task A, wait for an answer, perhaps give an error. Solve problem B, wait for an answer, or give an error. ” And in Node, because of the need to constantly “jump” between function calls, it is much more difficult to achieve this.
â–ŤDynamic types
Anyone who regularly programs in JavaScript, sooner or later will get acquainted with the error "undefined is not an object". This error occurs when you try to access a method or property of a variable that you consider to be an object, but which contains the value null. It is not enough to control what data is transferred between asynchronous parts of the code, you also need to be aware of what is happening with the types anywhere in the application code. Each time an application receives data from one process and transfers it to another, it may fail. If you do not anticipate the possibility of processing all possible values, the server will generate an error, or, much worse, do something unexpected.
Crystal
While working with Node, I explored many other languages ​​and platforms, including Python, PHP, Ruby and Go. They, as a rule, were either slower than Node, or not as convenient for development purposes. Speed ​​and syntax are two things in a language that can be optimized only up to a certain limit.
Then, last year, I read an article about the Crystal language. It is from a new generation of languages ​​that are compiled into machine code via LLVM. Its syntax is similar to Ruby (I like it), but it works as fast as Go (and this beast doesn’t take speed!).
Crystal is still very young, but I decided to redo some server parts of our content management system on it. It turned out just fine. Here is what I found out in the course of work:
- Crystal has a high performance. For my tasks, it was 2 times faster than Node.
- It uses very little memory. Crystal usually needs less than 5 MB per process, and Node more than 200 MB.
- It has an excellent standard library, and as a result, we only needed 12 dependencies to solve a typical problem, compared to a hundred Node dependencies.
- The code, by default, looks synchronous, it uses, like Node, a cycle of events, but for the organization of parallelism, lightweight streams are used (fibers), the interaction is organized through channels like Go. This makes it easier to understand the code.
- Crystal is statically typed, so errors can be found during compilation.
- Crystal prints types, and as a result, its type system is easy to use, since you don't have to use type annotations too often.
I liked Crystal programming so much that we rewrote the whole backend of our CMS in this language. Its API is compatible with our Node-based CMS, as a result, websites can be transferred to a new system, or returned to the old system, with relatively little effort. This is important because Crystal is still a young project and we need insurance.
After our DuoCMS was completely rewritten to Crystal, I needed to test it in production. As a matter of fact, the original of this material is placed on the site that works on Crystal.
â–ŤCompare code on Node and Crystal
Below, for comparison, is a slightly simplified version of the controller code written in Crystal and Node.
Here is the controller on Node (using the Express framework).
const express = require('express') const app = express() const bodyParser = require('body-parser') const UserService = require('user-service') app.use(bodyParser.json()) app.get('/', function (req, res) { res.send('Hello World!') }) app.post('/api/users', function (req, res) { if(request.body){ UserService.save(request.body) .then(function(){ res.send('user saved') }) .catch(function(err){ res.send(err) }) }else{ res.send("no user provided") } }) app.listen(3000, function () { console.log('Example app listening on port 3000!') }) Here is the Crystal controller (using the Kemal framework).
require "kemal" require "user" require "user-service" get "/" do "Hello World!" end post "/api/users" do |ctx| if (json = ctx.request.body) user = User.from_json(json) UserService.new.save(user) "user saved" else "no user provided" end end Kemal.run It is easy to see that the structure of these two examples is very similar. However, when there was no need for promises, the total amount of code decreased. When writing larger applications, this is noticeably stronger. The DuoCMS 5 server code consists of approximately 15,609 lines of JavaScript. The volume of the DuoCMS 6 code is close to 10186 lines. At the moment, DuoCMS 6 has more features, the implementation of which required 30% less code. At the same time, due to the absence of promises, this code is much easier to read and maintain.
What is missing in Crystal?
Developers call the current release of the Crystal alpha version. Here I must say that I had to use much less developed frameworks intended for production. At worst, I would say that Crystal is now in beta. However, I can understand the caution of the developers. They talk about the alpha version, as it gives them room to maneuver, to make changes, even to break some API, and so on.
I have been using Crystal for about a year now and have only encountered a few changes, which are explained by the development of the project and the fact that this is still the alpha version. I had more problems updating the React on the front end. In addition, it is worth saying that Crystal is written in Crystal, that is, if something turns out to be non-working, you can make corrections to the language and the standard library (I did).
At the moment, the main missing capabilities of Crystal include the following:
- There is still no Windows support (it doesn't bother me, I work on a Mac, I deploy the code on Linux).
- Until now, there is no real parallelism (in the Node it does not exist either).
- There is no incremental compilation (it would be very convenient, as now, to compile our system after making changes to the code, it takes about 8 seconds).
- There are not many well-supported open source libraries for Crystal, but everything will be in order when the use of Crystal begins in serious projects.
For us, none of these shortcomings was the reason for abandoning Crystal. I needed to make a few additions that implement the missing features in the libraries we use, but this also happened when working with Node. In the end, I can say that I am very pleased with Crystal.
Results
Should you try Crystal? Yes worth it! Stuck is really wonderful. Crystal is nice to program, just read and edit the code. And by the way, the more people will use Crystal and contribute to the development of this language - the better it will become. Want to see everything with your own eyes? Here are the installation instructions . Here is the project site . And this is Crystal chat , if anything - write me at @crisward.
If you ask if you should use Crystal in production, well, that's what you want. Personally, I think that the only way to make something suitable for practical use is to try it on those tasks that fail, and then gradually move to it in larger projects. We do not use Crystal everywhere, for example - on projects with very high traffic, or on critical areas. We monitor all of our sites and back them up regularly, in addition, Node is always on the pickup - in case something happens to Crystal.
Dear readers! Do you plan to try Crystal in your projects?
Source: https://habr.com/ru/post/337992/
All Articles