Who uses Node.js: Trello (Part 2)
This is a continuation of the translation of The Trello Tech Stack.
Part 1
Part 2
Trello server part is based on Node.js. We knew that we would need instant distribution of updates, which means having to keep open many connections [with the server], so event-driven non-blocking seemed like a good choice. Node.js also proved to be an amazing tool for building a prototype single page application. The prototype of the Trello server was actually a simple function library that processed model arrays located in the memory of a single Node.js process, and the client application simply called these functions using a small WebSocket wrapper. For us, this was the fastest way to start trying to use Trello and make sure that the development was moving in the right direction. We used a prototype version to manage the development of Trello and the other internal projects of Fog Creek.
By the time the prototype was completed, we had enough experience with Node.js and we were shocked by its capabilities and performance. So we continued to work and made from our Trello-Pinocchio a real boy, we gave him:
Node.js is great and gets better every time the active community releases new and useful libraries. The huge amount [and nesting] callbacks that you are forced to use at first seems like a problem, but you get used to it in a couple of weeks. We use the excellent async library (as well as the short code CoffeeScript provides) to maintain control over our code. There are other elegant solutions, but we have enough use of async, whose behavior we fully understand.
')
We use HAProxy for load balancing between our web servers. It distributes TCP connections between the carousel (round robin) servers, and the rest is left by Node.js. It keeps connections open long enough to enable WebSockets and the ability to reuse connections for AJAX requests.
Trello uses Redis for non-short-lived data, which is used during exchanges between server processes, but is not saved to disk. Objects such as session activity level or OpenID temporary key are stored in Redis, and the application is built in such a way as to recover normally after partial (or complete) loss of this data. We run [Redis] with the specified 'allkeys-lru' key and working space (working set) at least five times larger than required. Thus, Redis automatically deletes data that has not been used for a long time, and restores it if necessary.
One of our most interesting uses of Redis is the alternative use of short requests when sending model changes back to browsers. When an object changes on the server, we send a JSON message to all relevant WebSockets to notify clients and store this message in a fixed-length list for the model involved, noting how many messages have been added to this list for all time. Then, when a client using AJAX requests [and not a WebSocket connection] pings the server for updates, since its last request, we can get the entire server response before checking permissions and in most cases checking for a single Redis value. Redis is so insane that he can process thousands of such checks per second without a significant sinking [performance] on a single CPU.
Also, Redis serves as our publish / subscribe server, and we use it to distribute object change messages from the server process that made the initial request to all other server processes. As soon as you have a configured Redis server, you start using it for all sorts of things.
MongoDB satisfies most of our database queries. We wanted the [server] Trello to be very fast. One of the coolest and most concerned with the performance teams we know is the neighboring partner company StackExchange. One day, talking over lunch with their lead developer David (David), I learned that although they use a SQL server to store data, in fact they store a lot of data in a de-rated form to improve performance and normalize it only when of necessity.
Trello today.
In MongoDB, we refused to use the capabilities of relational databases (that is, arbitrary JOINs) in favor of very fast writing, in most cases faster reading, and better support of de-rated data — we can store the properties of the card [Trello element] in one document in the database and still be able to query (and index) the document's nested fields. As we quickly grew, having a database with performance capable of withstanding a fair amount of irregularities in terms of read / write was a very good thing. Also, MongoDB is very easy to replicate, back up and restore (despite the Foursquare fiasco ).
Another benefit from using the storage of incoherent documents is simply using different versions of the Trello code with the same database, without dancing with a tambourine around migrations of database schemas. This has many advantages when releasing a new version of Trello - it is very rare (if there is any) the need to close access to the application while we update or fill out the database.
For development, this is also just super - when you use hg bisect (or git bisect) to search for a bug in the source code and a test relational database, it requires additional steps to roll back or return the database to the test version (or create a new database with the required fields) . This can seriously slow down the work.
We like our technology suite. As Joel noted, we were bleeding all the way through the development, but I have never seen a team making an interesting app without bloodshed related to tools and components. And not everyone can say that he really likes what they came to the finish line with. As is true for most applications, there are no components or implementation details required by nature. However, we believe that this set of excellent open-source projects has accelerated our development, has given us a solid and supported code base, with which we have intensively moved forward and made Trello a more responsive and beautiful application. Thanks to everyone who participated in these projects, this is a great time to be a programmer.
Sounds good? Try Trello! It's free.
Still not enough talk? Here is a presentation of Trello, which I did for recent discussions.
Part 1
- CoffeeScript
- Customer
* Backbone.js
* HTML5 History API
* Mustache - Pushing and polling
* Socket.io and WebSockets
* AJAX requests
Part 2
- Server
* node.js
* HAProxy
* Redis
* MongoDB - So do we like it?
Server
Node.js
Trello server part is based on Node.js. We knew that we would need instant distribution of updates, which means having to keep open many connections [with the server], so event-driven non-blocking seemed like a good choice. Node.js also proved to be an amazing tool for building a prototype single page application. The prototype of the Trello server was actually a simple function library that processed model arrays located in the memory of a single Node.js process, and the client application simply called these functions using a small WebSocket wrapper. For us, this was the fastest way to start trying to use Trello and make sure that the development was moving in the right direction. We used a prototype version to manage the development of Trello and the other internal projects of Fog Creek.
By the time the prototype was completed, we had enough experience with Node.js and we were shocked by its capabilities and performance. So we continued to work and made from our Trello-Pinocchio a real boy, we gave him:
- real database and schema ( node-mongodb-native and Mongoose )
- core web technologies such as routes (routes) and cookies ( Express and Connect )
- many processes on the server with no downtime when restarting [processes] ( Cluster )
- interprocess communication (publish / subscribe) and exchange (sharing) structural data via Redis ( node_redis )
Node.js is great and gets better every time the active community releases new and useful libraries. The huge amount [and nesting] callbacks that you are forced to use at first seems like a problem, but you get used to it in a couple of weeks. We use the excellent async library (as well as the short code CoffeeScript provides) to maintain control over our code. There are other elegant solutions, but we have enough use of async, whose behavior we fully understand.
')
Haproxy
We use HAProxy for load balancing between our web servers. It distributes TCP connections between the carousel (round robin) servers, and the rest is left by Node.js. It keeps connections open long enough to enable WebSockets and the ability to reuse connections for AJAX requests.
Redis
Trello uses Redis for non-short-lived data, which is used during exchanges between server processes, but is not saved to disk. Objects such as session activity level or OpenID temporary key are stored in Redis, and the application is built in such a way as to recover normally after partial (or complete) loss of this data. We run [Redis] with the specified 'allkeys-lru' key and working space (working set) at least five times larger than required. Thus, Redis automatically deletes data that has not been used for a long time, and restores it if necessary.
One of our most interesting uses of Redis is the alternative use of short requests when sending model changes back to browsers. When an object changes on the server, we send a JSON message to all relevant WebSockets to notify clients and store this message in a fixed-length list for the model involved, noting how many messages have been added to this list for all time. Then, when a client using AJAX requests [and not a WebSocket connection] pings the server for updates, since its last request, we can get the entire server response before checking permissions and in most cases checking for a single Redis value. Redis is so insane that he can process thousands of such checks per second without a significant sinking [performance] on a single CPU.
Also, Redis serves as our publish / subscribe server, and we use it to distribute object change messages from the server process that made the initial request to all other server processes. As soon as you have a configured Redis server, you start using it for all sorts of things.
MongoDB
MongoDB satisfies most of our database queries. We wanted the [server] Trello to be very fast. One of the coolest and most concerned with the performance teams we know is the neighboring partner company StackExchange. One day, talking over lunch with their lead developer David (David), I learned that although they use a SQL server to store data, in fact they store a lot of data in a de-rated form to improve performance and normalize it only when of necessity.
Trello today.
In MongoDB, we refused to use the capabilities of relational databases (that is, arbitrary JOINs) in favor of very fast writing, in most cases faster reading, and better support of de-rated data — we can store the properties of the card [Trello element] in one document in the database and still be able to query (and index) the document's nested fields. As we quickly grew, having a database with performance capable of withstanding a fair amount of irregularities in terms of read / write was a very good thing. Also, MongoDB is very easy to replicate, back up and restore (despite the Foursquare fiasco ).
Another benefit from using the storage of incoherent documents is simply using different versions of the Trello code with the same database, without dancing with a tambourine around migrations of database schemas. This has many advantages when releasing a new version of Trello - it is very rare (if there is any) the need to close access to the application while we update or fill out the database.
For development, this is also just super - when you use hg bisect (or git bisect) to search for a bug in the source code and a test relational database, it requires additional steps to roll back or return the database to the test version (or create a new database with the required fields) . This can seriously slow down the work.
So do we like it?
We like our technology suite. As Joel noted, we were bleeding all the way through the development, but I have never seen a team making an interesting app without bloodshed related to tools and components. And not everyone can say that he really likes what they came to the finish line with. As is true for most applications, there are no components or implementation details required by nature. However, we believe that this set of excellent open-source projects has accelerated our development, has given us a solid and supported code base, with which we have intensively moved forward and made Trello a more responsive and beautiful application. Thanks to everyone who participated in these projects, this is a great time to be a programmer.
Sounds good? Try Trello! It's free.
Still not enough talk? Here is a presentation of Trello, which I did for recent discussions.
Source: https://habr.com/ru/post/160447/
All Articles