Fear and loathing in a single startup. Part 1 - Fear
It's time to describe the architecture and features of the operation of a single application. And for myself, so as not to forget, and for others - try to show how not to do it. All matches are random, all characters are made up. Only the technologies that are used and the described, hhm, architectural solutions are real. Go.
It all started innocently: a project on Ruby on Rails, a micro-services architecture, a separate balancer with a “hot” backup ... Only then I learned that: the servers are running on Gentoo; Microservices use a clever algorithm for registering themselves in the application, which causes the entire application to explode if at least one service crashes; no monitoring; no documentation; inhabited by robots. But enough of the lyrics, let's see how the project lived when I came to it.
To date, 90% of the written below is irrelevant - so it looked about two years ago. The narrative, however, is conducted in the present tense - it is more convenient for me.
As I said, there is a Rails application, and a number of services on the backend. Messaging between Rails applications and services is done via a self-written message bus. Next is Redis, which stores a slice of the current state of the entities, the application on NodeJS + socket.io (the exchange of messages between the front-end and the backend). Simplified, it looks like this:
')
The application operates with certain entities that are stored in the “Rails db” database. Services operate on the same entities as the main application, but each have their own separate database. Synchronization of entities occurs through messages in ZeroMQ by events, which generates the main application. Each service consists of the service itself, and a set of event listeners (separate processes in essence) that receive messages (each listener processes messages of the same type) and tell the service what to do (which fields in the database need to be changed). The process of the service itself also accepts a certain type of messages, by which it sends data from its database to the main application. Both services and the main application are actively working with Redis (read and write); there is no synchronization / locking (i.e. races are possible, and they exist). This is for the application architecture. Now we will briefly go through the servers and move on to the most interesting - the features of the extrapath.
So, servers are regular dedicated servers (who said “virtual”? Ordinary pieces of iron). The average configuration is 2x Core i7, 32–64 GB RAM, 2x 1T HDD / RAID1. In terms of quantity, we count. Two “balancers” (one main, the second - “hot” reserve, switching - floating IP). Three servers for services. One server for the database. One server for broker and redis. One server is a test server (all components are spinning on it at once). Total - 8 pieces. There is no LAN between the servers, all traffic walks through the DC (some servers are separated by different racks).
What is wrong with the above, you ask? Yes, everything is so, but the devil, as usual, is in the details, but in our case it is the daily operation and support. Next, just unstructured describe some points that I think problematic.
The instructions for adding a new server begin with the words “Install Gentoo”. Yes, the whole stack is spinning on Gentoo, with all the consequences. Provisioning is done in a bash script. For configuration management synctool is used (such rsync on steroids in fact). Depla application - git pull hands on all seven servers. In the code of the main application in the top ten places, the host names of the database, Redis, the broker, are nailed. The separation in production / development code is very conditional (it takes 2-3 days for a programmer to set up an environment for development). Services are so closely integrated with the main application that when one falls, the whole platform collapses. The service itself is from three to fifteen processes, each of which is simply started up by hands on the screen (yes, 15-20 screen sessions are constantly running on one of the servers, and in the documentation repository there is a file “these screens should be on this server after rebut ”).
And most importantly - the admin of the application. Born in the hidden recesses of the consciousness of the gloomy Teutonic genius, an explosive mixture of bash, Ruby and Falcon . Did you know that there is such a programming language - Falcon? Now you know (follow the link, by the way, the language is worthy of attention as an extension of the horizon). The admin panel itself is simply a collection of SQL queries, and pieces of Ruby code, which allows you to operate with different parameters of entities. There is nothing in it that would justify using anything else bash. The same falcon - for what the hell is it used there, I did not understand; in reality, everything that falcon scripts do is form something like this under certain conditions, and perform it through system ():
Everything described above was created by one person for two years. I will not evaluate his programmer skills, but the admin (or devops, as you wish) from him, frankly, is crappy - which is worth only one Gent per share, instead of a normal binary distribution. At some point, even a business realized that it wouldn’t live normally, and I was asked to see what could be done to try to take off with all this garbage. In the next part I will tell you what and how I did to diminish the amount of entropy in the Universe.
We had 2 balancers, 7 servers for the application, 5 samopisny gems, a samopisny config of the kernel and a whole variety of programming languages ​​and technologies of all varieties and colors, a database, as well as todo lists in Basecamp, NodeJS, soket.io and 2 dozen keys per entity in redis. Not that it was the necessary reserve for a startup, but if you started writing code, it becomes difficult to stop. The only thing that worried me was ZeroMQ. Nothing in the world is more helpless, irresponsible and vicious than a self-written message bus based on ZeroMQ. I knew that sooner or later we will move on to this rubbish.
It all started innocently: a project on Ruby on Rails, a micro-services architecture, a separate balancer with a “hot” backup ... Only then I learned that: the servers are running on Gentoo; Microservices use a clever algorithm for registering themselves in the application, which causes the entire application to explode if at least one service crashes; no monitoring; no documentation; inhabited by robots. But enough of the lyrics, let's see how the project lived when I came to it.
To date, 90% of the written below is irrelevant - so it looked about two years ago. The narrative, however, is conducted in the present tense - it is more convenient for me.
As I said, there is a Rails application, and a number of services on the backend. Messaging between Rails applications and services is done via a self-written message bus. Next is Redis, which stores a slice of the current state of the entities, the application on NodeJS + socket.io (the exchange of messages between the front-end and the backend). Simplified, it looks like this:
')
The application operates with certain entities that are stored in the “Rails db” database. Services operate on the same entities as the main application, but each have their own separate database. Synchronization of entities occurs through messages in ZeroMQ by events, which generates the main application. Each service consists of the service itself, and a set of event listeners (separate processes in essence) that receive messages (each listener processes messages of the same type) and tell the service what to do (which fields in the database need to be changed). The process of the service itself also accepts a certain type of messages, by which it sends data from its database to the main application. Both services and the main application are actively working with Redis (read and write); there is no synchronization / locking (i.e. races are possible, and they exist). This is for the application architecture. Now we will briefly go through the servers and move on to the most interesting - the features of the extrapath.
So, servers are regular dedicated servers (who said “virtual”? Ordinary pieces of iron). The average configuration is 2x Core i7, 32–64 GB RAM, 2x 1T HDD / RAID1. In terms of quantity, we count. Two “balancers” (one main, the second - “hot” reserve, switching - floating IP). Three servers for services. One server for the database. One server for broker and redis. One server is a test server (all components are spinning on it at once). Total - 8 pieces. There is no LAN between the servers, all traffic walks through the DC (some servers are separated by different racks).
What is wrong with the above, you ask? Yes, everything is so, but the devil, as usual, is in the details, but in our case it is the daily operation and support. Next, just unstructured describe some points that I think problematic.
The instructions for adding a new server begin with the words “Install Gentoo”. Yes, the whole stack is spinning on Gentoo, with all the consequences. Provisioning is done in a bash script. For configuration management synctool is used (such rsync on steroids in fact). Depla application - git pull hands on all seven servers. In the code of the main application in the top ten places, the host names of the database, Redis, the broker, are nailed. The separation in production / development code is very conditional (it takes 2-3 days for a programmer to set up an environment for development). Services are so closely integrated with the main application that when one falls, the whole platform collapses. The service itself is from three to fifteen processes, each of which is simply started up by hands on the screen (yes, 15-20 screen sessions are constantly running on one of the servers, and in the documentation repository there is a file “these screens should be on this server after rebut ”).
And most importantly - the admin of the application. Born in the hidden recesses of the consciousness of the gloomy Teutonic genius, an explosive mixture of bash, Ruby and Falcon . Did you know that there is such a programming language - Falcon? Now you know (follow the link, by the way, the language is worthy of attention as an extension of the horizon). The admin panel itself is simply a collection of SQL queries, and pieces of Ruby code, which allows you to operate with different parameters of entities. There is nothing in it that would justify using anything else bash. The same falcon - for what the hell is it used there, I did not understand; in reality, everything that falcon scripts do is form something like this under certain conditions, and perform it through system ():
echo "SELECT stuff FROM entities" | psql -U postgres db_name Conclusion
Everything described above was created by one person for two years. I will not evaluate his programmer skills, but the admin (or devops, as you wish) from him, frankly, is crappy - which is worth only one Gent per share, instead of a normal binary distribution. At some point, even a business realized that it wouldn’t live normally, and I was asked to see what could be done to try to take off with all this garbage. In the next part I will tell you what and how I did to diminish the amount of entropy in the Universe.
Source: https://habr.com/ru/post/317212/
All Articles