Architecture highload project on the example of a web consultant
Our team is engaged in remote administration of servers and, not so long ago, representatives of the WebConsult service turned to us with the task of building an easily scalable server architecture that will withstand serious workloads. We decided that maybe it would be interesting for Habrahabr users who are somehow connected with the administration of Highload projects. The project turned out to be fast-growing and the existing structure was already working at the limit, so we had to launch a new one in an accelerated mode.
')
To do this quickly, we tried to distribute the tasks as much as possible among all our administrators. To do this, we used the project and task management system - Redmine and in the same place maintained contact with the project developers, who promptly answered our questions and carried out the necessary improvements in the code.
To understand the essence of the task before us, let's see what the principle of the project is. WebConsult is a system through which visitors can contact the store or company manager and ask their questions in real time. Managers, for their part, have a choice of using two counseling options - through a panel open in a browser or through any familiar Jabber client. In turn, visitors to the sites where the system is installed see the call button of a consultant, which is loaded from the WebConsult server (which means that the speed of the service is very important). Since the button is downloaded several million times per day, this creates a significant load on web servers. Also, a serious part of the burden is made up of operators who use the main method of consulting - through a web client. In this case, HTTP requests for updates (a new client, a message, a change in various statuses) come from thousands of consultants who are simultaneously online after a certain number of seconds. This also consumes a huge amount of HTTP servers and database servers.
Let's move on to considering the architecture we have built. The basis of everything is balancing at the DNS level (round-robin), which transmits traffic on two fronts (this is convenient, including for cases when one of the servers is briefly unavailable - modern browsers transfer the request automatically to the second server). The fronts, in turn, proxy traffic using nginx to HTTP servers — web1, web2, web3, and so on.
We tried to create our OpenVZ container for each individual task in order to isolate the logical parts of the project from each other, as well as to be able to quickly allocate the container to a separate physical server. Why we chose this virtualization, we described in one of the past articles .
In order to save traffic and optimize load, all passing statics are cached using nginx tools. In the previous architecture, NFS was used to store files and source code, but it was decided to abandon it due to the fact that it exerted additional load. Instead, we began to catch requests for static downloads (avatars of consultants, company logos) and redirect them to one server. NGINX is set up so that if it does not find the pictures locally, it searches for them on other web servers using try_files $ uri servers , and in the named servers locale, the search of web servers is registered. Also, static is synchronized with other servers several times a day, which makes them available locally.
An important part of the WebConsult project is working with clients on the site via Jabber. As a XMPP server, Openfire is used with a number of self-written modules that allow you to integrate with the internal structure of the project and send messages from Jabber to HTTP and back. For the convenience of customers, the task was to make the ability to connect to the jabber-server not only through the direct domain jabber.consultsystems.ru, but also through the main domain consultsystems.ru. For this it was necessary to organize the proxying of ports. We decided to replace the previously used rinetd with a more advanced solution and used HAProxy for this task, which forwards ports 5222, 5223 in TCP mode.
Since WebConsult is a project that is constantly being developed and expanded, it was necessary to come up with a solution that allows developers to see the changes made on the server in their own sandbox, so that customers notice them only after everything has been tested and completed. To do this, a separate OpenVZ container was created with a minimum amount of allocated resources, where a version of the site was launched for testing and debugging. GIT was chosen as a version control system as a familiar and modern solution. As a result, it turned out that developers can introduce new functions, test them on a separate server, and after everything is ready to produce an updated version of the project in production, which virtually eliminates the possibility of failure and errors in the working copy of the project. Also, for convenience, we have written a small plugin for Redmine, which allows authorized users to do warm deploy by pressing a button.
We chose Percona Server as the database server as a more efficient version of the mysql server.
Memcached is used for central storage of user sessions. Access to it is open only for the web by iptables rules.
As a result, we have a structure that easily horizontally scales by adding new web servers by “cloning” the existing OpenVZ container and transferring it to a new machine. As for the scalability of the database, at the moment, one powerful server handles everything, and in the future replication will be applied. Also, some of the “heavy” functions for which MySQL is currently used will be transferred to NoSQL solutions, for example Redis, which will seriously relieve the servers. Another very important task at the moment is to transfer chat to web sockets, in order to refuse to perform regular HTTP requests and instead keep a constant connection to the server - work in this direction is already underway.
Thank you for your attention, if you have any questions or suggestions, we will be happy to discuss in the comments.
')
To do this quickly, we tried to distribute the tasks as much as possible among all our administrators. To do this, we used the project and task management system - Redmine and in the same place maintained contact with the project developers, who promptly answered our questions and carried out the necessary improvements in the code.
To understand the essence of the task before us, let's see what the principle of the project is. WebConsult is a system through which visitors can contact the store or company manager and ask their questions in real time. Managers, for their part, have a choice of using two counseling options - through a panel open in a browser or through any familiar Jabber client. In turn, visitors to the sites where the system is installed see the call button of a consultant, which is loaded from the WebConsult server (which means that the speed of the service is very important). Since the button is downloaded several million times per day, this creates a significant load on web servers. Also, a serious part of the burden is made up of operators who use the main method of consulting - through a web client. In this case, HTTP requests for updates (a new client, a message, a change in various statuses) come from thousands of consultants who are simultaneously online after a certain number of seconds. This also consumes a huge amount of HTTP servers and database servers.
Balancing
Let's move on to considering the architecture we have built. The basis of everything is balancing at the DNS level (round-robin), which transmits traffic on two fronts (this is convenient, including for cases when one of the servers is briefly unavailable - modern browsers transfer the request automatically to the second server). The fronts, in turn, proxy traffic using nginx to HTTP servers — web1, web2, web3, and so on.
Virtualization
We tried to create our OpenVZ container for each individual task in order to isolate the logical parts of the project from each other, as well as to be able to quickly allocate the container to a separate physical server. Why we chose this virtualization, we described in one of the past articles .
NGINX web server
In order to save traffic and optimize load, all passing statics are cached using nginx tools. In the previous architecture, NFS was used to store files and source code, but it was decided to abandon it due to the fact that it exerted additional load. Instead, we began to catch requests for static downloads (avatars of consultants, company logos) and redirect them to one server. NGINX is set up so that if it does not find the pictures locally, it searches for them on other web servers using try_files $ uri servers , and in the named servers locale, the search of web servers is registered. Also, static is synchronized with other servers several times a day, which makes them available locally.
Jabber
An important part of the WebConsult project is working with clients on the site via Jabber. As a XMPP server, Openfire is used with a number of self-written modules that allow you to integrate with the internal structure of the project and send messages from Jabber to HTTP and back. For the convenience of customers, the task was to make the ability to connect to the jabber-server not only through the direct domain jabber.consultsystems.ru, but also through the main domain consultsystems.ru. For this it was necessary to organize the proxying of ports. We decided to replace the previously used rinetd with a more advanced solution and used HAProxy for this task, which forwards ports 5222, 5223 in TCP mode.
For developers
Since WebConsult is a project that is constantly being developed and expanded, it was necessary to come up with a solution that allows developers to see the changes made on the server in their own sandbox, so that customers notice them only after everything has been tested and completed. To do this, a separate OpenVZ container was created with a minimum amount of allocated resources, where a version of the site was launched for testing and debugging. GIT was chosen as a version control system as a familiar and modern solution. As a result, it turned out that developers can introduce new functions, test them on a separate server, and after everything is ready to produce an updated version of the project in production, which virtually eliminates the possibility of failure and errors in the working copy of the project. Also, for convenience, we have written a small plugin for Redmine, which allows authorized users to do warm deploy by pressing a button.
Database
We chose Percona Server as the database server as a more efficient version of the mysql server.
Memcached is used for central storage of user sessions. Access to it is open only for the web by iptables rules.
Scaling
As a result, we have a structure that easily horizontally scales by adding new web servers by “cloning” the existing OpenVZ container and transferring it to a new machine. As for the scalability of the database, at the moment, one powerful server handles everything, and in the future replication will be applied. Also, some of the “heavy” functions for which MySQL is currently used will be transferred to NoSQL solutions, for example Redis, which will seriously relieve the servers. Another very important task at the moment is to transfer chat to web sockets, in order to refuse to perform regular HTTP requests and instead keep a constant connection to the server - work in this direction is already underway.
Thank you for your attention, if you have any questions or suggestions, we will be happy to discuss in the comments.
Source: https://habr.com/ru/post/166739/
All Articles