Architecture Bitrix24 - Inside View

On April 12, we launched a large new project, Bitrix24 : a social intranet, a SaaS service that combines classic teamwork tools (calendars, tasks, CRM, work with documents) and social communications (likes, social search, instant messages). and much more).



The first prototype of this service was launched in February last year. On one server, without any special opportunities for scaling, without redundancy at the data center level ... :) Only concept.

')

With this publication we will open a series of posts in which we would like to tell you what was done during the year of development, what the resulting architecture of the project turned out to be; what we are doing to ensure real “24” hours of project work per day; what changes had to be made in the 1C-Bitrix development platform; features of working in the Amazon cloud and more.



We really want our experience to be useful for all developers who are only planning to launch or are already exploiting and supporting large web projects and services.



* * *



So, the first post is about the architecture of the project as a whole. Go!



In the process of developing the very idea of Bitrix24, we formulated for ourselves several business objectives:

• From the very beginning, we assumed that the first tariff for Bitrix24 would be free.
• This means that the cost of such a free account for us should be very low.
• Our project is a business application, and it means that the load on it will be very uneven: more during the day, less at night. Ideally, it would be good to be able to scale (in both directions) and at each moment of time use exactly as many resources as necessary.
• At the same time - for any business application reliability is extremely important: the constant availability of data and their safety.
• We started in several markets at once: Russia, USA, Germany.

These business requirements ultimately formed two large “fronts” of work: the formation of a scalable fault-tolerant (running a little ahead - “cloud”) development platform and the choice of a technological platform for the project infrastructure.



Development Platform "1C-Bitrix"



The traditional device of web applications is very poorly scaled and reserved. At best, we can spread the application itself, the cache and the database to different servers. We can somehow scale the web (but at the same time we will have to solve the issue of data synchronization on web servers). The cache and base scale is worse. And we are not talking about a distributed geo-cluster (for redundancy at the level of data centers).



A huge step in the development of the 1C-Bitrix platform was the emergence of the Web Cluster module in version 10.0 last spring.



We wrote in detail about him on Habré . I will briefly repeat the main features:

• Vertical sharding (making separate modules on separate MySQL servers)
• MySQL replication and load balancing between servers at the platform core level
• Distributed data cache (memcached)
• Session Continuity Between Web Servers (Storing Sessions in Database)
• Web server clustering

As a result, we were able to present our project in the form of a web cluster of interchangeable servers. Everything looked very, very good: with an increase in attendance it was possible to quickly add new servers to the cluster; in case of failure of one or several servers of the cluster, everything continued to work on the remaining servers, the system continued to continuously serve clients.







But the ideal was still far away, there were “narrow” places:

• The issue of file synchronization across different web servers was not fully resolved. On our own site, we used csync2 , it has proven itself very well. However, on large amounts of data, we simply would not have time to transfer changes across all servers.
• The failure of the master server in MySQL replication meant some manual or semi-automatic operations to transfer one of the slaves to master mode. It took time, which means we could not guarantee the uninterrupted operation of the service.
• slaves allow you to distribute the load on the database to read, and master is still alone. This means that the record remains a bottleneck in the database.
• As our own sad experience has shown, lightning strikes datacents and can completely disable them. So, you need to reserve not only individual servers, but also the entire data center.

In the 1C-Bitrix platform version 11.0, released in the fall of 2011, we also solved these tasks. Cloud file storage support solved the problem of synchronization of static content, and the implementation of master-master replication support in MySQL allowed us to build geographically distributed web clusters.



And we came close to the second big task ...



Platform for deploying infrastructure



To be honest, the choice was not very difficult. :)



“Cloud” or not “cloud” - this question did not even stand. :)



Own or rented equipment requires quite serious investments in infrastructure at the start of the project. Scaling physical "glands" is quite difficult (long and expensive). To administer (especially in different DCs) is inconvenient.



Therefore - the "cloud"!



Which one exactly? We chose Amazon AWS.



All of our sites are working in Amazon for a long time. We like the fact that there are many ready-made services that you can just take and use in your project, and not reinvent your own bikes: cloud storage S3 , Elastic Load Balancing , CloudWatch , AutoScaling and much more.



In a very simplified form, the entire architecture of Bitrix24 looks like this:







Web - auto scaling



The application (web) is scaled not vertically (increase in server capacity), but horizontally (we add new machines).



For this we use a bunch of Elastic Load Balancing + CloudWatch + Auto Scaling. All client requests (HTTP and HTTPS) arrive at one or more Amazon balancers (ELB). Growth and load reduction monitor through CloudWatch. There are two interesting metrics - the state of EC2 nodes (% CPU Utilization) and the balancer (latency time - in seconds).



We use data on the load of machines as the main characteristic, since latency can vary not only because of the real load, but also for some other reasons: network delays, errors in the application, etc. In this case, "false" triggers are possible, and then we will be extremely inefficient to add new cars.



Now, as a result, new machines will automatically start if the average utilization of the CPU (in terms of Amazon) exceeds 60%, and automatically stops and is taken out of service if the average load is less than 30%.



We experimented with these threshold values ​​for a long time and now we consider them optimal. If the upper threshold is set higher (for example, 70-80%), then a general degradation of the system begins - users are uncomfortable to work (pages load for a long time). The lower threshold is less - the balancing system becomes not very efficient, the machines can run idle for a long time.







Static content of service users



In the scheme described above, web nodes essentially become "consumable". They can start at any time and be extinguished at any time. This means that there should be no user content on them.



That is why when you create each new portal in Bitrix24, it creates a personal account in Amazon for storing data in S3. Thus, the data of each portal are completely isolated from each other.



At the same time, the S3 storage itself is very reliable. Amazon itself describes in detail its device (and, in principle, there is no reason not to believe them).



Data in S3 is replicated at several points. At the same time - to geographically distributed points (different data centers).



Each of the storage devices is monitored and quickly replaced in the event of certain failures.



When you upload new files to the repository, you will receive an answer about successful upload only when the file is completely saved at several different points.



Typically, data is replicated to three or more devices - to ensure fault tolerance even in case of failure of two of them.



In other words, reliability is good. The same Amazon, for example, says that their S3 architecture is designed in such a way that they are ready to ensure availability at the level of two nines after the comma. And the probability of data loss is one billionth of a percent.



Two data centers and master-master replication



The whole project is now located in two data centers. We solve two problems with this: first, we distribute the load (now Russian users work in one DC, and American and European - in the other), and second, we reserve all services - in case of failure of one of DC, we just switch traffic to another.



A little bit more - how it all works.



The base in each DC is a master relative to the slave in the second DC and simultaneously a slave relative to the master.



Important settings in MySQL for implementing this mechanism are auto_increment_increment and

auto_increment_offset . They set the offset values ​​for the auto_increment fields - in order to avoid duplicate entries. Roughly speaking, in one master - only even IDs, in the other - only odd ones.



Each portal (all employees registered in it), started up in Bitrix24, at each particular time works with only one DC and one base. Switching to another DC is carried out only in case of any failure.



Databases in different data centers are synchronous, but independent of each other: the loss of connectivity between data centers can be hours, data is synchronized after recovery.



Reliability, Reliability, Reliability



One of the most important priorities in Bitrix24 is the constant availability of the service and its resiliency.



Remember the simple service scheme? As a result (it’s still simplistic, but nonetheless :)) it looks like this:







In the event of an accident on one or several web nodes, Load Balancing determines the failed machines and, based on the specified parameters of the balancing group (the minimum number of running machines), the necessary number of instances is automatically restored.



If the connectivity between data centers is lost, then each data center continues to serve its customer segment. After the restoration of connectivity, the data in the databases are automatically synchronized.



If the data center fails, or, for example, a failure occurs at the base, then all traffic automatically switches to a working data center.



If this causes an increased load on the machines, then CloudWatch determines the increased utilization of the CPU and adds the necessary number of machines in one data center in accordance with the rules for AutoScaling.



At the same time, our master replication is suspended. After carrying out the necessary work (recovery - in the event of an accident, or planned - for example, we, at exactly the same time, made the transition from standard MySQL to Percona Server - without any downtime for users of the service) , we include base in work and we restore replication.



If everything went smoothly, the traffic is again distributed to both data centers. If at the same time the average load was below the threshold value, then the extra machines that we raised to service the increased load are automatically extinguished.



* * *



I tried to make this first post an “easy” review. :) If I had put together everything that I wanted to share, in one text and told about all our experiences at once, then there would have been too many letters. :)



It seems to me more logical to make a series of thematic posts. Topics that we are already preparing:

• Amazon's nuances: non-obvious limits, different operating modes of balancers, working with AMI images
• The specifics of web nodes: software update mechanisms, Apache / non Apache, user security and isolation
• MySQL: why Percona, master-master features, replication of large amounts of data, nuances of query cache, sharding
• Data backups and most importantly - their recovery
• Monitoring thousands of objects, how not to spam yourself with alerts, automate the response to notifications

What is most interesting for you? You can take a spontaneous vote and mark directly in the comments. :)



And of course - try Bitrix24 itself! The first fare is free. If its limits or functionality is not enough - it will be possible to switch to higher tariffs. :)



Get connected - and work with pleasure!