Clustering Technologies - Improving the Reliability and Performance of a Web Project
Good afternoon, colleagues!
Today I want to talk about clustering technologies that appeared in the 10th version of the 1C-Bitrix platform . We will also analyze what important tasks they will allow you to solve, as a web project manager, and how to effectively use these technologies if you are a developer. At the end of the article we will look at specific architectures of working web clusters on the 1C-Bitrix platform, incl. resource www.1c-bitrix.ru .
But first - we will try to objectively answer the question - in which cases it is still advisable to choose a framework or a boxed solution to implement a web project, and in which cases - to create a web project from scratch.
It would seem that the process of developing a web project from scratch looks relatively simple and leads to success:
But ... in practice:
Unfortunately, it often happens that after launching a web project into operation, the web solution code created by the development team is of poor quality (different people write, different qualifications, at different times, no one checks their work, etc., and if “Coercion to work”, then write in general is gone :-)). For you, this means that making changes / improvements to the web project will be more and more expensive (an attempt to put concrete walls on a clay foundation), and the system will work slower and slower with an increase in website traffic. Periodically, in order to somehow move forward and develop a web project, you will most likely need to completely rewrite its individual parts - which may be fraught with a violation of the functionality and stability of other parts (which is fraught with a stream of complaints from site visitors, loss of orders, disappearance of financial transactions, etc.).
')
And with a significant increase in web resource traffic, for example, up to millions of hits per day, it may turn out that the web solution was not created at all for such workloads and should be “rewritten” urgently.
Choosing a framework , in theory, you reduce the risks and development time. Indeed, because I use something like a frame / foundation on which I “string” the functionality of the website I need. But many risks, unfortunately, remain:
Debugging tools of the 1C-Bitrix platform: poor-quality development on the framework - will not work :-)
The load tools of the 1C-Bitrix platform make it quite easy to assess the web project’s resilience to high loads.
One of the cornerstone features of a quality framework is to enable, by creating simple add-ons, to implement different scenarios for its use. For example, can I embed a non-standard algorithm for calculating discounts in an online store without costly rewriting the core of the framework by highly qualified programmers with 5 years of experience in developing the framework :-)? Can I use different currencies whose rates depend on the region? Is it possible to use an external database of accounts from the existing system for authorization on a web project?
In BitrixFramework, due to the mandatory phase of in-depth analytics of the problem area when developing the kernel and platform modules, the main common business processes that may be required in 95% of web projects are implemented. To create add-ons, you can always use the mechanism of simple event handlers , with which you can implement almost any fantastic scenario of a business process by a programmer of primary or secondary qualification.
However, sometimes it may still turn out that the task on the framework is more difficult to implement and longer than writing from scratch, and you will find out about it already in the development process :-). In this case, frameworks, as a rule, provide the possibility of modular expansion. In BitrixFramework, you will need to create a module that implements non-standard logic. Writing a module is usually longer than creating an event handler, but it achieves maximum flexibility. Writing a module is relatively simple, but perhaps such a module has already been written - therefore, it is worthwhile to pre-rummage in the marketplace .
This approach is likely to minimize the time and risks of developing a web project, if your requirements are “close” to the functionality of the box. For example, for creating and launching a typical online store with a multi-currency catalog, authorization, discounts, personal section, search and support for multiple payment systems, the boxed solution “1C-Bitrix: Site Management” ( edited by “Small Business” ) is likely to be suitable.
Additionally, the packaged solution, being like a “success story” and a “book of recipes”, helps you, as a project customer, to “touch” practical examples of the implementation of business tasks. And the box allows developers to study the “source code” of the solution in order to duplicate it, or make it better :-). Frequent case when using the box: "Do as here and here, so that the admin panel can be controlled in the same way as there."
The “correct” box does not impose its decisions as the ultimate truth, allowing you to “cut off” the layer of the working implementation and replace it with your own implementation, based on the framework. In 1C-Bitrix, you can easily uninstall a ready-made solution from the system, install a new one for experiments, including from the Marketplace, and choose your own, most optimal way.
If you are already using or planning to develop on a 1C-Bitrik framework / box, then most likely:
It is to ensure the high availability and performance of a web project on the 1C-Bitrix platform in version 10 it became possible to use the popular and efficient clustering technologies in the product:
1) Vertical sharding (making system modules on separate MySQL servers). Now it is possible to bring out the intensively used “Search” and “Web analytics” modules on your own hardware.
2) MySQL replication and load balancing between servers. Now you can distribute the load on the web project DBMS on a group of servers, adding them if necessary. Technical details can be read here .
3) Distributed data cache (memcached). Reliable and scalable caching is critical for high-load projects. Technical details here .
4) Clustering sessions between web servers. This is necessary when running a group of web servers behind the balancer. Why we do not store the session in memcached - we will write about it separately :-).
5) Clustering web server files. This is necessary to automatically synchronize content between machines. Technical details here .
I would like to draw particular attention to the fact that clustering technologies are used at the kernel level of the 1C-Bitrix platform and to refine the code of the web project - no need! (every time I write about it, I don’t believe it, but ... it really is) Of course, if necessary, you can specify in the code which replication servers to read / write from — but this is necessary for the overwhelming minority of web projects.
Also, if you are developing a web project that may need clustering technologies in the future - you can, in principle, “connect” them later, when they become necessary, without modifying the code of the web project. This seriously reduces the cost of developing a web project, because if you immediately create a system under high load, working on 5 databases, with a cluster file system and horizontal sharding - then in case of its lack of demand - you will waste time and money.
The most interesting section of the article - finally, we will look at specific examples of existing web clusters on the 1C-Bitrix platform.
The introduction of a web cluster was conducted by our certified partner Linemedia , Moscow. Editorial platform - Business Web Cluster.
The web project is visited daily by several hundreds of thousands of users, and project partners who have access to the project through the Kupibonus API also generate additional workload. Active two-way integration with the 1C: Enterprise back-office, intensive channels of interaction with external services.
Due to the implementation of the web cluster, high performance and fault tolerance are provided not only for the public part, but also for internal business processes and APIs. The implementation took place in a rather short time - two weeks.
The introduction of a web cluster was held by a gold certified partner “Professional Internet Solutions” , Izhevsk. Editorial platform - "Web Cluster".
The web project is a high-loaded platform for electronic trading by a group of companies - KOMOS GROUP. The group includes 13 largest food producing companies in Udmurtia and the Perm Territory. The resulting bid price amounted to almost 5 billion rubles in less than a year. Due to the implementation of the web cluster, high performance and fault tolerance, easy scalability of the electronic trading platform were ensured.
The web cluster is hosted on an EBS / EC2 group of machines in the Amazon cloud. A sufficiently large amount of static content. High load on both the public part and the integration API (marketplace), as well as the update system. Due to the transition to the web cluster, high performance and stability are provided, fast scaling if necessary.
We wanted to use the balancer built into the cloud, but for now, because of its “inflexibility,” we are using a separate machine for this. We make backups in S3 with convenient cloud snapshots built into the cloud, for backing up raid10 based on EBS we put XFS on it and, for holistic snapshots, freeze it using the ec2-consistent-snapshot utility. Statics are gradually brought to the cloud and CDN. Most routine operations: lift the machine in the event of a load, load switching, backups, etc. - scripted on amazon API.
Currently, several web clusters are preparing to launch from the domestic cloud provider Oversan-Scalaxi . We will write and draw about them.
Our upcoming technology plans for cloud and clustering
At the recent partner conference, we announced that we will do in the next release of platform 10.5, which will be released in the fall, support for cloud storage: Amazon / S3, Google Storage, data storage technologies in Microsoft Azure.
Also, in version 10.5, we want to offer our clients a solution that makes their web cluster as reliable as possible and ... abstracted from the cloud provider - so that in case of serious problems in the cloud / DC, the web cluster continues to serve clients in another cloud region / other continent:
Cluster grid (this is still one of their working technology names)
We are looking at the latest cloud clustering technologies with interest and we see considerable potential in them to ensure the reliability and performance of web projects. We will continue to implement the best technologies in the 1C-Bitrix platform - making them useful and accessible to our many clients .
I wish you all technological luck, optimism and see you again in our corporate blog!
References:
Alexander Serbul, head of quality control of integration and implementation of 1C-Bitrix
Today I want to talk about clustering technologies that appeared in the 10th version of the 1C-Bitrix platform . We will also analyze what important tasks they will allow you to solve, as a web project manager, and how to effectively use these technologies if you are a developer. At the end of the article we will look at specific architectures of working web clusters on the 1C-Bitrix platform, incl. resource www.1c-bitrix.ru .
But first - we will try to objectively answer the question - in which cases it is still advisable to choose a framework or a boxed solution to implement a web project, and in which cases - to create a web project from scratch.
Web development from scratch
It would seem that the process of developing a web project from scratch looks relatively simple and leads to success:
- We design and estimate the scope of work.
- We are recruiting a team of web developers, web designers (perhaps designers)
- We program, we type, we test
- We fill the web project with content
- Run
- We support and develop the web project
But ... in practice:
- You will need qualified system analysts and technical experts to design and estimate the scope of work, which are few in the market and they are expensive
- You need to recruit a team of qualified web developers, which are few in the market, but they are expensive
- You need to select and configure an effective development process, incl. provide the team with tools such as tracker, development environment, version control systems, etc.
- To quickly fill a web project with content you need ... to develop an "admin panel" for all types of content, which means additional time and money.
- It will require close cooperation with qualified system administrators for setting up servers and software, organizing backup and monitoring, security audits
- It is necessary to select and implement tools for the “transfer” of web-site modifications from the development team to the “live broadcast” —for this, for example, version control systems and / or continuous integration tools are used.
Unfortunately, it often happens that after launching a web project into operation, the web solution code created by the development team is of poor quality (different people write, different qualifications, at different times, no one checks their work, etc., and if “Coercion to work”, then write in general is gone :-)). For you, this means that making changes / improvements to the web project will be more and more expensive (an attempt to put concrete walls on a clay foundation), and the system will work slower and slower with an increase in website traffic. Periodically, in order to somehow move forward and develop a web project, you will most likely need to completely rewrite its individual parts - which may be fraught with a violation of the functionality and stability of other parts (which is fraught with a stream of complaints from site visitors, loss of orders, disappearance of financial transactions, etc.).
')
And with a significant increase in web resource traffic, for example, up to millions of hits per day, it may turn out that the web solution was not created at all for such workloads and should be “rewritten” urgently.
Development of a web project based on the framework
Choosing a framework , in theory, you reduce the risks and development time. Indeed, because I use something like a frame / foundation on which I “string” the functionality of the website I need. But many risks, unfortunately, remain:
- Now you need "special" experts who know the framework well for designing and evaluating labor costs. When developing on the basis of BitrixFramework, you can quite simply “pump” employees through their free certification for our courses , after which they can, in principle, fairly accurately estimate the amount, complexity of work and determine what to write “from scratch” and which solutions can be used in finished form.
- The frameworks have different levels of entry, so to work with popular "academic" frameworks, such as ZendFramework , Symfony , Yii , you will need programmers of higher cost and qualification than, for example, working with BitrixFramework (based on products and solutions 1C- Bitrix), using simple architectural concepts and patterns, understandable to most developers of primary and secondary qualifications. To speed up the entry of BitrixFramework developers into production, the “Quick Start Developer” course will help.
- The technological process of developing a web project based on the framework, as a rule, is not much different from developing from scratch. However, some frameworks, such as BitrixFramework, provide convenient tools for auditing and monitoring development quality: performance monitoring, SEO auditing , debugging tools.
Debugging tools of the 1C-Bitrix platform: poor-quality development on the framework - will not work :-)
- For quick and convenient filling of a web project with content, some frameworks offer automated creation of “admins” (a semi-finished product, on the basis of which you can, in principle, speed up the creation of certain types of administrative pages). BitrixFramework has a built-in functional and quite convenient administrative interface that allows customization for the needs of the project. For example, you can make your "admin" to add news or process orders.
- For load testing a web project, before launching, frameworks sometimes provide technical tools for load building and auditing. BitrixFramework has efficient and popular embedded client audit subsystems for hardware and software configuration and load creation.
The load tools of the 1C-Bitrix platform make it quite easy to assess the web project’s resilience to high loads.
- Not all frameworks, unfortunately, are able to “update” themselves - and this is necessary for quick elimination of errors, improvement of the current and getting new functionality. The system of updates is built into BitrixFramework - which allows you to constantly maintain the core of the web project in a permanently good tone.
Empowerment - Blessing or Curse?
One of the cornerstone features of a quality framework is to enable, by creating simple add-ons, to implement different scenarios for its use. For example, can I embed a non-standard algorithm for calculating discounts in an online store without costly rewriting the core of the framework by highly qualified programmers with 5 years of experience in developing the framework :-)? Can I use different currencies whose rates depend on the region? Is it possible to use an external database of accounts from the existing system for authorization on a web project?
In BitrixFramework, due to the mandatory phase of in-depth analytics of the problem area when developing the kernel and platform modules, the main common business processes that may be required in 95% of web projects are implemented. To create add-ons, you can always use the mechanism of simple event handlers , with which you can implement almost any fantastic scenario of a business process by a programmer of primary or secondary qualification.
However, sometimes it may still turn out that the task on the framework is more difficult to implement and longer than writing from scratch, and you will find out about it already in the development process :-). In this case, frameworks, as a rule, provide the possibility of modular expansion. In BitrixFramework, you will need to create a module that implements non-standard logic. Writing a module is usually longer than creating an event handler, but it achieves maximum flexibility. Writing a module is relatively simple, but perhaps such a module has already been written - therefore, it is worthwhile to pre-rummage in the marketplace .
Development of a web project based on a box solution
This approach is likely to minimize the time and risks of developing a web project, if your requirements are “close” to the functionality of the box. For example, for creating and launching a typical online store with a multi-currency catalog, authorization, discounts, personal section, search and support for multiple payment systems, the boxed solution “1C-Bitrix: Site Management” ( edited by “Small Business” ) is likely to be suitable.
Additionally, the packaged solution, being like a “success story” and a “book of recipes”, helps you, as a project customer, to “touch” practical examples of the implementation of business tasks. And the box allows developers to study the “source code” of the solution in order to duplicate it, or make it better :-). Frequent case when using the box: "Do as here and here, so that the admin panel can be controlled in the same way as there."
The “correct” box does not impose its decisions as the ultimate truth, allowing you to “cut off” the layer of the working implementation and replace it with your own implementation, based on the framework. In 1C-Bitrix, you can easily uninstall a ready-made solution from the system, install a new one for experiments, including from the Marketplace, and choose your own, most optimal way.
For what types of web projects will the 1C-Bitrix clustering technology be useful?
If you are already using or planning to develop on a 1C-Bitrik framework / box, then most likely:
- Your project has complex business logic and extensive functionality that fits well with the capabilities of the 1C-Bitrix platform.
- You want to reduce risks and increase development speed by using the BitrixFramework industrial framework.
- You are concerned about the stability of a web project to high loads ( millions of hits per day ) and a simple website for an hour is simply unacceptable, because faces serious losses and reputation damage
It is to ensure the high availability and performance of a web project on the 1C-Bitrix platform in version 10 it became possible to use the popular and efficient clustering technologies in the product:
1) Vertical sharding (making system modules on separate MySQL servers). Now it is possible to bring out the intensively used “Search” and “Web analytics” modules on your own hardware.
2) MySQL replication and load balancing between servers. Now you can distribute the load on the web project DBMS on a group of servers, adding them if necessary. Technical details can be read here .
3) Distributed data cache (memcached). Reliable and scalable caching is critical for high-load projects. Technical details here .
4) Clustering sessions between web servers. This is necessary when running a group of web servers behind the balancer. Why we do not store the session in memcached - we will write about it separately :-).
5) Clustering web server files. This is necessary to automatically synchronize content between machines. Technical details here .
I would like to draw particular attention to the fact that clustering technologies are used at the kernel level of the 1C-Bitrix platform and to refine the code of the web project - no need! (every time I write about it, I don’t believe it, but ... it really is) Of course, if necessary, you can specify in the code which replication servers to read / write from — but this is necessary for the overwhelming minority of web projects.
Also, if you are developing a web project that may need clustering technologies in the future - you can, in principle, “connect” them later, when they become necessary, without modifying the code of the web project. This seriously reduces the cost of developing a web project, because if you immediately create a system under high load, working on 5 databases, with a cluster file system and horizontal sharding - then in case of its lack of demand - you will waste time and money.
Cluster architectures of working web projects based on 1C-Bitrix
The most interesting section of the article - finally, we will look at specific examples of existing web clusters on the 1C-Bitrix platform.
www.kupibonus.ru - uninterrupted service delivery
The introduction of a web cluster was conducted by our certified partner Linemedia , Moscow. Editorial platform - Business Web Cluster.
The web project is visited daily by several hundreds of thousands of users, and project partners who have access to the project through the Kupibonus API also generate additional workload. Active two-way integration with the 1C: Enterprise back-office, intensive channels of interaction with external services.
Due to the implementation of the web cluster, high performance and fault tolerance are provided not only for the public part, but also for internal business processes and APIs. The implementation took place in a rather short time - two weeks.
www.komos.ru - electronic exchange
The introduction of a web cluster was held by a gold certified partner “Professional Internet Solutions” , Izhevsk. Editorial platform - "Web Cluster".
The web project is a high-loaded platform for electronic trading by a group of companies - KOMOS GROUP. The group includes 13 largest food producing companies in Udmurtia and the Perm Territory. The resulting bid price amounted to almost 5 billion rubles in less than a year. Due to the implementation of the web cluster, high performance and fault tolerance, easy scalability of the electronic trading platform were ensured.
www.1c-bitrix.ru is us
The web cluster is hosted on an EBS / EC2 group of machines in the Amazon cloud. A sufficiently large amount of static content. High load on both the public part and the integration API (marketplace), as well as the update system. Due to the transition to the web cluster, high performance and stability are provided, fast scaling if necessary.
We wanted to use the balancer built into the cloud, but for now, because of its “inflexibility,” we are using a separate machine for this. We make backups in S3 with convenient cloud snapshots built into the cloud, for backing up raid10 based on EBS we put XFS on it and, for holistic snapshots, freeze it using the ec2-consistent-snapshot utility. Statics are gradually brought to the cloud and CDN. Most routine operations: lift the machine in the event of a load, load switching, backups, etc. - scripted on amazon API.
Currently, several web clusters are preparing to launch from the domestic cloud provider Oversan-Scalaxi . We will write and draw about them.
Our upcoming technology plans for cloud and clustering
At the recent partner conference, we announced that we will do in the next release of platform 10.5, which will be released in the fall, support for cloud storage: Amazon / S3, Google Storage, data storage technologies in Microsoft Azure.
Also, in version 10.5, we want to offer our clients a solution that makes their web cluster as reliable as possible and ... abstracted from the cloud provider - so that in case of serious problems in the cloud / DC, the web cluster continues to serve clients in another cloud region / other continent:
Cluster grid (this is still one of their working technology names)
We are looking at the latest cloud clustering technologies with interest and we see considerable potential in them to ensure the reliability and performance of web projects. We will continue to implement the best technologies in the 1C-Bitrix platform - making them useful and accessible to our many clients .
I wish you all technological luck, optimism and see you again in our corporate blog!
References:
Alexander Serbul, head of quality control of integration and implementation of 1C-Bitrix
Source: https://habr.com/ru/post/123375/
All Articles