Reliable maintenance of MS SQL Server databases for employees
You probably know that database maintenance is a whole complex of procedures: creating backups, checking integrity, maintaining indexes, statistics, etc. On open spaces of a network (and on Habré including) on ​​this subject many articles and recommendations are written. However, in implementing the implementation of "1C: Enterprise", we often have to deal with the fact that database maintenance is configured either incorrectly or in a very simplified scheme. For example, in order not to bother with managing transaction logs, the Simple Recovery model is set up for “combat” databases. And this is despite the fact that the loss of information for a couple of hours is already critical for the company. Sometimes the task of compressing the database files is included in the regular maintenance (“the shots did not grow”), or after updating the indexes, statistics are being destroyed and other similar lapses. This happens because, most often, companies do not have an experienced DBA and have to deal with someone from the IT service - the “unwitting” database administrator ( DBA ). Moreover, such a DBA is not always aware of all the risks and the responsibility assigned to it.
For database maintenance, Microsoft offers Maintenance Plans in SQL Server Management Studio (SSMS). However, as practice shows, only an experienced DBA can create and configure a high-quality and reliable service plan. I note that reliable service is as automated as possible and does not require regular manual monitoring by the administrator, and also ensures that data can be restored in the event of a failure.
Third-party programs that are available on the market and can make life easier, basically automate the creation of backups. The choice of such programs is very wide. They allow you to make backups compressed and encrypted, on FTP / GoogleDrive / Amazon and so on. Backups here can be compared with shrimps, which Babba spoke about in the picture “Forest Gamp”: “ ... they can be fried, cooked, baked, stewed, you can cook shrimp kebab, Creole shrimps, gumbo shrimps, fried with rice ... ”.
')
However, as has been said, setting up backups is far from everything, so such programs close only a fraction of the questions.
As a result, the “unwitting” DBA has to read articles, deal with SSMS, develop a backup strategy, search for scripts, set up notifications. It takes a lot of time, but there is always something to wipe out ... And I want to live peacefully! So that one did and forgot.
In this article, I would like to review our Quick Maintenance & Backup ( QMB ) program, which will help you to quickly and easily set up database maintenance on Microsoft SQL Server. There is no doubt that for large and high-loaded databases you cannot do without experienced DBA and individual tuning performance, but if you have to deal with many small databases (usually up to 50-80 GB), then this utility will be useful for both beginners and advanced. users.
The short video below shows the end-to-end QMB service setup example. The following description complements the video and tells about some features of the program.
On the one hand, we tried to make the program accessible to beginners and implement the most common maintenance scenarios. On the other hand, we wanted to make the program convenient for advanced users and help them set up a variety of scenarios, including combining database operations using Transact SQL with other regulatory procedures for your applications. For example, in QMB, you can make a script that will first load the data into 1C: Enterprise, and then make a backup and perform the rest of the maintenance. The result was a scheduler that provides its own framework for executing T-SQL scripts and batch files (with the ability to store the results of their execution).
The program has three components: the client GUI, the QMB Service, and the file base for storing your data. When installing QMB, all three components of the program are installed. Maintenance plans are not created, so the SQL Server Agent service is not required. Read more about architecture here .
As mentioned above, QMB does not create maintenance plans on SQL Server. Instead, a Service Policy is created, which is stored in the local storage (file database). In essence, a policy is a grouping of databases with similar properties that are served according to the same rules. The policy contains a list of databases, settings for storing and copying backups. The policy includes one or more maintenance scenarios . The script contains a set of tasks executed sequentially for each (policy included) database. If we draw an analogy with the maintenance plans, then the scenario can be compared with the nested Maintenance Plans (Maintenance Plan).
A task in QMB can be of one of five types:
The program has two sets of built-in tasks. The first set of tasks is based on T-SQL scripts, obtained from open sources and created by QMB developers. The second set is based on Ola Halengren’s scripts (a database administrator from Sweden), who developed three popular stored procedures for database maintenance. Ola procedures are installed automatically in the system master database when creating a policy from a template.
You can create a maintenance policy from a template or manually from scratch. The current version of the program includes 7 templates, which mainly differ:
For business databases with daily online data entry ( OLTP database), it is recommended to choose a policy with a Full Recovery Model - for example, for 1C: Enterprise databases, into which data are entered daily. This model allows you to restore the database to the current or arbitrary point in time.
A simple recovery model is recommended for archival and test databases, as well as various storages with rare occasional data downloads.
After you create a policy from a template, you can change any of its settings — scripts and schedule, tasks, and the order of notifications. In the future, the created policy can be copied to other servers registered in the program.
More information about the differences in the templates can be found in the help .
As mentioned above, QMB has 5 types of tasks - some of their features are described below.
Most system tasks are T-SQL scripts. The script itself can be viewed in the task form:
Script texts (T-SQL, CMD, VBS, PowerShell, and others) may contain markers that will be replaced with the corresponding values ​​before they are executed. For example, the ? DataBaseName? will be replaced with the database name, and the marker ? BackupDirectory? - to the path to the archived copies directory specified in the policy. A complete list of markers can be found in the help .
Optimization of the maintenance window
It happens that in a limited time window it is necessary to fit not only the database maintenance using SQL Server, but also the execution of other routine operations of your application. For example, testing and fixing 1C databases, unloading using the 1C: Enterprise platform, exchanging, etc. Typically, this is done using the Windows Task Scheduler or the 1C: Enterprise Scheduler. However, it is necessary to distribute the procedures in time with a good margin - so that they are guaranteed not to overlap. As a result, tasks may not fit into the existing time window.
With QMB, you can maximize the use of the maintenance window by combining the execution of T-SQL scripts and batch files in VBS, JavaScript, CMD, PowerShell, and others into scripts. Below is a simple example of an alternative backup copying task using the Robocopy utility:
It should be noted that the batch file can be executed both on the machine where the program is installed and on the SQL Server side. This allows you to operate backup files on the SQL Server side. For example, you can write a script that will archive the last backup and upload it to any cloud storage or implement your own copying algorithm. In the following articles, I plan to tell you more about this feature and provide scripts for working with 1C: Enterprise 8 databases.
Message display and maintenance log
All messages displayed during script execution are redirected to the program maintenance log. This applies to messages output by the print, raiserror commands for T-SQL scripts, as well as messages output to the console by echo commands for other CMD scripts and batch files. And this is great! Because readable and understandable logs are a tremendous time saving, and as a bonus, the text of errors is sent in an email notification.
The presence of backups does not mean that it will be possible to recover data in case of failure - recovery can fail with a variety of reasons. For example, it may happen that the chain of archive copies will be interrupted, and you will not even know about it until you try to recover the data. That is why the best practitioners say that a good DBA should regularly check archived copies, performing a recovery from them. There is no other 100% way. Microsoft also recommends at least once to test all backup copies. There are no tasks for automated recovery in SSMS, and every day there are not many people who want to check backups manually.
QMB has a special task that will consistently restore the entire backup chain for each policy database: Full backup -> Differential backup -> Transaction log backup. Restoration is performed in a temporary test database, which is deleted after checking its integrity.
For example, in our company on a virtual SQL Server there are about 60 small databases, with a total volume of 100 GB. QMB performs a nightly check that all databases can be restored. The check takes about one and a half hours and this gives us a guarantee that all backup copies are checked. If the chain of backups is interrupted, a notification will come up with something like this:
Such errors rarely occur, usually due to carelessness or ignorance of employees. In this case, we simply make an additional full backup.
Tips for those who want to set up a similar check:
With the program, you can keep copies of databases up to date. For example, for 1C developers, you can update the test database every night. For this you need to create a task similar to the built-in “Restore from the arch. copies to a temporary database. In the task, you must specify the base-source backups and the base to which the restoration will be performed. And only then post the task in the night scenario. The figure below shows the task that performs the recovery of backups of the Accounting database to the AccountingCopy database. And if there is no AccountingCopy database on SQL Server, then it will be created automatically.
During the recovery procedure, AccountingCopy will be put into single user mode, with all user connections disconnected.
In the video it was shown how the program configures additional copying of backups to a network or local folder. Copying backups allows you to protect yourself to a certain extent from damage to files, a disk or a server entirely. In cases with a virtual SQL Server, copying backups to a real physical disk allows you to quickly restore one or more databases, without waiting for the entire virtual machine to be restored.
Below, I would like to focus on several features of copying backups using QMB:
You can restore the database in the standard SSMS console. However, in QMB there is an analog with simpler settings:
The “Restore from archive copy” command allows you to:
If you have ever used email alerts in SSMS, you probably know that the messages of the DataBase Mail component contain minimal information. For example, in case of an error, a similar message will be sent:
According to this message, you can understand that an error occurred, but in order to understand which one (and assess its seriousness), you need to look at the server logs. In addition, Database Mail will send a notification every time an error occurs - there is a situation where you will have hundreds of similar messages in your mail.
Unlike Database Mail, QMB sends the first 15 lines of error text in a notification. Usually this is enough to understand the reason and take the necessary action. You can view the full log in the program maintenance log. Sample error message:
There is also a mechanism for preventing the sending of a large number of identical letters, for example, if the error is repeated regularly.
The full version of the program can be downloaded on our website . There is a trial period (30 days after the first registration of SQL Server), after which a license must be purchased for each SQL Server. However, QMB allows for free (with some restrictions) to maintain databases on SQL Express . Also for SQL Express there are inexpensive commercial licenses from 1560 rubles. At the moment, the cost of a Professional license for Russian companies is 7100 rubles. Features and prices can be found here .
Licenses are eternal and not limited in time. If necessary, the license can be easily transferred from one server to another.
Support
From the moment you purchase a commercial license, you can roll out any program updates for 1 year, in order to install subsequent updates, you will need to extend support.
Sometimes I come across the opinion that third-party programs to SQL Server are considered solely as “crutches”. That, allegedly, backups or maintenance configured using such programs are by definition worse than using a regular SQL Server agent. In this case, I have to explain that SQL Server understands only Transact SQL instructions and it doesn’t matter to him exactly who will send this instruction — the SQL Server agent or another program. For example, to check the integrity of the database, it needs to send a DBCC CHECKDB command, and to make a backup - BACKUP DATABASE . Obviously, the result will always be identical, regardless of who sends this command.
I hope that this review was useful to you. Remember that poor performance and sudden stops of SQL Server damage the reputation of the entire IT service, while data loss in most cases has even more serious consequences. If you have a maintenance plan, but there is no certainty about its reliability, then you are sitting on a time bomb - I strongly advise you to prevent a state of emergency in advance, rather than clearing its consequences.
Thank you for your attention, I am ready to answer your questions in the comments.
For database maintenance, Microsoft offers Maintenance Plans in SQL Server Management Studio (SSMS). However, as practice shows, only an experienced DBA can create and configure a high-quality and reliable service plan. I note that reliable service is as automated as possible and does not require regular manual monitoring by the administrator, and also ensures that data can be restored in the event of a failure.
Third-party programs that are available on the market and can make life easier, basically automate the creation of backups. The choice of such programs is very wide. They allow you to make backups compressed and encrypted, on FTP / GoogleDrive / Amazon and so on. Backups here can be compared with shrimps, which Babba spoke about in the picture “Forest Gamp”: “ ... they can be fried, cooked, baked, stewed, you can cook shrimp kebab, Creole shrimps, gumbo shrimps, fried with rice ... ”.
')
However, as has been said, setting up backups is far from everything, so such programs close only a fraction of the questions.
As a result, the “unwitting” DBA has to read articles, deal with SSMS, develop a backup strategy, search for scripts, set up notifications. It takes a lot of time, but there is always something to wipe out ... And I want to live peacefully! So that one did and forgot.
In this article, I would like to review our Quick Maintenance & Backup ( QMB ) program, which will help you to quickly and easily set up database maintenance on Microsoft SQL Server. There is no doubt that for large and high-loaded databases you cannot do without experienced DBA and individual tuning performance, but if you have to deal with many small databases (usually up to 50-80 GB), then this utility will be useful for both beginners and advanced. users.
QMB key features
- Easy and quick setup
- Servicing multiple SQL Server in one program. Supports SQL Server 2000 and later versions, including Express editions.
- 30 built-in tasks with open scripts, including the popular Ola Hallengren scripts:
- backups - full, differential, transaction log
- integrity check
- maintenance of indexes and statistics
- maintenance of system databases
- copying backup files with the ability to specify the retention period
- automated verification of backups through recovery
- keeping database copies up to date - 7 preset maintenance policies for Full and Simple Recovery Models
- Monitoring free disk space on SQL Server
- Custom tasks in the scripting languages ​​Transact SQL, CMD, VBScript, JavaScript, PowerShell and others
- Statistics of database resizing. Calculate the average data growth
- Email Notifications
- Detailed maintenance log
The short video below shows the end-to-end QMB service setup example. The following description complements the video and tells about some features of the program.
Concept: available to beginners, convenient to professionals
On the one hand, we tried to make the program accessible to beginners and implement the most common maintenance scenarios. On the other hand, we wanted to make the program convenient for advanced users and help them set up a variety of scenarios, including combining database operations using Transact SQL with other regulatory procedures for your applications. For example, in QMB, you can make a script that will first load the data into 1C: Enterprise, and then make a backup and perform the rest of the maintenance. The result was a scheduler that provides its own framework for executing T-SQL scripts and batch files (with the ability to store the results of their execution).
Architecture
The program has three components: the client GUI, the QMB Service, and the file base for storing your data. When installing QMB, all three components of the program are installed. Maintenance plans are not created, so the SQL Server Agent service is not required. Read more about architecture here .
Maintenance policy, scenarios and tasks
As mentioned above, QMB does not create maintenance plans on SQL Server. Instead, a Service Policy is created, which is stored in the local storage (file database). In essence, a policy is a grouping of databases with similar properties that are served according to the same rules. The policy contains a list of databases, settings for storing and copying backups. The policy includes one or more maintenance scenarios . The script contains a set of tasks executed sequentially for each (policy included) database. If we draw an analogy with the maintenance plans, then the scenario can be compared with the nested Maintenance Plans (Maintenance Plan).
A task in QMB can be of one of five types:
- T-SQL script
- Creating an archive copy (T-SQL script)
- Restoring an archive copy (dynamic T-SQL script)
- Arbitrary script (not T-SQL)
- Copying backup copies (used in the system task of the same name)
The program has two sets of built-in tasks. The first set of tasks is based on T-SQL scripts, obtained from open sources and created by QMB developers. The second set is based on Ola Halengren’s scripts (a database administrator from Sweden), who developed three popular stored procedures for database maintenance. Ola procedures are installed automatically in the system master database when creating a policy from a template.
Maintenance of large and small databases. Policy Templates
You can create a maintenance policy from a template or manually from scratch. The current version of the program includes 7 templates, which mainly differ:
- Database recovery model. 5 policies with a full recovery model (Full recovery model) and 2 with a simple (Simple recovery model).
- The order of maintenance indexes. For small databases, defragmentation of indexes is performed every night, and updated statistics are updated throughout the day; for large databases, defragmentation of indexes is performed once a week.
- A set of used tasks in scripts. For maintenance, the tasks / scripts of Ola Halengren or QMB are used.
For business databases with daily online data entry ( OLTP database), it is recommended to choose a policy with a Full Recovery Model - for example, for 1C: Enterprise databases, into which data are entered daily. This model allows you to restore the database to the current or arbitrary point in time.
A simple recovery model is recommended for archival and test databases, as well as various storages with rare occasional data downloads.
After you create a policy from a template, you can change any of its settings — scripts and schedule, tasks, and the order of notifications. In the future, the created policy can be copied to other servers registered in the program.
More information about the differences in the templates can be found in the help .
Tasks
As mentioned above, QMB has 5 types of tasks - some of their features are described below.
Script execution
Most system tasks are T-SQL scripts. The script itself can be viewed in the task form:
Script texts (T-SQL, CMD, VBS, PowerShell, and others) may contain markers that will be replaced with the corresponding values ​​before they are executed. For example, the ? DataBaseName? will be replaced with the database name, and the marker ? BackupDirectory? - to the path to the archived copies directory specified in the policy. A complete list of markers can be found in the help .
Optimization of the maintenance window
It happens that in a limited time window it is necessary to fit not only the database maintenance using SQL Server, but also the execution of other routine operations of your application. For example, testing and fixing 1C databases, unloading using the 1C: Enterprise platform, exchanging, etc. Typically, this is done using the Windows Task Scheduler or the 1C: Enterprise Scheduler. However, it is necessary to distribute the procedures in time with a good margin - so that they are guaranteed not to overlap. As a result, tasks may not fit into the existing time window.
With QMB, you can maximize the use of the maintenance window by combining the execution of T-SQL scripts and batch files in VBS, JavaScript, CMD, PowerShell, and others into scripts. Below is a simple example of an alternative backup copying task using the Robocopy utility:
It should be noted that the batch file can be executed both on the machine where the program is installed and on the SQL Server side. This allows you to operate backup files on the SQL Server side. For example, you can write a script that will archive the last backup and upload it to any cloud storage or implement your own copying algorithm. In the following articles, I plan to tell you more about this feature and provide scripts for working with 1C: Enterprise 8 databases.
Message display and maintenance log
All messages displayed during script execution are redirected to the program maintenance log. This applies to messages output by the print, raiserror commands for T-SQL scripts, as well as messages output to the console by echo commands for other CMD scripts and batch files. And this is great! Because readable and understandable logs are a tremendous time saving, and as a bonus, the text of errors is sent in an email notification.
Automated check of backups through recovery
The presence of backups does not mean that it will be possible to recover data in case of failure - recovery can fail with a variety of reasons. For example, it may happen that the chain of archive copies will be interrupted, and you will not even know about it until you try to recover the data. That is why the best practitioners say that a good DBA should regularly check archived copies, performing a recovery from them. There is no other 100% way. Microsoft also recommends at least once to test all backup copies. There are no tasks for automated recovery in SSMS, and every day there are not many people who want to check backups manually.
QMB has a special task that will consistently restore the entire backup chain for each policy database: Full backup -> Differential backup -> Transaction log backup. Restoration is performed in a temporary test database, which is deleted after checking its integrity.
For example, in our company on a virtual SQL Server there are about 60 small databases, with a total volume of 100 GB. QMB performs a nightly check that all databases can be restored. The check takes about one and a half hours and this gives us a guarantee that all backup copies are checked. If the chain of backups is interrupted, a notification will come up with something like this:
1. The 'Restore from Arch. copies to a temporary database with the subsequent integrity check '(database: Buh_Oazis)
Message: 4305, Level: 16, State: 1, Line: 21
The log in this backup set of data begins with the LSN number 5235000000291100001, which cannot yet be applied to the database. An earlier log backup, including the LSN number 5228000000281600001, can be restored.
Message: 3013, Level: 16, State: 1, Line: 21
RESTORE LOG was interrupted with an error.
Message: 50000, Level: 16, State: 1, Line: 119
An error occurred during recovery
Such errors rarely occur, usually due to carelessness or ignorance of employees. In this case, we simply make an additional full backup.
Tips for those who want to set up a similar check:
- The restore operation is resource-intensive, so it should be included in scripts that are executed only during off-hours.
- To create a temporary test database and restore backups, it requires a reserve of disk space equal to at least the largest database in the policy + 10% of its size.
- Recovery of large databases can take considerable time. Do not enable the scan task if you are not sure that the operation will complete in the maintenance window.
- Correctly place the task in the script. Note that the restore is performed at the current time, i.e. at the time of execution of the task. For example, if the backup verification task is placed immediately after creating a full backup, then only the last backup will be tested, since it will be enough to restore the database to the current point in time.
- If there are not enough maintenance windows for checking backups of all policy bases, you can check backups of only certain databases. Or distribute tasks by day of the week. For example, tonight check backups of bases A and B, and tomorrow - bases C and D.
- It is not recommended to backup to a network folder, because during recovery, you have to “drag” backup files over the network, which significantly increases recovery time. It would be more correct to configure the creation of backups on a local disk with daily copying to a network folder.
Automated support for database copies up to date
With the program, you can keep copies of databases up to date. For example, for 1C developers, you can update the test database every night. For this you need to create a task similar to the built-in “Restore from the arch. copies to a temporary database. In the task, you must specify the base-source backups and the base to which the restoration will be performed. And only then post the task in the night scenario. The figure below shows the task that performs the recovery of backups of the Accounting database to the AccountingCopy database. And if there is no AccountingCopy database on SQL Server, then it will be created automatically.
During the recovery procedure, AccountingCopy will be put into single user mode, with all user connections disconnected.
Copying backup files
In the video it was shown how the program configures additional copying of backups to a network or local folder. Copying backups allows you to protect yourself to a certain extent from damage to files, a disk or a server entirely. In cases with a virtual SQL Server, copying backups to a real physical disk allows you to quickly restore one or more databases, without waiting for the entire virtual machine to be restored.
Below, I would like to focus on several features of copying backups using QMB:
- The frequency of copying is determined by the schedule of the script containing the “Copy Archive Copies” task. A task can be placed in one or more scenarios.
- Only new and modified backup files are copied - this reduces the load on the network and allows frequent copying. For example, you can copy every time you create a new transaction log backup.
- For a network folder, you can set the file retention period. Thus, backups can be stored on a local SQL Server disk, for example, for 1 week, and in a network folder for 1 month.
- It is possible to configure copying backups only for selected policy bases.
Database Recovery
You can restore the database in the standard SSMS console. However, in QMB there is an analog with simpler settings:
The “Restore from archive copy” command allows you to:
- Restore the database from backups at a specified point in time with automatic selection of backups chain (if backups were created on the same SQL Server).
- Restore the database from a file with a full backup.
- Restore backups of one database to another database, including a new one.
- Check the integrity of the database after it is restored.
Email Alerts
If you have ever used email alerts in SSMS, you probably know that the messages of the DataBase Mail component contain minimal information. For example, in case of an error, a similar message will be sent:
TASK RUNNING:
"Working databases. AttachedPlan_1" began on 05/19/2015 5:00:00 PM
DURATION:
0 hours, 0 min., 5 sec.
STATE:
Mistake
MESSAGES:
Failed to complete the task. The start of the task was produced by Schedule 9 (MaintenancePlan). The last step performed was step 1 (Transaction Log Backup).
According to this message, you can understand that an error occurred, but in order to understand which one (and assess its seriousness), you need to look at the server logs. In addition, Database Mail will send a notification every time an error occurs - there is a situation where you will have hundreds of similar messages in your mail.
Unlike Database Mail, QMB sends the first 15 lines of error text in a notification. Usually this is enough to understand the reason and take the necessary action. You can view the full log in the program maintenance log. Sample error message:
The script 'Resource-intensive tasks for medium-sized OLTP databases (every night)' was executed with errors on the server 'Srv05'.
Scenario start: 06.06.2015 1:00
Completion of work: 06/06/2015 1:29
Duration: 00:29:28
Total tasks: 7
Tasks completed: 7
With errors: 1
1. The 'Restore from Arch. copies to a temporary database with subsequent integrity check '(database: IPGor)
Message: 4305, Level: 16, State: 1, Line: 21
The log in this backup set of data begins with the LSN number 5235000000291100001, which cannot yet be applied to the database. An earlier log backup, including the LSN number 5228000000281600001, can be restored.
Message: 3013, Level: 16, State: 1, Line: 21
RESTORE LOG was interrupted with an error.
Message: 50000, Level: 16, State: 1, Line: 119
An error occurred during recovery
There is also a mechanism for preventing the sending of a large number of identical letters, for example, if the error is repeated regularly.
Licensing policy and cost
The full version of the program can be downloaded on our website . There is a trial period (30 days after the first registration of SQL Server), after which a license must be purchased for each SQL Server. However, QMB allows for free (with some restrictions) to maintain databases on SQL Express . Also for SQL Express there are inexpensive commercial licenses from 1560 rubles. At the moment, the cost of a Professional license for Russian companies is 7100 rubles. Features and prices can be found here .
Licenses are eternal and not limited in time. If necessary, the license can be easily transferred from one server to another.
Support
From the moment you purchase a commercial license, you can roll out any program updates for 1 year, in order to install subsequent updates, you will need to extend support.
Conclusion
Sometimes I come across the opinion that third-party programs to SQL Server are considered solely as “crutches”. That, allegedly, backups or maintenance configured using such programs are by definition worse than using a regular SQL Server agent. In this case, I have to explain that SQL Server understands only Transact SQL instructions and it doesn’t matter to him exactly who will send this instruction — the SQL Server agent or another program. For example, to check the integrity of the database, it needs to send a DBCC CHECKDB command, and to make a backup - BACKUP DATABASE . Obviously, the result will always be identical, regardless of who sends this command.
I hope that this review was useful to you. Remember that poor performance and sudden stops of SQL Server damage the reputation of the entire IT service, while data loss in most cases has even more serious consequences. If you have a maintenance plan, but there is no certainty about its reliability, then you are sitting on a time bomb - I strongly advise you to prevent a state of emergency in advance, rather than clearing its consequences.
Thank you for your attention, I am ready to answer your questions in the comments.
Source: https://habr.com/ru/post/266033/
All Articles