Development methodology on 1C-Bitrix - the experience of a fool
Not so long ago, our company received a fairly large online store on 1C-Bitrix for maintenance and revision. The project a couple of months, as was launched into commercial operation, but it had a number of serious problems. In addition, the customer planned to complete the tasks of finalizing the new functionality as soon as possible. I was assigned the task of organizing effective work on the project with minimal downtime and maximum customer satisfaction.
Next, I will describe the main problems that I had to deal with in this, and several similar projects, my tested solutions and results of work.
')
Software development in general, and web projects in particular, I have been doing for about 8 years. During this time I came across projects of varying complexity and, at first glance, the task did not seem to me too difficult. When implementing projects in our company, as a rule, the SCRUM methodology is applied. From her I began to make a start.
First of all, I got access to the source code of the project. Superficially analyzed. I agreed with the customer a list of primary tasks. Made a development plan for 3 developers and, as Gagarin said, let's go!
As is usually the case, everything is to blame for everything except the customer. The designer made a layout that weighs too much, the hoster provided a server that works slowly, the developers made a site that is constantly buggy and broken, the managers completed some tasks that we did not ask to perform, after switching from the old version of the site to 1C- Bitrix experienced a sharp decline in search traffic, etc. The situation is not unambiguous. On the one hand, the main responsibility of course should rest with the company developer. It was necessary to convey to the customer the consequences of all actions with the site and prepare for the result. In carrying out the work, propose a holistic architecture of the future system and a development plan, which should be followed before the completion of the main stages. Conduct thorough functional testing and pass the work. On the other hand, it is not rare that I encounter a situation where a customer knows everything better, because his mother used to draw, and therefore he is the best designer, and his 7-year-old son is well versed in SEO optimization, because he spends all the time at the computer, playing in the GTA.
Who is to blame, who is not right to judge us. In this case, the previous contractor was a fairly well-known reliable company and I can’t say anything bad about their development. And the customer, objectively, worries about his product and tries to improve it. As it was, I don’t know, for myself I found an explanation in the insufficient amount of analytics provided by the contractor to the customer.
As a result:
I see only one solution to the problem: gradually systematically clean out all the modules of the site in turn to bring the product to the desired state. Some mistakes were made in separate tasks and executed immediately, and we managed to fix something in parallel with the development of a new functional. The result - with each update, after the operational cleaning of the crawled out bugs, the site is getting better and more stable.
In accordance with the licensing policy of 1C-Bitrix, each site license allows you to use 2 copies of the system. One, as a production site, the second - for development. The problem is that the development is carried out by several, in my case, three, developers continuously. In the case of the classical development, everything is simple. Each developer deals with its own module. Then functional testing of each module is carried out, all the improvements are merged into the repository of some version control system, then it is tested all together (integration testing). If the result is normal, the test version is presented to the customer. After accepting the test version, the production server is updated. In accordance with the SCRUM methodology, I determined that I will upload new versions to the combat site once a week. Accordingly, there is 3-4 days for the main development. 1 day for testing and correction of errors and half a day for updating the combat server. The dates of course fluctuate, but the “release every Thursday” rules tried to be strictly adhered to.
The first one I encountered, in 1C-Bitrix there are situations in which the same file is simultaneously involved in different functionalities at different ends of the site. The simplest and most obvious solution is to use a version control system, in my case, SVN, which I use in all other projects. But to use version control, it is necessary that each developer has his own version of the code, which he rules and then merges into a common repository.
How to be with the license? Turned to the technical support of 1C-Bitrix. Received an offer to purchase additional. licenses for development. To say the least, I was not happy, but I received no other offers. Found a way out quickly. Decided to use NFR keys. Fortunately, the partner status allows. As a result, I created 5 online store istations:
Over time, went even further. There was also a separate installation for the tester. It turned out that with my luck the customer always comes to the test server at the moment when something is updated there. A lot of unnecessary tasks arise in the bugtrack, which are already completed, and the customer has the impression that we are not doing our job well.
Now I use the following scheme:
From the obvious disadvantages of this approach, I want to highlight the low level of customer involvement in the development. The result is visible to the customer only at the final stage. This approach is applicable if you have a good analyst who rarely makes mistakes and is in constant contact with the customer. Otherwise, many tasks will have to be redone from scratch.
As the construction of the scheme encountered another problem. The project occupies about 80 GB of disk space. Without the cache and temporary files - about 60. I tried to remove pictures and videos from the version control at the beginning - it didn't work. Information on the site is constantly changing. Test need to actual data. The first commit of the site to my repository went in chunks for more than 2 days. The first checkout to the development folder takes several hours (SVN server on the local development network). If, God forbid, accidentally make a full update of the project folder - you can quit smoking, dine, play ping-pong or curling. Comit only selected files or folders passes quickly enough. Solution: completed the task - to commit a dozen of changed files at once.
The problem is the most important, complex and, to the end, not solved. After all, if the remaining problems relate to the internal kitchen of the project, then the site’s reputation and income of the customer, and, consequently, my income, depend on the work of the site.
Here, Murphy's laws work fine:
Of course, I am exaggerating, but in every joke there is some joke. The minimum load on the site from 4 to 6 in the morning. Update at this time would certainly be better, but I really do not want to.
In the case of most web applications, there is a clear structure for splitting the application into layers and updating the site can be divided into 2 parts:
In the case of 1C-Bitrix, everything is a bit more complicated. First, there are a lot of files. I have over a million of them in the project. A regular update from the repository takes at least 20-30 minutes. You can of course update only the modified files, but then the whole point of the repository is lost. Secondly, and it is much more sad, often with an update you have to make manual changes and settings through the admin panel. And it is always slow, you need to remember all the changes that need to be made, it is likely to accidentally make a mistake. You can, of course, write a SQL script that will itself make all the necessary changes to the database. In the simplest cases, of course, we do so. But in most cases, writing and debugging such a script takes more time than the development itself and much more time than performing all actions manually with subsequent verification.
A good solution to the problem has not yet found. Now we update the settings in the database manually. To minimize errors, a checklist is compiled with a list of what needs to be done when updating. We try to make the update as carefully and accurately as possible. After updating the whole team, we check the main functionality of the server production and conduct additional testing. The number of errors has been minimized, but it hasn’t been possible to get rid of bugs and downtime during the upgrade.
The second, with what faced, this collaboration with the customer. Since The project is large, about 30 people are constantly working on it. Content managers, sales managers, SEO optimizers, marketers and many more. Naturally, everyone makes some changes to the site pages and module settings. The first decision was to take the rights from the customer to make changes to the program code of the site. The decision is absolutely correct, but it only got worse. If earlier the customer assumed that he could also go to the site and accidentally break something, now all the bumps began to pour on us only. With that. Even if the content manager has crookedly edited the text on the page and has not closed a tag, the developer is still to blame. The solution was found quite simple. In the marketplace there is a free version control module for pages. It didn’t remove the problem, anyway, from time to time, somebody will work for something, but now it’s possible to see who’s changed, what’s changed and why it’s broken at any moment. The result, of course, is not ice, but it saves me a lot of nerves.
Additionally decided before each update of the test server, we take on it a copy of the combat. It also takes a lot of time. Archive project, transfer to another server, unzip. All it takes is a few hours. But, new improvements are tested almost in combat conditions. If the settings of the test and production server are identical, the difference in work will be minimal and the number of errors will be significantly reduced. Experience has shown that in a week a production server can change so much that a part of the new functionality that works without problems on a copy of a week ago can not work at all on a fresh copy.
The problem relates not so much to 1C-Bitrix, as to the refinement and support of working projects. Often, the customer has a desire to make some trifle, but urgently and immediately on the combat site. The result is always the same - nothing good comes of it. At best, this revision will simply be forgotten with the next release, at worst - the server will simply fall and it will take several hours to restore it from backup.
The solution found only one thing - never to be led by the customer at the expense of reliability and safety. No matter how the customer requests, the developer will always be guilty. As my former boss told me: “I didn’t ask you to do badly”.
And since I have touched on the topic of backups, I want to note. Backup means 1C-Bitrik is nice and convenient, but very slowly. If you urgently need to restore 1-2 files or several values ​​in the database, you have to wait until all 60 GB are unzipped. Here, the following scheme seems to be the most effective:
Thanks to everyone who read to the end. I hope my experience will be useful to you. I would be happy for suggestions on more successful ways to solve problems voiced in the comments or in a personal. Now I have tried to voice the main problems of finalizing and supporting already launched projects with high reliability requirements. If the material turns out to be interesting, I plan to write a sequel about the features of the 1C-Bitrix architecture, which distinguish the development of the Bitrix site from the development of other web projects.
Next, I will describe the main problems that I had to deal with in this, and several similar projects, my tested solutions and results of work.
Initial data:
- There is an online store on 1C-Bitrix
- Project a few years, but only a few months ago, the site was transferred to 1C-Bitrix
- Attendance of 10-15 thousand people per day
- The store catalog contains about 12,000 items of goods
- Downtime and site outages are not allowed.
- The project for six months was developed by another company:
- There are TK, about 100 sheets, corresponding to the work done by about 40%
- No project documentation
- Lack of understanding why previous developers used such architectural solutions.
')
Software development in general, and web projects in particular, I have been doing for about 8 years. During this time I came across projects of varying complexity and, at first glance, the task did not seem to me too difficult. When implementing projects in our company, as a rule, the SCRUM methodology is applied. From her I began to make a start.
First of all, I got access to the source code of the project. Superficially analyzed. I agreed with the customer a list of primary tasks. Made a development plan for 3 developers and, as Gagarin said, let's go!
Problem number 1 - the developers are to blame
As is usually the case, everything is to blame for everything except the customer. The designer made a layout that weighs too much, the hoster provided a server that works slowly, the developers made a site that is constantly buggy and broken, the managers completed some tasks that we did not ask to perform, after switching from the old version of the site to 1C- Bitrix experienced a sharp decline in search traffic, etc. The situation is not unambiguous. On the one hand, the main responsibility of course should rest with the company developer. It was necessary to convey to the customer the consequences of all actions with the site and prepare for the result. In carrying out the work, propose a holistic architecture of the future system and a development plan, which should be followed before the completion of the main stages. Conduct thorough functional testing and pass the work. On the other hand, it is not rare that I encounter a situation where a customer knows everything better, because his mother used to draw, and therefore he is the best designer, and his 7-year-old son is well versed in SEO optimization, because he spends all the time at the computer, playing in the GTA.
Who is to blame, who is not right to judge us. In this case, the previous contractor was a fairly well-known reliable company and I can’t say anything bad about their development. And the customer, objectively, worries about his product and tries to improve it. As it was, I don’t know, for myself I found an explanation in the insufficient amount of analytics provided by the contractor to the customer.
As a result:
- The project is not brought to a logical conclusion. Many tasks are thrown mid way.
- The project is not documented. The work of the functional is not obvious. When developing a new functional, it turns out that the functional stopped working, which previously worked and the existence of which the new developer did not even suspect
- Part of the code written by the previous executor has to be rewritten from scratch.
- The designed architecture of the project in the first weeks / months of work to a new contractor is not clear. Refinement of the functionality of a single module leads to a loss of functionality of a functional module not related to it.
- The customer is nervous, the performer is nervous, visitors are not happy, attendance decreases, sales fall
I see only one solution to the problem: gradually systematically clean out all the modules of the site in turn to bring the product to the desired state. Some mistakes were made in separate tasks and executed immediately, and we managed to fix something in parallel with the development of a new functional. The result - with each update, after the operational cleaning of the crawled out bugs, the site is getting better and more stable.
Problem number 2 - parallel development.
In accordance with the licensing policy of 1C-Bitrix, each site license allows you to use 2 copies of the system. One, as a production site, the second - for development. The problem is that the development is carried out by several, in my case, three, developers continuously. In the case of the classical development, everything is simple. Each developer deals with its own module. Then functional testing of each module is carried out, all the improvements are merged into the repository of some version control system, then it is tested all together (integration testing). If the result is normal, the test version is presented to the customer. After accepting the test version, the production server is updated. In accordance with the SCRUM methodology, I determined that I will upload new versions to the combat site once a week. Accordingly, there is 3-4 days for the main development. 1 day for testing and correction of errors and half a day for updating the combat server. The dates of course fluctuate, but the “release every Thursday” rules tried to be strictly adhered to.
The first one I encountered, in 1C-Bitrix there are situations in which the same file is simultaneously involved in different functionalities at different ends of the site. The simplest and most obvious solution is to use a version control system, in my case, SVN, which I use in all other projects. But to use version control, it is necessary that each developer has his own version of the code, which he rules and then merges into a common repository.
How to be with the license? Turned to the technical support of 1C-Bitrix. Received an offer to purchase additional. licenses for development. To say the least, I was not happy, but I received no other offers. Found a way out quickly. Decided to use NFR keys. Fortunately, the partner status allows. As a result, I created 5 online store istations:
- Production server
- Test server
- 3 development servers (one per developer)
Over time, went even further. There was also a separate installation for the tester. It turned out that with my luck the customer always comes to the test server at the moment when something is updated there. A lot of unnecessary tasks arise in the bugtrack, which are already completed, and the customer has the impression that we are not doing our job well.
Now I use the following scheme:
- Each developer uses only his local copy for work.
- At the agreed time, all completed revisions merge into a common thread in the repository.
- QA picks up a "sudzhennuyu" version for testing
- After testing and correcting errors, the demo server for the customer is updated.
- After verification and acceptance by the customer, the modifications are transferred to the combat server.
From the obvious disadvantages of this approach, I want to highlight the low level of customer involvement in the development. The result is visible to the customer only at the final stage. This approach is applicable if you have a good analyst who rarely makes mistakes and is in constant contact with the customer. Otherwise, many tasks will have to be redone from scratch.
As the construction of the scheme encountered another problem. The project occupies about 80 GB of disk space. Without the cache and temporary files - about 60. I tried to remove pictures and videos from the version control at the beginning - it didn't work. Information on the site is constantly changing. Test need to actual data. The first commit of the site to my repository went in chunks for more than 2 days. The first checkout to the development folder takes several hours (SVN server on the local development network). If, God forbid, accidentally make a full update of the project folder - you can quit smoking, dine, play ping-pong or curling. Comit only selected files or folders passes quickly enough. Solution: completed the task - to commit a dozen of changed files at once.
Problem number 3 - update the production server and collaboration with the customer
The problem is the most important, complex and, to the end, not solved. After all, if the remaining problems relate to the internal kitchen of the project, then the site’s reputation and income of the customer, and, consequently, my income, depend on the work of the site.
Here, Murphy's laws work fine:
- If something does not work well on a test server, it will break on the combat server.
- If something works fine on a test server, it will still break on the combat server anyway.
- If the bug on the site exists for only 5 seconds, it will be surely found by one of the visitors and will necessarily write about it in reviews or in the feedback form.
- If the site does not work for 1 minute during the update, then it is at this minute that the owner of the company will show it to his friend or competitor (and this is despite the timing of the update and the update procedure).
Of course, I am exaggerating, but in every joke there is some joke. The minimum load on the site from 4 to 6 in the morning. Update at this time would certainly be better, but I really do not want to.
In the case of most web applications, there is a clear structure for splitting the application into layers and updating the site can be divided into 2 parts:
- Update code
- Database update using SQL scripts
In the case of 1C-Bitrix, everything is a bit more complicated. First, there are a lot of files. I have over a million of them in the project. A regular update from the repository takes at least 20-30 minutes. You can of course update only the modified files, but then the whole point of the repository is lost. Secondly, and it is much more sad, often with an update you have to make manual changes and settings through the admin panel. And it is always slow, you need to remember all the changes that need to be made, it is likely to accidentally make a mistake. You can, of course, write a SQL script that will itself make all the necessary changes to the database. In the simplest cases, of course, we do so. But in most cases, writing and debugging such a script takes more time than the development itself and much more time than performing all actions manually with subsequent verification.
A good solution to the problem has not yet found. Now we update the settings in the database manually. To minimize errors, a checklist is compiled with a list of what needs to be done when updating. We try to make the update as carefully and accurately as possible. After updating the whole team, we check the main functionality of the server production and conduct additional testing. The number of errors has been minimized, but it hasn’t been possible to get rid of bugs and downtime during the upgrade.
The second, with what faced, this collaboration with the customer. Since The project is large, about 30 people are constantly working on it. Content managers, sales managers, SEO optimizers, marketers and many more. Naturally, everyone makes some changes to the site pages and module settings. The first decision was to take the rights from the customer to make changes to the program code of the site. The decision is absolutely correct, but it only got worse. If earlier the customer assumed that he could also go to the site and accidentally break something, now all the bumps began to pour on us only. With that. Even if the content manager has crookedly edited the text on the page and has not closed a tag, the developer is still to blame. The solution was found quite simple. In the marketplace there is a free version control module for pages. It didn’t remove the problem, anyway, from time to time, somebody will work for something, but now it’s possible to see who’s changed, what’s changed and why it’s broken at any moment. The result, of course, is not ice, but it saves me a lot of nerves.
Additionally decided before each update of the test server, we take on it a copy of the combat. It also takes a lot of time. Archive project, transfer to another server, unzip. All it takes is a few hours. But, new improvements are tested almost in combat conditions. If the settings of the test and production server are identical, the difference in work will be minimal and the number of errors will be significantly reduced. Experience has shown that in a week a production server can change so much that a part of the new functionality that works without problems on a copy of a week ago can not work at all on a fresh copy.
Problem number 4 - “make me urgent, it's a task for 5 minutes”
The problem relates not so much to 1C-Bitrix, as to the refinement and support of working projects. Often, the customer has a desire to make some trifle, but urgently and immediately on the combat site. The result is always the same - nothing good comes of it. At best, this revision will simply be forgotten with the next release, at worst - the server will simply fall and it will take several hours to restore it from backup.
The solution found only one thing - never to be led by the customer at the expense of reliability and safety. No matter how the customer requests, the developer will always be guilty. As my former boss told me: “I didn’t ask you to do badly”.
And since I have touched on the topic of backups, I want to note. Backup means 1C-Bitrik is nice and convenient, but very slowly. If you urgently need to restore 1-2 files or several values ​​in the database, you have to wait until all 60 GB are unzipped. Here, the following scheme seems to be the most effective:
- There should be a daily backup of files and databases in the form of an archive to an external data source.
- Always make backup just before updating in one of 2 options:
- Light option - Copy the entire project folder to the next folder on the server. We save the database as a dump into a separate file. We do not archive anything. In case you need to restore some value in the database or one of the files, everything will be at hand and easily accessible
- The strong option is the same as the previous one, just copying the database to another MySQL database. In the case of a complete crash, this will allow fixing the root folder of the site in the hosts file in 1-2 minutes and the project will start working from the next folder with a copy of the database.
Conclusion
Thanks to everyone who read to the end. I hope my experience will be useful to you. I would be happy for suggestions on more successful ways to solve problems voiced in the comments or in a personal. Now I have tried to voice the main problems of finalizing and supporting already launched projects with high reliability requirements. If the material turns out to be interesting, I plan to write a sequel about the features of the 1C-Bitrix architecture, which distinguish the development of the Bitrix site from the development of other web projects.
Source: https://habr.com/ru/post/189630/
All Articles