We optimize workflow
Good day. I decided to share my experience in organizing the workflow of developing web projects and not only the web. I'll tell you my vision of the most convenient way to use the bundle: bugtraker + git + ci + deploy.
Increasingly, I encounter the fact that such optimization will be useful to a wide range of people. Perhaps in the proposed version there is nothing new, but, for me personally, it was, one might say, a revelation. If someone other than me wants to implement something similar on my project, I will assume that the article was not written in vain.
Somehow I was lucky to work in a startup project. I can only say that it was a forex broker. Our team needed to develop a website, a personal account and sishny modules for the trading platform. The team is small, the demands are exaggerated, in general, a typical startup.
')
Having started developing a project, I decided to set several goals for myself:
Surely, many have already encountered such a problem when, without looking at the history of the game, it is difficult to say which of the tasks is cast on production. On many projects, developers all commit in one branch, and before the release they try to finish, in emergency mode. The tasks are partially already in the repositories and there is simply no possibility to transfer them to the next release.
Another common problem is the urgent fixes that should be made yesterday. Sometimes, the programmer is forced to climb on the server himself and make edits for live. Unfortunately, this often causes immortal bugs that they simply forgot to add to the repository. Rest assured, the next release will reappear.
Automation of routine tasks is necessary for the release of hands, in conditions of a limited team and to assist in self-control.
After a lengthy search for optimal solutions, I settled on a rather interesting bunch. First of all, we, of course, have Git, with master and staging branches. Master contains what is now in production. And the staging branch is the future of the master branch. It is worth remembering that staging is not a test branch, as is customary for many, namely the future master and it is necessary to treat it appropriately. We used Bitbucket as a git.
The next thing we need is a convenient bugtracker. We used Youtrack, there is a comfortable work with sprints and cards in them.
To test the project with automatic tests, you need any CI (Continuous Integration) platform. In our case, it was TeamCity. Before use, you should configure 3 separate stages: only quick tests, long tests and no tests at all. Why so, I will explain below. And finally, we arm ourselves with a deployment tool, for example capistrano.
Here is a normal working day with such a system. The programmer, having come to work and brewing himself fresh coffee, starts to get acquainted with a new task. Having found out all the details and having thought over the future decision, he changes the status of the task, simultaneously notifying everyone wishing to know that he has begun work by automatically writing from the bugtracker. Then he goes to the gita, updates the staging branch to the current state and creates a new one with the task number.
At this stage, there is still no magic and everything is quite familiar. In the process of working on a task, the developer can periodically leave comites with comments. Each comment should indicate which or which tasks it belongs to. This is quite important, since on the server of the gita, at each commit, information about it is sent to a small script. He parses the comment and conveniently posts a link to the commit, immediately to the bugtracker. Thanks to this, you can see the history of code changes regarding this particular task. You can specify several numbers in the committee for cases when the task is already closed, and a bug was found in it that could be fixed by solving another task. And in order not to search for a long time where it was fixed, you can specify the number of the closed task in the fix commit, so the message will appear in both tasks. Such a transfer of commits to the bugtracker allows you to get rid of the permanent unsubscribing to the task of the work done, and the manager is always up to date with the process. But, sometimes it is more convenient to make several comets with a short description and then add a more detailed comment to the task about what was done in them.
This is not the end of automation. The CI constantly monitors the git repository and starts the execution process for each commite in the threads. On the status of these tests, you can notify the author by mail, jabber or general dashboard. It is important to set up so that at this stage only quick unit tests are launched, otherwise, with a large number of commits, you can score a CI line to the string.
In case of successful passing of tests, CI goes to the automatic deployment of the project, to the test server, in a separate folder, with the number of the task in the name. This server is nginx and configured so that the folder name was the name of the subdomain on the test domain. Thus, the developer can check his creation on an external server. This is useful when you need to check the work, for example with payment systems in which the link is transmitted to notify about the change of the status of payment.
When our hard-working programmer decided that he had completed the task, it is enough for him to simply translate her status in testing into a bugtracker and proceed to the next one. After changing the status, the tester receives a notification that this task can be checked. First of all, he should check the status of the tests from the last commit, and in case of failure, immediately return the task to the author. If all the tests have passed, then accordingly, he already has a domain on which he can safely check them. After checking, the tester adds a comment to the task that everything is OK or describes the problems found and returns to the programmer changing the task status. If there are problems, then we repeat everything, and if not, then we merge the branch of the task in staging. When a commit occurs in staging, CI launches already longer unit tests and functional tests on a real base.
By the way, about the real database ... once a week, a small script running on the crown is used to make a clone of the real database on the staging server, of course, with the substitution of emails and clearing users' personal data. At the end of all tests, if successful, CI will deploy to the test server so that the tester can once again check that nothing has broken after the merge.
At the end of the sprint, we just have staging in the master and CI, without any tests, deploying the project in production. Since staging branches from the end of master, and no task is merged into master bypassing staging, there are never any problems with merging the staging branch to master. If everything worked on staging, then it will work on master too. Various incidents connected with iron or real base are solved individually. After that, it is necessary to bring staging into compliance with master, in order to see a similar problem in the future before deploying to production.
First of all, achieved goals:
Secondly, it is the convenience of work. Almost all control and interaction occurs through git. And thanks to the integration of the Youtrack bugtracker with jabber, you don’t even need to go to the web interface to manage your cards. With proper use of chat bots (for example, hubot), you can completely leave the CI interface, learning the statuses via jabber.
The system managed to be implemented on the sish modules, with the exception of deployment, it was carried out not on a web server, but on a Windows server, with the test server installed, and restarting it to activate the modules.
If anyone has more ideas for improving the system and optimizing the workflow, I will be very happy to listen. If possible, try to introduce and add to the article conclusions and experience.
Increasingly, I encounter the fact that such optimization will be useful to a wide range of people. Perhaps in the proposed version there is nothing new, but, for me personally, it was, one might say, a revelation. If someone other than me wants to implement something similar on my project, I will assume that the article was not written in vain.
Somehow I was lucky to work in a startup project. I can only say that it was a forex broker. Our team needed to develop a website, a personal account and sishny modules for the trading platform. The team is small, the demands are exaggerated, in general, a typical startup.
')
Why change something?
Having started developing a project, I decided to set several goals for myself:
- confidence in the fact that it is now working on production;
- if possible, eliminate the human factor;
- automate everything possible.
Surely, many have already encountered such a problem when, without looking at the history of the game, it is difficult to say which of the tasks is cast on production. On many projects, developers all commit in one branch, and before the release they try to finish, in emergency mode. The tasks are partially already in the repositories and there is simply no possibility to transfer them to the next release.
Another common problem is the urgent fixes that should be made yesterday. Sometimes, the programmer is forced to climb on the server himself and make edits for live. Unfortunately, this often causes immortal bugs that they simply forgot to add to the repository. Rest assured, the next release will reappear.
Automation of routine tasks is necessary for the release of hands, in conditions of a limited team and to assist in self-control.
Ingredients
After a lengthy search for optimal solutions, I settled on a rather interesting bunch. First of all, we, of course, have Git, with master and staging branches. Master contains what is now in production. And the staging branch is the future of the master branch. It is worth remembering that staging is not a test branch, as is customary for many, namely the future master and it is necessary to treat it appropriately. We used Bitbucket as a git.
The next thing we need is a convenient bugtracker. We used Youtrack, there is a comfortable work with sprints and cards in them.
To test the project with automatic tests, you need any CI (Continuous Integration) platform. In our case, it was TeamCity. Before use, you should configure 3 separate stages: only quick tests, long tests and no tests at all. Why so, I will explain below. And finally, we arm ourselves with a deployment tool, for example capistrano.
How it works
Here is a normal working day with such a system. The programmer, having come to work and brewing himself fresh coffee, starts to get acquainted with a new task. Having found out all the details and having thought over the future decision, he changes the status of the task, simultaneously notifying everyone wishing to know that he has begun work by automatically writing from the bugtracker. Then he goes to the gita, updates the staging branch to the current state and creates a new one with the task number.
At this stage, there is still no magic and everything is quite familiar. In the process of working on a task, the developer can periodically leave comites with comments. Each comment should indicate which or which tasks it belongs to. This is quite important, since on the server of the gita, at each commit, information about it is sent to a small script. He parses the comment and conveniently posts a link to the commit, immediately to the bugtracker. Thanks to this, you can see the history of code changes regarding this particular task. You can specify several numbers in the committee for cases when the task is already closed, and a bug was found in it that could be fixed by solving another task. And in order not to search for a long time where it was fixed, you can specify the number of the closed task in the fix commit, so the message will appear in both tasks. Such a transfer of commits to the bugtracker allows you to get rid of the permanent unsubscribing to the task of the work done, and the manager is always up to date with the process. But, sometimes it is more convenient to make several comets with a short description and then add a more detailed comment to the task about what was done in them.
This is not the end of automation. The CI constantly monitors the git repository and starts the execution process for each commite in the threads. On the status of these tests, you can notify the author by mail, jabber or general dashboard. It is important to set up so that at this stage only quick unit tests are launched, otherwise, with a large number of commits, you can score a CI line to the string.
In case of successful passing of tests, CI goes to the automatic deployment of the project, to the test server, in a separate folder, with the number of the task in the name. This server is nginx and configured so that the folder name was the name of the subdomain on the test domain. Thus, the developer can check his creation on an external server. This is useful when you need to check the work, for example with payment systems in which the link is transmitted to notify about the change of the status of payment.
When our hard-working programmer decided that he had completed the task, it is enough for him to simply translate her status in testing into a bugtracker and proceed to the next one. After changing the status, the tester receives a notification that this task can be checked. First of all, he should check the status of the tests from the last commit, and in case of failure, immediately return the task to the author. If all the tests have passed, then accordingly, he already has a domain on which he can safely check them. After checking, the tester adds a comment to the task that everything is OK or describes the problems found and returns to the programmer changing the task status. If there are problems, then we repeat everything, and if not, then we merge the branch of the task in staging. When a commit occurs in staging, CI launches already longer unit tests and functional tests on a real base.
By the way, about the real database ... once a week, a small script running on the crown is used to make a clone of the real database on the staging server, of course, with the substitution of emails and clearing users' personal data. At the end of all tests, if successful, CI will deploy to the test server so that the tester can once again check that nothing has broken after the merge.
At the end of the sprint, we just have staging in the master and CI, without any tests, deploying the project in production. Since staging branches from the end of master, and no task is merged into master bypassing staging, there are never any problems with merging the staging branch to master. If everything worked on staging, then it will work on master too. Various incidents connected with iron or real base are solved individually. After that, it is necessary to bring staging into compliance with master, in order to see a similar problem in the future before deploying to production.
What do we have in the end?
First of all, achieved goals:
- In production, we always have only completed and proven tasks.
- None of the developers have direct access to production. Depla only occurs through merge branches.
- Whenever possible, routine processes associated with task management, test launch, test space deployment are automated. Each team has no more than 2 working tools (the developer has a bugtracker and git, the tester has a bugtracker and CI).
Secondly, it is the convenience of work. Almost all control and interaction occurs through git. And thanks to the integration of the Youtrack bugtracker with jabber, you don’t even need to go to the web interface to manage your cards. With proper use of chat bots (for example, hubot), you can completely leave the CI interface, learning the statuses via jabber.
The system managed to be implemented on the sish modules, with the exception of deployment, it was carried out not on a web server, but on a Windows server, with the test server installed, and restarting it to activate the modules.
If anyone has more ideas for improving the system and optimizing the workflow, I will be very happy to listen. If possible, try to introduce and add to the article conclusions and experience.
Source: https://habr.com/ru/post/245349/
All Articles