We roll into battle

General Staff Plan

• Each developer's application should use personal databases and file storages. Use a common "trash" - evil, causing serious errors.
• The development of autotests is not always justified : their implementation should not take more than 20% of the working time. Develop through testing - the extreme degree of waste.
• The more participants in the process, the more often it is necessary to merge and check the collective result. Continuous integration can help, but maintaining it can be expensive - you need to balance between automation and cost.

Hard to learn

• The scenario of the introduction of the candidate should be as close as possible to the scenario of the introduction of the release. If in the final release it is planned to update the database structure, migrate data, add third-party libraries - all this must be done in the sequence in which the release is supposed to be implemented.
• The test platform conditions should be as close as possible to the combat conditions. If the application uses a cluster of 5 memcached servers, prepare> 1 server on the test platform. If the application uses 1 DBMS master server in RW mode and 2 slave servers in RO mode, reconstruct the complete replication scheme. To save on equipment for testing distributed systems will help free VMWare Server (or any other equivalent, except Virtuozzo). Versions of system libraries, OS kernels, etc. should also coincide with the versions of the corresponding components of the combat platform.
• Load testing is a floating acceptance testing problem. It is impossible to completely recreate the release conditions: we do not know what the load will be exactly, we can only assume. Yes, and create a complete copy of the combat platform is expensive. For adequate testing, bottlenecks are usually identified in the application, their maximum stability values ​​are measured for them and the results are approximated to the expected loads.
• Testing should be regression , but you need to balance between quality and cost. If only cosmetic changes are made to the interface, it makes no sense to run the test plan completely.
• Do not forget to roll back the application , return to the previous version should be carefully worked out.

I once ruled the development of a portal about advertising. The team then had to roll out ~ 20 candidates before the release was introduced. The release was implemented from the 10th time. The final test report was successful at ~ 70%. Not the most successful project in my life.

To arms

1. We introduce new versions of platform components . We update libraries, services, etc., while keeping the tool for rolling back to the old versions.
2. We implement a DB and file storage . If the previous version of the application is working in combat, you need to prepare in advance the old release to support the new storage structure. The story is this:
   • We add to the previous release support for a new storage scheme
   • we introduce the changed previous release
   • we introduce a new storage structure and migrate data (the previous release should work in the old logic with the new storage)
   • introducing a new release
3. We implement the application .
4. Testing the application .
5. Roll back (if necessary).

1. Roll back the application .
2. Roll back the database and file storage (if necessary).
3. Roll back platform components (if necessary).

Application to battle

 / var / www / myproject /
	 / etc /
	 / lib /
	 / classes /
	 / images /
	 / templates /
	 index.php
 / var / www / userdata /
	 / images /
	 / video /

 / var / www / myproject /
	 / etc /
	 / lib /
	 / classes /
	 / images /
	 / templates /
	 / userdata /
		 / images /
		 / video /
	 index.php

 user @ stable: ~ / apps / myproject $ ls -la
 drwxr-xr-x 12 user www-data 4096 Oct 29 11:38.
 drwxr-xr-x 5 user www-data 4096 May 13 23:22 ..
 drwxr-xr-x 8 user www-data 4096 Oct 22 17:11 rel_0.5.1
 drwxr-xr-x 8 user www-data 4096 Oct 27 13:17 rel_0.5.1.1
 drwxr-xr-x 8 user www-data 4096 Oct 28 02:07 rel_0.5.1.2
 drwxr-xr-x 8 user www-data 4096 Oct 29 11:38 rel_0.5.2
 user @ stable: ~ / apps / myproject $ svn co svn + ssh: //user@svn.myproject.ru/myproject/tags/rel_0.5.2.1
 ...
 user @ stable: ~ / apps / myproject $ ls -la
 drwxr-xr-x 12 user www-data 4096 Oct 29 11:38.
 drwxr-xr-x 5 user www-data 4096 May 13 23:22 ..
 drwxr-xr-x 8 user www-data 4096 Oct 22 17:11 rel_0.5.1
 drwxr-xr-x 8 user www-data 4096 Oct 27 13:17 rel_0.5.1.1
 drwxr-xr-x 8 user www-data 4096 Oct 28 02:07 rel_0.5.1.2
 drwxr-xr-x 8 user www-data 4096 Oct 29 11:38 rel_0.5.2
 drwxr-xr-x 8 user www-data 4096 Oct 29 12:31 rel_0.5.2.1

 user @ stable: ~ / httpdocs / $ ls –la
 drwxr-xr-x 3 user www-data 4096 Oct 29 11:40.
 drwxr-xr-x 9 user www-data 4096 Oct 28 21:25 ..
 lrwxrwxrwx 1 user www-data 27 Oct 29 11:40 pro -> ../apps/myproject/rel_0.5.2
 user @ stable: ~ / httpdocs / $ rm pro && ln –s ../apps/myproject/rel_0.5.2.1 pro
 user @ stable: ~ / httpdocs / $ ls –la
 drwxr-xr-x 3 user www-data 4096 Oct 29 11:40.
 drwxr-xr-x 9 user www-data 4096 Oct 28 21:25 ..
 lrwxrwxrwx 1 user www-data 27 Oct 30 17:12 pro -> ../apps/myproject/rel_0.5.2.1

• disable some backends by redirecting all traffic from the fronts to the remaining workers
• On disconnected backends we perform local implementations and tests.
• switching traffic to fresh backends
• repeat the procedure for the released backends
• As a result of block introduction, the front-ends are switched to the normal state and distribute the load across all backends.

PS If you have enough strength to finish the story, wait for the second part about the virtues of LiquiBase , the rsync pitfalls and the migration of file storages.

PPS In the course of correspondence with one of the habrayuser it was formulated: the main thing in this informational appeal is not to impose one's recommendations, but the fact that I make a person think about the problems that may arise in his particular situation. General recommendations on * ^ & no one needs. In the end, everyone makes their own way. And it is right. Because every project is unique.