📜 ⬆️ ⬇️

Variant of the script for updating the organization’s software package

Introduction


In this post, I will tell you how we update the software in the organization.
Our organization provides some services around the clock, and the process of updating our programs, databases is very acute, because idle time of the system without work is very critical and leads to the loss of money of our customers. So I decided to share with the community about what we have and listen to all the comments, criticism and suggestions for improving this process and reducing the downtime of the system as a whole.

We have two IBM x3550 servers installed in a rack (one working main, the second test mirror). The servers are running Windows Server 2008 R2.
The software complex consists of:

The main problem and its solution


So the main problem that torments me is the process of updating all our system components (databases, programs). When I came to work in this organization, the update process could take from 15 minutes to 30-60 minutes (depending on various reasons: programmer errors, system brakes, etc.). The main downtime was the launch of our programs, which started sometimes well, very long.
Then I bought a second server and immediately decided to set up a mirror for the second server to save important data (this is how I decided to the database). I set it up and thought that by activating the second backup server we will be updating through it. But it was not there, the mirror base is generally unavailable, even for reading.

Our update script

To reduce the downtime of our services during the update, I set up the following update scheme (and I think it is confused). I set up two servers identically with each other so that they can be swapped and there was no need to do anything extra, so that our services work the same way both on the main and standby (performing only failover in MS SQL), for this I configured the same IIS and firewall .
And then when the time comes to update, the following happens:
  1. Failover of databases configured as mirrored;
  2. The second server is updated on the databases and then our programs are launched;
  3. After starting the servers, the IP addresses change among themselves;
  4. After that, software updates are performed on the main server;
  5. Finally, steps 1, 2 and 3 are performed again;

Result

Using this scheme, I managed to reduce the downtime of our services from 15-60 minutes to 7-15 minutes. On average, idle time is obtained twice for 5 minutes.

Conclusion


I understand that the scheme is very confused, time consuming and complex. But for now we are updating this way. Update according to the scheme as it was before me and, say, postpone the update time for the night, also not a very option. Because after the update, there may be problems in the system and programmers are needed to fix these errors immediately. Calling a taxi at night is idle time, which can then be calculated for hours. To call immediately to update programmers for security is also not an option, because their work rhythm will be broken and will affect the quality of their code.
')

Possible solutions


I have an idea to switch to merge replication so that databases on the second server are available, update on the second server and swap them later: this will reduce downtime, but it will add problems with the replication and database operation, since conflicts will be possible.
I really want to find a good, uncomplicated update solution, preferably without replacing servers, so you can not think about other third-party services like corporate mail, Sharepoint portal, etc. Willingly heal the sentences. Will a third server in the rack help solve my problem?

Source: https://habr.com/ru/post/141406/


All Articles