Fault tolerant garbage system
Actually the story was as follows.
For one firm N, it was necessary to develop a cheap and reliable data storage and processing system. Briefly about the data. It is necessary to take information from clients (I’m missing what, something like tax reports) and keep it for many years. Quite often it was required to search for this information and even more often to modify the data entered in the last couple of hours. Loss of information is unacceptable in any case. Including in case of fire or earthquake. Previously, all this was done on paper and stored in large folders. For parsing folders there was a whole department of meaningless and merciless people.
All this had to be transferred to an automated basis. The most interesting thing is that the development was paid quite decently, but they didn’t allocate money for iron at all - they asked for all this to be raised on the existing hardware. The car park consisted of a dozen morally dead monsters and it was on them that the database server and backup server had to be raised.
The machines were mainly P3, 128mb operational, 16mb video, 10GB screw. That is not a standard server, to put it mildly.
')
It was proposed to backup every record - that is, to mirror the database server and backup server. Given the volume of data and the slowness of the machines at the first tests, this showed unpleasantly long results when sampling from the database. The base itself, by the way, did not greatly contribute to the speed of work, because it was completely normalized and compact.
Since the editing of records in the database was quite frequent, this caused inconvenience.
After a brainstorming session, it was decided to pervert somewhat.
Total in the end:
Server A. The main working server of the system, keeps the Base for the current day and executes all the main logic.
Server B. The second server stores the database for all time, not counting the current day. At midnight, data from A was poured into B. Here are scripts for compiling reports on periods.
Server S. Backup server. After overflowing at midnight, backup B, erased the old backup. And mirror A at all times.
Total: with the loss of any of the servers, we had all the information. That is, A + B or C.
Server C was settled separately in the case of an emergency (the customer is still paranoid)
On all servers stood Linux, Apache and muscle. The main code is written in PHP, backups on Python.
Fire did not happen, earthquakes, too. Everything works fine for the second year. Later, the “red button” was added - the ability to work at your own risk and risk in case of loss of one of the machines. It seems not useful)
I think this is a bicycle or excessive paranoia, but everything worked, everything passed crash tests and everyone was satisfied.
% username%, how would you improve this system?
UPD: advise which blog to move.
For one firm N, it was necessary to develop a cheap and reliable data storage and processing system. Briefly about the data. It is necessary to take information from clients (I’m missing what, something like tax reports) and keep it for many years. Quite often it was required to search for this information and even more often to modify the data entered in the last couple of hours. Loss of information is unacceptable in any case. Including in case of fire or earthquake. Previously, all this was done on paper and stored in large folders. For parsing folders there was a whole department of meaningless and merciless people.
All this had to be transferred to an automated basis. The most interesting thing is that the development was paid quite decently, but they didn’t allocate money for iron at all - they asked for all this to be raised on the existing hardware. The car park consisted of a dozen morally dead monsters and it was on them that the database server and backup server had to be raised.
The machines were mainly P3, 128mb operational, 16mb video, 10GB screw. That is not a standard server, to put it mildly.
')
It was proposed to backup every record - that is, to mirror the database server and backup server. Given the volume of data and the slowness of the machines at the first tests, this showed unpleasantly long results when sampling from the database. The base itself, by the way, did not greatly contribute to the speed of work, because it was completely normalized and compact.
Since the editing of records in the database was quite frequent, this caused inconvenience.
After a brainstorming session, it was decided to pervert somewhat.
Total in the end:
Server A. The main working server of the system, keeps the Base for the current day and executes all the main logic.
Server B. The second server stores the database for all time, not counting the current day. At midnight, data from A was poured into B. Here are scripts for compiling reports on periods.
Server S. Backup server. After overflowing at midnight, backup B, erased the old backup. And mirror A at all times.
Total: with the loss of any of the servers, we had all the information. That is, A + B or C.
Server C was settled separately in the case of an emergency (the customer is still paranoid)
On all servers stood Linux, Apache and muscle. The main code is written in PHP, backups on Python.
Fire did not happen, earthquakes, too. Everything works fine for the second year. Later, the “red button” was added - the ability to work at your own risk and risk in case of loss of one of the machines. It seems not useful)
I think this is a bicycle or excessive paranoia, but everything worked, everything passed crash tests and everyone was satisfied.
% username%, how would you improve this system?
UPD: advise which blog to move.
Source: https://habr.com/ru/post/87988/
All Articles