Epidemic in the data
I want to tell a story about how one day an unforeseen problem caused financial damage to the business of a large corporation. I will talk about the causes of this problem, how we overcame it and how to solve it correctly. I hope this article will help designers to avoid similar situations in the future.
We have such an entity in our database - a quota - it’s like an order model. It has a header and countless substructures: order lines, calculated prices, all accompanying information, etc. Before becoming an order, quotas pass from hand to hand, from department to department - are checked and rechecked. The quotas themselves are immutable (immutable), so any change in quotas creates a copy of it with the new version. In addition, quotas can be used as templates for other quotas. In all cases when one quota spawns another, all the data of the original quota is inherited generated by except for those that are changed. A new quota can also be created from several by merging. The initial source of quotas are several front-office systems.
So, once, after the deployment of a new version of one of the front-office systems, the audit department staff raised an alarm for the reason that some orders went into processing with slightly lower prices. Developers connected to the investigation, and very soon it turned out that the cause of the problem with prices was, no matter how trite, an error in the code. By itself, the error was easily fixable, but the data damage caused to it was nontrivial. In short, according to all business rules, the damaged quotas were valid; a special computational procedure was required in order to determine that the debit did not converge with the credit. The situation was aggravated by the fact that when the new quota was generated from the damaged one, the new quota became damaged.
The number of damaged quotas grew at a speed of more than a thousand per hour, and with each new company the company lost more and more money. The task was to prohibit the creation of new quotas damaged. Initially, programmers developed a SQL query, which with 100% accuracy determined the damaged quotas, but it turned out that there is no way to prohibit a quota to generate another in all possible cases! It simply did not provide the designers. The only available way turned out to be - validate quotas when loading, mark the damaged ones and add the saving validator so that it does not allow to save the marked quotas. The new version of the front-office system was deployed as soon as possible, and this stopped the epidemic.
')
Since then, in the code of our system there is a call to a method that, as a seal, holds back the great evil that lives in our database. It would have been better, of course, to mark the quotas at the base itself, but for bureaucratic reasons, the management considered the decision to be sufficient and did not change the base.
The moral of this story is this: if you design a data structure that generates copies of itself, consider a simple thing - a flag that prohibits it from multiplying. Otherwise, when this thing mutates and turns into a monster, you will have nothing to fight with it.
We have such an entity in our database - a quota - it’s like an order model. It has a header and countless substructures: order lines, calculated prices, all accompanying information, etc. Before becoming an order, quotas pass from hand to hand, from department to department - are checked and rechecked. The quotas themselves are immutable (immutable), so any change in quotas creates a copy of it with the new version. In addition, quotas can be used as templates for other quotas. In all cases when one quota spawns another, all the data of the original quota is inherited generated by except for those that are changed. A new quota can also be created from several by merging. The initial source of quotas are several front-office systems.
So, once, after the deployment of a new version of one of the front-office systems, the audit department staff raised an alarm for the reason that some orders went into processing with slightly lower prices. Developers connected to the investigation, and very soon it turned out that the cause of the problem with prices was, no matter how trite, an error in the code. By itself, the error was easily fixable, but the data damage caused to it was nontrivial. In short, according to all business rules, the damaged quotas were valid; a special computational procedure was required in order to determine that the debit did not converge with the credit. The situation was aggravated by the fact that when the new quota was generated from the damaged one, the new quota became damaged.
The number of damaged quotas grew at a speed of more than a thousand per hour, and with each new company the company lost more and more money. The task was to prohibit the creation of new quotas damaged. Initially, programmers developed a SQL query, which with 100% accuracy determined the damaged quotas, but it turned out that there is no way to prohibit a quota to generate another in all possible cases! It simply did not provide the designers. The only available way turned out to be - validate quotas when loading, mark the damaged ones and add the saving validator so that it does not allow to save the marked quotas. The new version of the front-office system was deployed as soon as possible, and this stopped the epidemic.
')
Since then, in the code of our system there is a call to a method that, as a seal, holds back the great evil that lives in our database. It would have been better, of course, to mark the quotas at the base itself, but for bureaucratic reasons, the management considered the decision to be sufficient and did not change the base.
The moral of this story is this: if you design a data structure that generates copies of itself, consider a simple thing - a flag that prohibits it from multiplying. Otherwise, when this thing mutates and turns into a monster, you will have nothing to fight with it.
Source: https://habr.com/ru/post/125719/
All Articles