How 45 minutes to lose $ 172 222 per second

This is perhaps the most painful error report I've ever read. He colorfully describes the steps that led to the loss of $ 465 million by Knight Capital due to a software bug that appeared last year and bankrupt the company.

This report contains all the characteristics of technical debt in a huge, support-free and running code base (an error occurred due to the execution of a code that was not used for almost 9 years) and a terrible and sad history of interaction between software developers and IT professionals.

Highlights:

To ensure the participation of its customers in the Liquidity Program (PL) on the New York Stock Exchange, which was scheduled to launch on August 1, 2012, Knight made a number of changes to its systems and program code associated with the order processing process. These changes included the development and deployment of new code in SMARS. SMARS is an automated, high-speed, algorithmic router that sends orders to the market. One of the main functions of SMARS is to receive orders from other components of the Knight trading platform (“parent” orders), and, as necessary, based on the available liquidity, sending one or several representative (or “daughter”) orders to external services for execution.

13. When deploying a new PL, the code in SMARS should have replaced the unused code in the corresponding part of the router. This unused code was previously needed for the Power Peg function, which the company has not used for many years. Despite this, it remained operational and invoked during the deployment of the submarine. The new PL code used the flag that was previously tied to Power Peg. Knight wanted to remove the Power Peg code so that when this flag is activated, the new submarine functionality was used, not Power Peg.
')
14. Previously, when using Power Peg, the summing function calculated the number of shares in the execution of the child orders and signaled the need to stop placing the child orders after the parent order was completed. In 2003, Knight stopped using Power Peg. In 2005, Knight changed the Power Peg code by moving the tracking function of the parent order to an earlier stage of the SMARS code sequence. Repeated testing of the Power Peg code after the change of Knight was not performed, and that the procedure is still working correctly was not verified.

15. Starting July 27, 2012, Knight has deployed a new submarine code in SMARS, placing it on a limited number of servers. During the deployment of a new code, one of the technicians did not copy the new code to one of the eight SMARS servers. In Knight, there was no second technician who would conduct a deployment check, and no one understood that the Power Peg code was not removed from the eighth server and the new PL code was not added. Knight did not have any written procedures that would require such verification.

16. On August 1, Knight received orders from brokers-dealers, whose clients could participate in the submarine. Seven servers processed orders correctly. But orders sent to server 8 with the launch flag set launched the defective Power Peg code that was still present on that server. As a result, the server took orders as parent and started sending child orders to the trading centers. Due to the fact that the function of checking the fulfillment of the parent order was moved to another stage of the process, the server continued to place the child orders non-stop - not paying attention to the fact that the parent order was already completed. Although some part of the order processing system determined that the parent order was completed, this information did not fall into SMARS.

19. On August 1, Knight also received orders that related to the submarines, but were intended to trade before the market opened. 6 SMARS servers processed these orders and, starting at approximately 8:01 am, internal systems generated automatic messages (called “BNET failure”) that referred to SMARS and described the error as “Power Peg Disconnected”. The Knight system sent 97 such messages until 9:30 in the morning when the market opened. Messages of this type were not regarded by the system as dangerous, and the staff did not read them at all.

Further more fun:

27. On 1 August, there were no incident response procedures at Knight. In other words, the company did not have control procedures for personnel management when serious problems occurred. On August 1, Knight used the services of his team of technicians to identify and fix problems in SMARS in a live trading environment. The Knight system continued to send millions of "daughter" orders while the staff tried to identify the source of the problem. The company even deleted the new PL code from the seven servers on which it was installed correctly. This aggravated the situation, because the new “parent” orders activated the Power Peg code that was present on these servers, similar to what already happened on the eighth server.

Of course, it is worth reading the entire document, it pays great attention to new verification procedures carried out by people in order to avoid such a tragedy. The mistakes of the developers, of course, were related to the human factor, but such consequences were the result of a bad deployment scenario and ugly monitoring. What is this office that does not even check the version of the cluster software? Not to mention the deployment scenarios in which return codes are checked.

We can only hope that the “written verification procedures” of unused code implied systematic tests, although Wikipedia says that this is not the case.

And for sweetness: the fine amounted to another $ 12 million, an audit showed that the system was constantly trying to implement speculative short sales.