Software for Boeing-737 Max was written by outsourcers earning $ 9 per hour

In the midst of the crisis around Boeing-737 Max, it still remains a mystery: how the company, renowned for its careful approach to design, has made, apparently, children's mistakes in the development of software that led to two disasters with human casualties. Engineers working in the company for many years, say that the development was complicated due to the delegation of part of the work to low-paying contractors.

The flaws in the software will probably leave the planes confined to the ground for another month - this week American regulators discovered additional problems. The software for the 737-Max series was written at a time when Boeing fired experienced engineers and put pressure on suppliers.
')
Moreover, the icon of the American aircraft industry and its subcontractors trusted temporary workers, who earned only $ 9 an hour, to develop and test their software. Often, these were workers from countries with underdeveloped aircraft construction, namely from India.

“Yesterday’s graduates hired by the Indian software company HCL Technologies Ltd occupy several rows of tables at the offices of Boeing Field in Seattle (officially King County International Airport, at this airport Boeing has its own hangar and conducts aircraft tests - approx. Transl.)” says Mark Rabin, a former Boeing engineer who worked in the test group of 737-Max series aircraft.

HCL coders are usually designed according to specifications sent from Boeing. But, according to Rabin, “this is a controversial decision, as it is much less efficient than simply letting the Boeing engineers write the code”. He recalls that “it was often necessary to redo everything several times, because the code was written incorrectly.”

Support from Indian companies may have brought other benefits. Over the past few years, Boeing has won several tenders for the supply of military and commercial aircraft to India, for example, a $ 22 billion contract for SpiceJet Ltd. This contract includes 100 737-Max 8 aircraft and is the largest order in the history of Indian airlines traditionally cooperating with Airbus.

According to the findings published in social networks, engineers from HCL participated in the development and testing of software for PFD (Primary flight display, the main flight display - approx. Transl.), And employees of another Indian company, Cyient Ltd., were engaged in software for instrumentation designed for flight tests.

Costly delay

In one of the posts, the HCL employee described his work responsibilities as follows: “I made a crutch in a quick way to solve the production problem and not delay the flight tests of the 737-Max (the delay of each flight costs Boeing a huge amount)”.

The company Boeing claims that it did not trust the engineers from HCL and Cyient to develop the MCAS system (Maneuvering Characteristics Augmentation System), which is linked to the crash of Lion Air JT-610 flight near Jakarta in October 2018 and ET302 Ethiopian Airlines flight in Addis Ababa in March 2019. Also, according to Boeing, none of these companies is associated with a problem discovered after a catastrophe — a signal lamp in the cab that is not working for most customers.

“Boeing has many years of experience working with suppliers and partners around the world,” said a spokesman for the company. “Our main goal is to always be sure that our products are safe, of the highest quality and are made according to all the rules.”

In turn, the company HCL in an official statement claims that “has a strong and long-standing business relationship with Boeing and is proud of the work that the company has done for its customers. However, HCL does not comment on what kind of work it was. HCL is in no way associated with current problems with the 737 Max. ”

Recent tests on the emulator, conducted by the Federal Aviation Administration of the United States, revealed that the problems with the software lie at a deeper level. Shares of the company this week fell in price after regulators discovered a problem with a microcircuit starting to give critical answers with a delay, in case it is overloaded with requests.

The development of the 737 Max began 8 years ago, and the engineers who worked on it complained of the pressure from the managers. Requirements have been made to limit changes that potentially create additional costs.

“Boeing did everything possible, everything that you can imagine, in order to reduce costs, including the transfer of development from Puget Sound (because the region is in the state of Washington, where the Boeing company’s production facilities are located). too expensive, ”says Rick Ludtke, a former flight test engineer who was fired in 2017. “This can be understood if you look at the situation from a business point of view. Gradually, over time, it became clear that this weakened the design ability of Puget Sound engineers. ”

Rabin (Mark Rabin), a former programmer who was dismissed in 2015, recalls that one of the managers at a general meeting said that the Boeing company does not need seniors, since their products are already mature enough. “I was shocked that in a hall filled with a couple of hundred mostly senior-engineers, we are quite seriously being told that we are not needed ...”

A typical jetliner consists of millions of parts, and millions of lines of code, and Boeing has long since transferred most of the work to suppliers who simply follow the detailed drawings.

Beginning with the launch of the 787 Dreamliner in 2004, Boeing sought to increase profits by providing high-level specifications instead of drawings, and then offering suppliers to work out the parts themselves. The idea was “they are experts, you know, and they will take care of these things for us,” says Frank McCormick, a former flight test engineer who later worked as a consultant for regulators and manufacturers. “It was just nonsense.”

An additional reason for the transfer of work abroad are sales. In return for a $ 11 billion contract with Air India, signed in 2005, Boeing pledged to invest $ 1.7 billion in Indian companies. This, of course, was a boon to HCL, Cyient, and other companies whose programmers were widely used in the computer industry but were not yet involved in aircraft construction.

Rockwell Collins, which manufactures electronics for airplane cockpits, was one of the first aircraft-building companies to transfer a significant part of its work to India, where, starting in 2000, HCL began testing their software. By 2010, HCL employed more than 400 people engaged in the development and testing of software for Rockwell Collins in the offices located in Chennai and Bangalore.

In the same year, Boeing, together with HCL, opened the so-called “center of excellence” in Chennai, stating that the companies will cooperate “to create mission-critical flight test software”. In 2011, Boeing added Cyient (at that time known as Infotech) to its list of “suppliers of the year” for designing, testing and developing software for models 787 and 747-8, carried out at another center in Hyderabad.

Competitors of the Boeing company also partially rely on outsourcers. In addition to sales support (as mentioned above), aircraft manufacturing companies claim that distributed design teams have higher efficiency because they work around the clock. But outsourcing has long become a sore point for some Boeing engineers, who, apart from fear of losing their jobs, say that this has led to problems with interaction between the teams and mistakes.

Moscow errors

Boeing also expanded its design center in Moscow. In 2008, during a meeting with the chief engineer responsible for Boeing-787, one of the employees complained that he sent drawings to the team 18 times in Russia before they realized that smoke detectors must be connected to the electrical system, said Cynthia Cole ( Cynthia Cole), a former engineer of the Boeing, who headed the union of engineers from 2006 to 2010.

“Design began to turn into a cheap product,” adds Vance Hilderman, co-founder of TekSci, which provided services to contracting engineers and began losing orders due to foreign competitors in the 2000s.

According to Hilderman, a security engineer with thirty years of experience, recent customers include Boeing’s main suppliers, American avionics companies have transferred over 30% of their software development over the past few years, compared to only 10% of European companies .

A strong dollar was the key to the attractiveness of this model. Engineers in India earned about $ 5 per hour, now it is $ 9 or $ 10, compared with $ 35-40 for those who are in the US on an H1B visa, adds Hilderman. But he explains to his customers that, in reality, the low price per hour costs them $ 80, because of the need for control, and says that his company partially returns customers who need to fix bugs.

HCL, formerly known as Hindustan Computers, was founded in 1976 by billionaire Shiv Nadar and has annual sales of more than $ 8.6 billion. According to Sukamal Banerjee, vice president of the company, HCL is a global company with 18,000 employees in the United States and 15,000 in Europe, and has extensive experience in computing. And she won an order from Boeing for this very reason, and not at all because of the price. He directly states: “We have a lot of experience in R & D (Research & Development, research and development work - approx. Transl.)”.

However, when working on 787, HCL put a great price on Boeing - for free, according to Sam Swaro, assistant vice president, who offered HCL services at a conference in San Diego, organized by Avionics International magazine in June. He said that the company did not take advance payments for 787 and began to issue bills only on the basis of sales after a few years - an “innovative business model”, which he proposed to extend to other companies in the industry.

The Boeing-787 model was commissioned in 2011, three years late, and exceeded the budget by billions of dollars, partly due to confusion caused by the outsourcing strategy. Under the leadership of Dennis Muilenburg, a long-time engineer at Boeing, who became CEO in 2015, the company said it plans to take back most of the work on the latest aircraft.

Engineering swamp

Boeing-737 Max became the sales leader soon after it was announced in 2011. But for ambitious engineers, it was something like a “swamp,” says Peter Lemme, who designed the autopilot for the Boeing-767, now a consultant. The Boeing-737 Max was a 50-year-old design upgrade, and the changes had to be limited enough that Boeing could stamp new aircraft like hotcakes, with minor changes to assembly lines or airlines. “For an engineer, this is not the best job,” Lemme added.

Rockwell Collins, currently a division of United Technologies Corp., won the contract for the supply of cabin displays for the 737 Max and in its work relied on HCL engineers working in India, in Iowa and in Seattle. A spokesman for United Technologies declined to comment on this situation.

Cyient contracting engineers helped with flight test equipment. Charles LoveJoy, a former Boeing employee, said that US engineers were forced to recheck drawings made in India at night every morning at 7:30. “We had problems with the Indian team. They met the requirements, but we could have done better. ”

Numerous investigations, including a criminal investigation conducted by the US Department of Justice, are trying to figure out how and when critical decisions were made regarding the 737 Max software. According to investigators, during the crash of Lion Air and Ethiopian Airlines aircraft, as a result of which 346 people died, the MCAS system pushed the aircraft to uncontrolled diving due to poor data from a single sensor.

According to Lemma, such a design violates the basic principles of redundancy, which were unshakable for several generations of Boeing engineers. Apparently, no one has ever tested how the software will react in this situation. “It was a deafening failure,” he said. "Not one person, but many people should have thought about this problem."

Boeing also reported that shortly after the start of 737-Max shipments in 2017, they found that a warning light, which could warn the crew about a problem with the sensor, was incorrectly configured in the flight display software. In a May statement by the Boeing company, explaining why the company did not inform regulators about it in time, it says that the engineers decided that this was not a safety issue.

“The general management of the company,” the statement said, “did not participate in this audit.”

From the translator: after reading the article, I stopped wondering at the situation in my industry (e-commerce). If the industrial giants responsible for human life, such a mess with the processes, then what to say in smaller offices. Well, I would add that Bloomberg, of course, distorts (there was such an impression), since their task is HYIP and views, so the writing should be divided into two.

Error messages, typos and other problems are welcome.