How SpaceX uses the GPU to calculate rocket engines

Ilon Musk bursts into the automotive and space industry with Tesla and SpaceX. Interestingly, the former completely abandons internal combustion engines, while the latter, on the contrary, is trying to invent new technologies for burning fuel and to carry out a manned flight to Mars.

At the last GPU Technology Conference, organized by Nvidia, we learned that delivering a group of people to Mars and back is not an easy task. One of the problems of such a mission is the need for a large and efficient rocket engine that will be able to deliver a lot of material into orbit, - this is how Adam Lichtl, director of SpaceX research group, explained to us. With a team of several dozen programmers, he is trying to cope with the difficult task of improving the simulation of combustion inside the rocket engine. To shorten flights to Mars, you also need a large engine.

“One of the main problems is the exposure of cosmonauts to radiation,” explains Lichtl. - For flight to Mars, you must wait for the desired window in accordance with the amount of energy that can produce a rocket. Those who played in the Kerbal Space Program know that in order to change the orbit, for example, around the Earth, you need to give yourself a change in speed and change the orbit to the one that intersects with the orbit of Mars. We need vehicles that give the maximum change in speed, so that the journey is shorter. The second obstacle hindering the work of the expedition to Mars, consisting of people, not just robots - the lack of infrastructure. Pioneers, crossing new lands, build their houses themselves, and they need to get food. It's quite difficult to do on Mars. ”


')
You not only need to take along a bunch of everything to Mars to maintain a colony there, but also somehow create fuel on Mars to return to Earth. All this affects the development of the engine.

The ISS weighs 450 tons and for its construction it took 36 flights of Shuttles and five launches of "Protons". NASA Mars Design Reference Architecture wants to launch 300 tons of materials into orbit and assemble a ship there to fly to Mars. This is desirable to do three flights of 100 tons. Building and fueling a ship in Earth orbit is a relatively simple matter, but a rocket must be able to work on methane, since it can be synthesized from the water of Mars and carbon dioxide from the atmosphere of Mars. SpaceX's current Merlin 1D engines run on kerosene, which has greater density and thrust. It is produced by distillation of oil.

And as if these problems are few, one more is added. Computational fluid dynamics, CFD, a computational fluid dynamics (VG) program for simulating the motion of fluid and gases and their combustion inside engines, is not well suited for simulating the operation of rockets.

“Methane is a fairly simple hydrocarbon, well-suited as a fuel,” says Lichtl. - Trunk in the development of an engine that works effectively on it. But the VG for rocket engines is extremely complex. ”

SpaceX cooperates with various institutes and Sandia national laboratories in the development of its own software for VG, which will allow in the future to create improved versions of Merlin engines suitable for flight to Mars and capable of operating on methane.

Lichtl is a good choice of the head of the development of rocket engines (in my opinion, it is generally a dream job). As a thesis at Carnegie Melone University, he developed simulators for high energy physics, and then worked in nat. Brookhaven laboratories on the same topic. Then he worked as an analyst and was engaged in strategic planning in Morgan Stanley, and then in another financial company. Lichtl joined the SpaceX team as chief engine engineer two years ago, and went up to head the development department for Mars missions.

Stephen Jones, the main software developer at SpaceX, who previously worked as an engineer at Nvidia, is leading a project to develop his own software for VG. The program has no name yet, and the techniques used in SpaceX can be used to improve all kinds of combustion engines - even automobile ones.

Existing software for VG is not suitable for calculating missiles

At a factory in Texas, SpaceX tries to change various injectors and other parameters to squeeze the most power from the engines, and tests them every day. Tests are expensive, and even if you build sensors across the surface of the engine, you still can't find out what's going on inside. It is much better to simulate all the components of the engine and the behavior of the fuel, and to minimize the set of options for installing injectors to a minimum, after which you can already build a real engine and test several optimal options.

“Another property of engines that we have learned is the instability of combustion as a result of propagating pressure waves and the release of chemical energy. This has always hampered the development of rocket engines. This is a curse of the industry. The engine starts to shake, and either falls apart, or just shakes so hard that you cannot install any payload on it. ”

The efficiency of automobile engines and turbines was attempted to be improved with the help of VG calculations. Because of their small size, this is much easier to do than for large rocket engines. Therefore, you can not just take the software and scale it.

At the molecular level, reactions in rocket fuel occur in about 10 ^-9 - 10 ^-11 seconds, and heat transfer through advection occurs at time intervals of the order of 10 ^-6 - 10 ^-7 seconds. Acoustic vibrations in the combustion chamber occur on time intervals of the order of 10 ^-3 - 10 ^-4 seconds. This is too big a scale to calculate in the VG-simulation.

Another problem with calculations is the physical size of the engine. The length of the combustion chamber can be a meter, and at the other end of the Kolmogorov scale are dimensions of the order of 1 micron, which control the rate of dispersion in the fuel stream and determine the rate of combustion in the chamber. It turns out the difference in a million times, which must be taken into account when calculating.

“If we start dividing the grid for calculating VG by powers of two, then it will be necessary to divide it about 20 times to accommodate such a scale of lengths,” says Licht. If you try to uniformly distribute the lattice throughout the combustion chamber, you will need to process the yottabytes (10 ²⁴ ) of data. This is unreal.

In this regard, the developers of rockets model what is happening either with a rough approximation and a variety of characteristics, or in more detail, but on a smaller scale.

“Why not do it at the same time? In turbulence, it is interesting that, despite the large scatter of scales, the flux density is low. You do not need to calculate everything up to microns at each point. By nature, it is fractal. And fractality allows us to concentrate computing power where they are needed. This is something like a clever compression algorithm. ”

It is only necessary to carry out all calculations on compressed fractal data without having to unclamp them. To do this, engineers invented a technique called local fractal compression of microwaves (wavelets local fractal compression). For simulations an adaptive grid is used with different resolutions in different places. Typically, such techniques involve predetermined lattice parameters, but SpaceX managed to do a simulation with dynamically adapting parameters.

To demonstrate the SH, Jones showed a simulation of fuel and oxidant that explode inside the box, and calculate the propagation of shock waves that are reflected and move inside the fluid, leading to a huge amount of fractal turbulence. ( link - video , simulation starts from 36 minutes). The simulation calculates the change in gas density in increments of several milliseconds.



“If you made these calculations in the usual way over the whole grid, you would have to calculate 300 million nodes,” says Jones. “And we did it with a single GPU, because we only calculate the parameters in those places where it is needed, and the algorithm itself knows where exactly the calculation should be made.”

Although the specific GPU was not named, it was probably the Tesla GPU Kepler class coprocessor, and apparently it used parallel computing capabilities to calculate the grid.

Here is another simulation on a single GPU, showing gas ignition and temperature distribution (in the video from 37:45), where gas explodes and reflects from the walls:



The yellow areas are the hottest. Jones says that you can make an increase in the image of 60000 times and there will still be visible internal structures.

Lichtl says similar techniques have been tried before, and these simulations are based on the work of Jonathan Riegel, a professor at the University of Iowa.

“However, without a GPU and without our architecture, even the simplest simulations would have to be cheated on thousands of processors within months. GPU dramatically speeds up the process. It is good that in SpaceX we were allowed to start from scratch in the manufacture of VG. What for? Then, if you are a little mistaken in the classic version of the VG simulator, it usually gets away with you. And when simulating combustion, if you are mistaken somewhere, all processes begin to go differently. ”

For example, a small error in the temperature of the gas leads to a large error in the assessment of its kinetic energy, because they are linked by an exponential relationship.

Another nice feature of their software, coupled with the GPU, is that they cheat well on chemical kinetic models describing the bidirectional chemical reactions that occur when a fuel encounters an oxidizer. Combustion of hydrogen with oxygen is not a simple reaction, leading to the appearance of two water molecules from two hydrogen molecules and one - oxygen. You will have to calculate 23 possible reactions and 11 intermediate ones in order to calculate all possible outcomes. When methane is burned with oxygen, 53 intermediate reactions and 325 final variants appear.

“This is not a sparrow figurine show,” says Lichtl. - It is impossible to overestimate the benefits of parallel architecture. By reducing the number of required grid points for calculation, you can transfer valuable computing resources to something more important, for example, to miscalculate chemical reactions. ”

This is exactly what the software from SpaceX does, calculating the chemical reactions and turbulence of the fuel in the combustion chamber and at the nozzle exit (video from 43 minutes):



In this case, the acoustic properties of combustion are shown, including the Mach disk and Mach rings, all derived from a mathematical model, and quite well imitating the actual combustion shown in the photo on the left above. At the nozzle exit the gases are transparent, and their speed exceeds the speed of sound. Faced with a shock wave, they slow down and shrink, becoming white.

Further software development involves scaling to multiple GPUs operating in clusters.

“Now the code runs on the same machine. In the near future we will be doing parallelization, ”says Lichtl.

There is no need to wait until NVIDIA completes its NVLink . Now they can make clusters based on the PCI-Express bus. It also considers the possibility of using the MPI protocol, when different servers will work in parallel, calculating the data for VG.

An intermediate solution while waiting for NVLink can be porting software to clusters from IBM Power8 processors, and using the InfiniBand network interface, which works with processors through the IBM Coherent Accelerator Processor Interface, or CAPI. This architecture is used by the US Department of Energy in its largest supercomputers Summit and Sierra, which are scheduled for 2017. In addition, there is a development from Mellanox Technologies , the network interface ConnectX-4, which allows you to work with Power8 via CAPI at speeds up to 100 Gb / s. There are several companies that make the system OpenPower , with which SpaceX will also be able to cooperate if it decides to go this route.

The main thing is that Lichtl wants to purchase servers with multiple GPUs. What can not please the company Nvidia.