How we built a simulator: an invaluable experience
Hello to all!
A little less than a year ago, I was involved in the process of building a device, which is located somewhere between the simulator and the professional simulator. This was not the first experience - in 2009 we had already solved such a task, therefore, starting the realization of a very ambitious dream, we tried not to step on all the possible rakes. Here I will describe the background with the "first pancake", and how it affected the second. If it is interesting, I will write in detail about our second simulator.
Fasting in “DIY” because none of us had any experience in building airplanes or professional simulators before. Moreover, as it turned out later, the approaches to the design of a low-budget aircraft simulator are radically different from the approaches in the professional field. In general, both the first and second simulator were built by amateurs, albeit with a technical background.
')
Beware , a lot of pictures, causing a nervous tic in aviation enthusiasts and engineers.
So…
In July 2009, one of my friends called and said that a businessman in Moscow is building a Boeing 737-800 airplane simulator for himself, and that he doesn’t have everything, and say if I would like to participate in the project. I wanted, came to this businessman, was stunned, and immediately joined the work.
I must say that as a result, the simulator was built, perhaps, the most expensive and with the maximum number of rakes. This was due to the lack of experience with us, to a very large extent - with the extreme underdevelopment of the market for low-cost flight simulators and their components, and, most importantly, we did not plan. It was a pure laboratory, knee and impromptu: we were extremely limited in time.
As a result, somehow, our gizmo came to life, appeared on MAKS, and we began to roll everyone on it.
Soon after the construction of the Boeing was completed, I left the company and returned to it in the summer of 2012. I was called back, because that wonderful businessman decided to build a second simulator. This time they became the Airbus A320.
This time we decided to approach the issue more proactively: a joint decision was made to spend time and energy for planning. In addition, I insisted on first appearing papers and pictures, and only then were purchases of numerous components.
Of course, one of the main tasks was to reduce the total cost of ownership, and for this we needed to analyze and think about the rake, which we stepped on with Boeing.
For a start, I decided to analyze the rake. In the end, these very rakes at the design and construction stage led to unexpected downtime for the simulator, which resulted in very tangible financial and image losses for the company. It was necessary not to try them in the second round.
In fact, when the Boeing came into being, it was supposed that “everything will work by itself”. There were only three suppliers: a cabin with all the electronics, a mobile platform and a video system. Soft (Project Magenta if it says something to someone) was supplied by the cab supplier, respectively, and the infrastructure was "given to us from above." As operation showed, it was necessary to at least correctly scatter components on computers, but even this was not done.
In the future, the constant changes (which I will discuss separately) led to the fact that the system healed with its own life. Every time a failure or failure occurred, it was necessary to untangle a node from a multitude of components in order to eventually discover that one of them stopped working if the log file reached a certain size.
There was no question of any regular backup, control of component versions and speech.
As for performance, as the installation of new scenarios, the performance of the simulator fell very significantly, which resulted in the replacement of MSFS with X-Plane. Why exactly this election was made - I do not know, I did not take part in it, but the fact remains.
Even after replacing the monolithic MSFS with a scalable X-Plane, the performance of the graphics subsystem was not calculated, with the result that the hardware of these servers was upgraded twice.
From this disaster, I concluded that it is just necessary to draw the architecture of the entire simulator, paint which component will work on which computer, provide for possible extensions, paint hardware performance requirements, estimate network load and make a small margin for further development. Ultimately, the purpose of the simulator is not the constant development of new technologies, but smiles on customers' faces.
Oh, it's just a holiday.
While MSFS was used as a simulator, everything was more or less stable: put a curve to the curve — get the simulator out of the way. Simply and without frills.
When we switched to X-Plane and scaled the system by installing four copies (one considers the dynamics, and each draws his own picture on his projector), it turned out that it is much more difficult to extend the same script to four copies without errors. Especially when you consider that one of them works under Windows, and three - under Linux. Errors and problems began, the flight of scenarios on one of the projectors (the most unpleasant). At some point, the control over the scenarios was completely lost, and a willful decision was made to completely synchronize all installations with one. This made it possible to almost completely get rid of the problems with the visualizer, but the root of the problem remained - we have four different installations of simulators, three of which form an image on one screen.
Sometimes iron (controllers) failed. Instead of reconciling and buying the same thing, controllers from other manufacturers were bought. As a result, after three years, as many as 5 vendors stood in Boeing, and each of them had their own interface software with their own features. Of course, it works together, but in the case of some unknown problems, the diagnosis turns into a living hell.
Again, new controllers are new cables on board, and at some point we came to the realization that we no longer know what the wire is, where it goes and what it is for. I had to stop the work of the simulator for several days, and to do almost complete re-wiring.
From these rakes, I concluded that the simulator should be built, if not for ages, then at least for years. It is necessary to assess in advance the directions of possible development, fix them and build a system based on these directions. If at some point there is a desire to improve something in another direction - well, well, something will have to be abandoned in favor of operational reliability.
The operation of the simulator was not like that at all.
There were no operating instructions, no restrictions on changes, a backup plan, routine maintenance, and there wasn’t much else.
Different hands climbed into the same simulator, changed settings, quickly checked and left, and the next morning the simulator did not start.
The lamps in the projectors burned out at the most inopportune moment, and the spare ones were in the store somewhere in Vidnoe, and on Friday evening our courier rushed through all Moscow in order to buy the last lamp in Moscow (except for jokes).
The oil in the cylinders of the drives of the movable platform imperceptibly flowed out, and we found that it only flows when a puddle has formed on the floor. And, of course, this brand of oil was not to buy.
Summarizing: the simulator worked, but was not exploited. Breaking - fix. Do not break - fly.
In the new project it was very important to foresee and describe the operation process: what kind of spare parts and accessories we need, how often we make diagnostics, what we do if this and that breaks, what updates we install and how, how we test them, how often do we make backups, etc. Looking ahead to say that this approach helped a lot. It turned out that if you agree on at least regular backups and don’t make changes when you hit, the system lives much longer and better.
It would seem that in such a technically complex device electrical problems should not take up much space.
But no! When I first joined Boeing, spontaneous shutdowns of the controllers appeared, which we could not diagnose for a long time. At some point, I figured out how to measure the potential difference between the “ground” of USB controllers and the computer in which this connector was plugged in: there were many, too many for the controllers, and for the USB hubs on the motherboards of the computers. It turned out that in some unknown way our suppliers managed to unleash the “lands” of the booths and racks with computers. Power supply was made from one point, with the correct layout of the "earth" and neutral - the number of rolls off has sharply decreased.
Another time, we had one of the phases knocked out, and an autopsy revealed that there was no load balancing. After that, I analyzed the power supply circuit of all components of the simulator and redid it, the problem is gone.
The conclusion was the need for the power supply scheme with the calculation and balancing of loads, their distribution among bespereboynikam, and, of course, with protection from short-circuit and from electric shock. At first - on paper, with checks “what will happen if this posting breaks off”, with the calculation of the cross-sections of wires, nominal values ​​of automata and the like electrical things. And only after the scheme has not changed - the purchase of components and the assembly of an electrical panel.
We never exactly considered how many bulbs and buttons are in the cabin, but the number of I / O controllers exceeds a dozen and a half. And each of them ends in USB, and it should turn on in the computer.
I have already written about the problems with the lack of a common “ground”, we have successfully solved it, but when there are three bundles of cables on three different computers, there are no problems to avoid.
We tried several different USB extenders, tried to install hubs from different manufacturers, tried to assemble all devices on one computer, in general, tried a lot of things, but the problem with the USB device “falling off” at the most inopportune moment even though this day.
The obvious conclusion was the rejection of the use of USB, but as I said, in our enlightened age, the market for low-budget simulators (i.e., those that individuals up to $ 50k can afford) is extremely underdeveloped, and USB reigns there. Moreover, none of the suppliers of components have in mind that the cabin can move - almost all (with the exception of units) of such simulators here - are fixed. We built the simulator on a mobile platform, and hence the need for long cables, and problems with vibration.
Well, well, if so, then there is no getting away from it, which means you need to make the cables as short as possible, do not drag the USB through extension cables 10 meters, and put the computers on board. Looking ahead: this meant that diskless computers should be on board. This is a separate beautiful task, which I will discuss in one of the following parts.
Mobility + people in the cockpit = various problems with the mechanics.
From vibration unscrew the threaded connections, self-tapping screws fall out, over time the plastic begins to crumble.
Our clients are not professional pilots, but ordinary people, sometimes not even very enthusiastic about aviation. They do not know what to do and what not to do.
An ordinary average man in the prime of his life may unknowingly tear the handle off a real aviation instrument, and not even notice it. However, according to the stories of our engineering friends from the “adult” training centers, real pilots sometimes do the same.
In the new simulator, we won this percent rake by 95%: all cables must be fixed, all threaded connections are locked. No screws. Screed, screed, screed. The remaining 5 percent is still our customers. Alas, here - only a gentle, but persistent word. This I attribute to the organizational conclusions.
Plan, design, write, think, analyze.
"From the sheet" you can probably make a scooter for a couple of evenings, but the meat grinder is no longer there. A flight simulator - and even more so.
Analysis of past failures played an important, if not decisive role. It was thanks to the systematization of the “learned lessons” that the need to think and draw, and then to do, became obvious.
Of course, having avoided parts of the old rake, we found new, but semi-annual experience showed that the operational reliability of the new simulator is much higher. Much less forced downtime, failures and failures. There are much more happy customers, and this is precisely the goal.
In the following posts, I will describe the process of building the A320 in detail.
I found photos from the construction of Boeing, spread.
The base of the cab with hand wheels. Originally used computer power supply, which turned out to be a great evil. Now MeanWell
Arrived movable platform from Italy. The mass of the most mobile platform is about 800 kg, the control cabinet is another 800.
We had the opportunity to use a loader (with Airbas - no longer, there was no room). On the left - the first version of the rack with servers. On the right - metal structures, on which the projection screen will be stretched.
Cockpit assembly. MIP (Main instrument panel) is almost assembled.
Cockpit with base and frame.
View through the windows on the open ceiling panel. Subsequently, it became clear that the chosen method of laying cables was incorrect.
Our Canadian experts configure the interface software - a layer between FSUIPC and controllers. Each button, toggle-lamp-indicator "tied" to the variable FSUIPC. Once we lost this configuration - it was fun.
Start the assembly of the base on which the ladder will ride.
Cab, platform, platform control cabinet. The “heart” of the control cabinet is Siemens PLC. Drive drivers - Baumuller. Drivers with engines deserve a separate article. For example, the driver inside considers the temperature of the motor windings depending on the load on the motor. In addition, there are several temperature sensors in the winding, which, of course, are also connected to the controller. That, however, does not prevent him from issuing an alarm signal and stopping all mobility, if the calculated temperature has jumped out of the permissible limit. A calculation is based on the unique parameters of the engine, which are removed in the process of testing at the factory. When we faced this difficulty (with an estimated temperature), we had to get in touch with the plant.
Platform control cabinet and rack with servers as they were 4 years ago.
And now - rrraz, and we loader hoisted the cabin on the platform.
ORE block (engine controls, Throttle Quadrant in English). Actually, the reason for my appearance in the project. It did not work at all. The block is made of a real block from a Boeing 737-300 by a German craftsman. Unfortunately, the craftsman threw out most of the Kondovoy Boeing mechanics, and replaced it with ridiculous rods, and in some places (no joke) with a lighter cap. A screw terminal decided to pour solder. Used controllers - Phidgets. There are 6 of them in one unit.
If I remember correctly, here we are trying to figure out how food is distributed (there are two of them - 12 and 5) inside the unit. This power is used to drive the motors (from the wipers) that emulate the operation of the traction machine when the handles are driven “by themselves”.
View of the base of the cabin. Pay attention to the number of USB extenders. These were the first, after them were more, and more, and more. Cables are not fixed, this was one of the reasons for the extension of the extension cords.
We pulled the projection screen and reduce the image. TrippleHead2Go was used as an image divider, and Sol7 was used as a software for curving the image under a cylindrical screen.
One of the working moments. If memory serves, we swore on the topic of how to make OREs move properly.
Running in
The final view of the cockpit in 2009.
View outside during operation of the mobile platform.
Right now the Boeing from this post looks like this:
and so
so:
All this is called the word "overhaul."
If someone has time and itching to pick our dusty trainer tomorrow and Friday, write to me in PM.
A little less than a year ago, I was involved in the process of building a device, which is located somewhere between the simulator and the professional simulator. This was not the first experience - in 2009 we had already solved such a task, therefore, starting the realization of a very ambitious dream, we tried not to step on all the possible rakes. Here I will describe the background with the "first pancake", and how it affected the second. If it is interesting, I will write in detail about our second simulator.
Fasting in “DIY” because none of us had any experience in building airplanes or professional simulators before. Moreover, as it turned out later, the approaches to the design of a low-budget aircraft simulator are radically different from the approaches in the professional field. In general, both the first and second simulator were built by amateurs, albeit with a technical background.
')
Beware , a lot of pictures, causing a nervous tic in aviation enthusiasts and engineers.
So…
Prehistory
In July 2009, one of my friends called and said that a businessman in Moscow is building a Boeing 737-800 airplane simulator for himself, and that he doesn’t have everything, and say if I would like to participate in the project. I wanted, came to this businessman, was stunned, and immediately joined the work.
I must say that as a result, the simulator was built, perhaps, the most expensive and with the maximum number of rakes. This was due to the lack of experience with us, to a very large extent - with the extreme underdevelopment of the market for low-cost flight simulators and their components, and, most importantly, we did not plan. It was a pure laboratory, knee and impromptu: we were extremely limited in time.
As a result, somehow, our gizmo came to life, appeared on MAKS, and we began to roll everyone on it.
Soon after the construction of the Boeing was completed, I left the company and returned to it in the summer of 2012. I was called back, because that wonderful businessman decided to build a second simulator. This time they became the Airbus A320.
This time we decided to approach the issue more proactively: a joint decision was made to spend time and energy for planning. In addition, I insisted on first appearing papers and pictures, and only then were purchases of numerous components.
Of course, one of the main tasks was to reduce the total cost of ownership, and for this we needed to analyze and think about the rake, which we stepped on with Boeing.
Rake
For a start, I decided to analyze the rake. In the end, these very rakes at the design and construction stage led to unexpected downtime for the simulator, which resulted in very tangible financial and image losses for the company. It was necessary not to try them in the second round.
Lack of IT infrastructure
In fact, when the Boeing came into being, it was supposed that “everything will work by itself”. There were only three suppliers: a cabin with all the electronics, a mobile platform and a video system. Soft (Project Magenta if it says something to someone) was supplied by the cab supplier, respectively, and the infrastructure was "given to us from above." As operation showed, it was necessary to at least correctly scatter components on computers, but even this was not done.
In the future, the constant changes (which I will discuss separately) led to the fact that the system healed with its own life. Every time a failure or failure occurred, it was necessary to untangle a node from a multitude of components in order to eventually discover that one of them stopped working if the log file reached a certain size.
There was no question of any regular backup, control of component versions and speech.
As for performance, as the installation of new scenarios, the performance of the simulator fell very significantly, which resulted in the replacement of MSFS with X-Plane. Why exactly this election was made - I do not know, I did not take part in it, but the fact remains.
Even after replacing the monolithic MSFS with a scalable X-Plane, the performance of the graphics subsystem was not calculated, with the result that the hardware of these servers was upgraded twice.
From this disaster, I concluded that it is just necessary to draw the architecture of the entire simulator, paint which component will work on which computer, provide for possible extensions, paint hardware performance requirements, estimate network load and make a small margin for further development. Ultimately, the purpose of the simulator is not the constant development of new technologies, but smiles on customers' faces.
Constant changes and multivendor
Oh, it's just a holiday.
While MSFS was used as a simulator, everything was more or less stable: put a curve to the curve — get the simulator out of the way. Simply and without frills.
When we switched to X-Plane and scaled the system by installing four copies (one considers the dynamics, and each draws his own picture on his projector), it turned out that it is much more difficult to extend the same script to four copies without errors. Especially when you consider that one of them works under Windows, and three - under Linux. Errors and problems began, the flight of scenarios on one of the projectors (the most unpleasant). At some point, the control over the scenarios was completely lost, and a willful decision was made to completely synchronize all installations with one. This made it possible to almost completely get rid of the problems with the visualizer, but the root of the problem remained - we have four different installations of simulators, three of which form an image on one screen.
Sometimes iron (controllers) failed. Instead of reconciling and buying the same thing, controllers from other manufacturers were bought. As a result, after three years, as many as 5 vendors stood in Boeing, and each of them had their own interface software with their own features. Of course, it works together, but in the case of some unknown problems, the diagnosis turns into a living hell.
Again, new controllers are new cables on board, and at some point we came to the realization that we no longer know what the wire is, where it goes and what it is for. I had to stop the work of the simulator for several days, and to do almost complete re-wiring.
From these rakes, I concluded that the simulator should be built, if not for ages, then at least for years. It is necessary to assess in advance the directions of possible development, fix them and build a system based on these directions. If at some point there is a desire to improve something in another direction - well, well, something will have to be abandoned in favor of operational reliability.
Exploitation
The operation of the simulator was not like that at all.
There were no operating instructions, no restrictions on changes, a backup plan, routine maintenance, and there wasn’t much else.
Different hands climbed into the same simulator, changed settings, quickly checked and left, and the next morning the simulator did not start.
The lamps in the projectors burned out at the most inopportune moment, and the spare ones were in the store somewhere in Vidnoe, and on Friday evening our courier rushed through all Moscow in order to buy the last lamp in Moscow (except for jokes).
The oil in the cylinders of the drives of the movable platform imperceptibly flowed out, and we found that it only flows when a puddle has formed on the floor. And, of course, this brand of oil was not to buy.
Summarizing: the simulator worked, but was not exploited. Breaking - fix. Do not break - fly.
In the new project it was very important to foresee and describe the operation process: what kind of spare parts and accessories we need, how often we make diagnostics, what we do if this and that breaks, what updates we install and how, how we test them, how often do we make backups, etc. Looking ahead to say that this approach helped a lot. It turned out that if you agree on at least regular backups and don’t make changes when you hit, the system lives much longer and better.
Electrical problems
It would seem that in such a technically complex device electrical problems should not take up much space.
But no! When I first joined Boeing, spontaneous shutdowns of the controllers appeared, which we could not diagnose for a long time. At some point, I figured out how to measure the potential difference between the “ground” of USB controllers and the computer in which this connector was plugged in: there were many, too many for the controllers, and for the USB hubs on the motherboards of the computers. It turned out that in some unknown way our suppliers managed to unleash the “lands” of the booths and racks with computers. Power supply was made from one point, with the correct layout of the "earth" and neutral - the number of rolls off has sharply decreased.
Another time, we had one of the phases knocked out, and an autopsy revealed that there was no load balancing. After that, I analyzed the power supply circuit of all components of the simulator and redid it, the problem is gone.
The conclusion was the need for the power supply scheme with the calculation and balancing of loads, their distribution among bespereboynikam, and, of course, with protection from short-circuit and from electric shock. At first - on paper, with checks “what will happen if this posting breaks off”, with the calculation of the cross-sections of wires, nominal values ​​of automata and the like electrical things. And only after the scheme has not changed - the purchase of components and the assembly of an electrical panel.
USB
We never exactly considered how many bulbs and buttons are in the cabin, but the number of I / O controllers exceeds a dozen and a half. And each of them ends in USB, and it should turn on in the computer.
I have already written about the problems with the lack of a common “ground”, we have successfully solved it, but when there are three bundles of cables on three different computers, there are no problems to avoid.
We tried several different USB extenders, tried to install hubs from different manufacturers, tried to assemble all devices on one computer, in general, tried a lot of things, but the problem with the USB device “falling off” at the most inopportune moment even though this day.
The obvious conclusion was the rejection of the use of USB, but as I said, in our enlightened age, the market for low-budget simulators (i.e., those that individuals up to $ 50k can afford) is extremely underdeveloped, and USB reigns there. Moreover, none of the suppliers of components have in mind that the cabin can move - almost all (with the exception of units) of such simulators here - are fixed. We built the simulator on a mobile platform, and hence the need for long cables, and problems with vibration.
Well, well, if so, then there is no getting away from it, which means you need to make the cables as short as possible, do not drag the USB through extension cables 10 meters, and put the computers on board. Looking ahead: this meant that diskless computers should be on board. This is a separate beautiful task, which I will discuss in one of the following parts.
Mechanical problems
Mobility + people in the cockpit = various problems with the mechanics.
From vibration unscrew the threaded connections, self-tapping screws fall out, over time the plastic begins to crumble.
Our clients are not professional pilots, but ordinary people, sometimes not even very enthusiastic about aviation. They do not know what to do and what not to do.
An ordinary average man in the prime of his life may unknowingly tear the handle off a real aviation instrument, and not even notice it. However, according to the stories of our engineering friends from the “adult” training centers, real pilots sometimes do the same.
In the new simulator, we won this percent rake by 95%: all cables must be fixed, all threaded connections are locked. No screws. Screed, screed, screed. The remaining 5 percent is still our customers. Alas, here - only a gentle, but persistent word. This I attribute to the organizational conclusions.
findings
Plan, design, write, think, analyze.
"From the sheet" you can probably make a scooter for a couple of evenings, but the meat grinder is no longer there. A flight simulator - and even more so.
Analysis of past failures played an important, if not decisive role. It was thanks to the systematization of the “learned lessons” that the need to think and draw, and then to do, became obvious.
Of course, having avoided parts of the old rake, we found new, but semi-annual experience showed that the operational reliability of the new simulator is much higher. Much less forced downtime, failures and failures. There are much more happy customers, and this is precisely the goal.
In the following posts, I will describe the process of building the A320 in detail.
Update
I found photos from the construction of Boeing, spread.
The base of the cab with hand wheels. Originally used computer power supply, which turned out to be a great evil. Now MeanWell
Arrived movable platform from Italy. The mass of the most mobile platform is about 800 kg, the control cabinet is another 800.
We had the opportunity to use a loader (with Airbas - no longer, there was no room). On the left - the first version of the rack with servers. On the right - metal structures, on which the projection screen will be stretched.
Cockpit assembly. MIP (Main instrument panel) is almost assembled.
Cockpit with base and frame.
View through the windows on the open ceiling panel. Subsequently, it became clear that the chosen method of laying cables was incorrect.
Our Canadian experts configure the interface software - a layer between FSUIPC and controllers. Each button, toggle-lamp-indicator "tied" to the variable FSUIPC. Once we lost this configuration - it was fun.
Start the assembly of the base on which the ladder will ride.
Cab, platform, platform control cabinet. The “heart” of the control cabinet is Siemens PLC. Drive drivers - Baumuller. Drivers with engines deserve a separate article. For example, the driver inside considers the temperature of the motor windings depending on the load on the motor. In addition, there are several temperature sensors in the winding, which, of course, are also connected to the controller. That, however, does not prevent him from issuing an alarm signal and stopping all mobility, if the calculated temperature has jumped out of the permissible limit. A calculation is based on the unique parameters of the engine, which are removed in the process of testing at the factory. When we faced this difficulty (with an estimated temperature), we had to get in touch with the plant.
Platform control cabinet and rack with servers as they were 4 years ago.
And now - rrraz, and we loader hoisted the cabin on the platform.
ORE block (engine controls, Throttle Quadrant in English). Actually, the reason for my appearance in the project. It did not work at all. The block is made of a real block from a Boeing 737-300 by a German craftsman. Unfortunately, the craftsman threw out most of the Kondovoy Boeing mechanics, and replaced it with ridiculous rods, and in some places (no joke) with a lighter cap. A screw terminal decided to pour solder. Used controllers - Phidgets. There are 6 of them in one unit.
If I remember correctly, here we are trying to figure out how food is distributed (there are two of them - 12 and 5) inside the unit. This power is used to drive the motors (from the wipers) that emulate the operation of the traction machine when the handles are driven “by themselves”.
View of the base of the cabin. Pay attention to the number of USB extenders. These were the first, after them were more, and more, and more. Cables are not fixed, this was one of the reasons for the extension of the extension cords.
We pulled the projection screen and reduce the image. TrippleHead2Go was used as an image divider, and Sol7 was used as a software for curving the image under a cylindrical screen.
One of the working moments. If memory serves, we swore on the topic of how to make OREs move properly.
Running in
The final view of the cockpit in 2009.
View outside during operation of the mobile platform.
Update # 2
Right now the Boeing from this post looks like this:
and so
so:
All this is called the word "overhaul."
If someone has time and itching to pick our dusty trainer tomorrow and Friday, write to me in PM.
Source: https://habr.com/ru/post/194150/
All Articles