The data center in the suitcase: one day in the life of an IT specialist in auto racing

Imagine that you have to carry a data center everywhere. Daniel Martland does it, that's his job. He is an IT engineer of the Formula 1 racing team Scuderia Toro Rosso and together with it visits to more than 20 countries of the world annually. According to him, this is not easy, but after each race he feels an amazing sense of satisfaction from the feeling of a job well done.

We met at the international racetrack Sepang during the Formula 1 Grand Prix in Malaysia. We sat at the boxes of Scuderia Toro Rosso, in an incredible heat and high humidity, and Daniel told how the huge amount of data is processed in the most technologically advanced sport on earth and what is required to ensure the operation of such a system ...


Daniel Martland, IT engineer for the Formula 1 racing team Scuderia Toro Rosso, explains what is needed to support the performance of IT systems for racing infrastructure.
')
Daniel, thanks for taking the time. Formula 1 race cars are flying past us and you are obviously quite sure that the Scuderia Toro Rosso network is in perfect order. Can you tell us a little about how the IT infrastructure of Formula 1 is created?

A lot of work. The main race of Formula 1 is held on Sunday, but we have been here for a week. After we arrive at the circuit, we need two days to arrange the whole team: offices, a fence separating the pit lane from the racetrack, a garage - and all this needs to be connected to each other by a local network. Like the rest of the teams, we bring with us our stand-alone portable data center to create a network. It needs to be unpacked, connected and tested. From the moment when it was last turned on, a week could have passed, so we have to make sure that everything works perfectly - as we need. A portable data center always stays in the transport compartment so that it can be easily loaded onto the plane and delivered to where we are going. Here all the server hardware that may be required during the race weekend.


Daniel Martland in the transport department.

And what happens if the portable data center does not get to the race?

This has never happened before, but if the center does not fall on the track, we have some additional equipment in reserve at the base in Faenza. Theoretically, it can be delivered where it is needed.

And after each race, do you send the portable data center back to Faenza?

Yes. If the races are held not one after another, and we have a break, we send the equipment to Faenza for maintenance. Accordingly, out of season, the data center is located in the same place - for maintenance and modernization of equipment.

Have you had any problems with the data center, and if so, how did you deal with them?

There were several cases, but small, nothing special. This is usually associated with the operation of uninterruptible power supplies or abbreviated UPS. They give us a lot of trouble, because they do not tolerate heat and humidity, and in Malaysia, where we are now, such conditions are exactly. Another problem unique to Formula 1 is carbon dust. For some reason, it is bad for electronic equipment. Heat, humidity and carbon dust are a bad combination of conditions for server hardware.


Server racks in the Scuderia Toro Rosso portable data center.

And what happens when a UPS breaks down?

We had several cases in which the UPS broke, but this is more a nuisance than a real problem. We have, of course, not one UPS. We have a “cold” and “hot” reserve. If something breaks during the race, the power is automatically connected and everything continues to work.

As I understand it, a UPS is needed not only in the event of a power failure, but also to stabilize the power supply?

Yes. On some Formula 1 autodromes, power and voltage are unstable. For our network we need a stable 50 Hz.

What servers do you have here?

We have eight physical servers plus regular network equipment. Of these eight, six belong to us and two belong to Ferrari, our engine supplier.

VMware and Windows are installed on our servers, and everyone has their own tasks. For VMware, we use HP servers connected through a storage network to an HP storage array. We have two storage arrays for load balancing and provisioning. Storage of this data is carried out on solid-state drives. They have not only better performance, but they are also less susceptible to vibration.


Engineers team Scuderia Toro Rosso analyzing telemetry data.

As I understand it, on one of these VMwares is there a virtual Atlas database server?

Yes, we have several Atlas servers. These are all servers in virtual machines. Atlas servers are critical to our race, as they process telemetry data, we provide resilience to VMware failures on each machine. So, if one of your VMs crashes when the machine is on track, telemetry data will still come in anyway.

When does telemetry begin?

Immediately at the time of launch. The Atlas server immediately starts recording data. Engineers perform several telemetry checks throughout the day, and start the car several times to make sure all sensors are working.


Daniel Martland shows the schedule of the location of cars on the track.

Are there any sensors that are particularly difficult with?

Still would! When you have more than 200 physical sensors and thousands of measurements, you can expect one of them to break! Fortunately, our system works perfectly.

And what do you do when it seems to you that something looks wrong?

[Laughs] We just press a key on the screen and move on! Seriously, our engineers always know exactly what the problem is.

By the way, about sensors, we have a lot of virtual sensors. The data from the machine goes to the electronic control unit. All available information on vehicle telemetry enters the virtual control unit, where all the necessary calculations are made. As you understand, there are places in the car where it is impossible to put a physical device, so we have to predict what may actually happen there. There are thousands of virtual sensors that process data in this way.


The graph shows the location of cars on the track.

Now tell, please, about your Windows computers, what are they for?

One of them is a backup server. Since we work in partnership with Acronis, you will hear a lot about this ! It has a storage of information of more than 60 terabytes on hard drives, since such storage is necessary for our work. To ensure reliability and protection from hardware failures, we use a RAID array.

Is this your only backup server?

Yes, this is the only server that we carry with us.

Do you copy data when you are out of place?

Returning home, we archive and transfer backups from the backup server in the portable data center to our backup servers on the base. Unfortunately, since we generate a huge amount of data, we are not able to transfer backups over the Internet while we are far from the base.

However, telemetry data is transmitted in real time to the base during each race. We have a channel with a 30 Mbps MPLS protocol connecting the racetrack and the Scuderia Toro Rosso base in Faenza.

Who provides you bandwidth?

For all races we have one provider - Riedel Communications. They work with local telecommunications companies to provide very high quality communications in each country. They are also physically present here, so if something goes wrong, we can go and ask them a beating! [Laughing]

It is very good that their people are on the racetrack. Just think: if you lose your MPLS channel, you lose the connection. You also cannot send a request to technical support via e-mail, since there will be no e-mail!


Part of the server rack team Scuderia Toro Rosso

What other services do you use?

Only standard set. The FIA ​​provides GPS data and weather information. We also use the services of Formula 1 management: getting information about time, television, etc.

You talked about the organization of the system. Now share, please: here you wake up in the morning on the weekend, when the races are held. What occupies your mind at this time?

Well, the very first thing I do when I get out of bed is to check the phone. I’m not looking at likes on my Facebook page, but I’m checking if there are any problems with our systems. I receive a lot of status messages and SMS messages from different servers. Some warnings are informational, such as backup status, but others may be more serious. So, I get up, check my mail and then worry all the way while I am going to work! [Laughing]

On Sunday, when the main races pass, a lot of nerves are spent! I am always very nervous when I arrive at the race track.


It remains less than those hours before the Formula 1 championship race - the Malaysian Grand Prix. Final preparations

So, you enter, slowly turn on the monitors and hope that all is well?

No, the system is actually very reliable. Arriving at the place, I study the daily checklist. I check if everything works. And then - the same thing again! [Smiles]

What do you use for monitoring?

Mostly Cacti with many different plugins. I like this program because it can be adapted to your needs. We experimented with various other monitoring programs, but have not yet found what we like.

Have you tried Acronis Monitoring?

Not yet. But I like what I saw at the moment. I like the idea of ​​complete data protection when data backup, file synchronization, and monitoring are provided by a single provider. Acronis is a leader in data backup , and soon we will find out if it is the “very-most” and in server monitoring!

So, you come to work, check the charts to make sure everything is in order and the charts are green?

Not really. We always have red graphics, but this is because some systems are not used or disabled. For example, our IT configuration in Europe is somewhat different. We use some technical means only for the European stages of Formula 1 racing and nowhere else. In Europe, we do not deliver everything by plane: we use trucks for transportation, and this is a bit different.



In your opinion, how will Acronis make your job easier?

For an IT engineer like me, backup speed is everything. Description of Acronis new products sounds exciting. Technologies like Acronis Instant Restore , that is, technology that provides the possibility of a very fast recovery - this is exactly what we are looking for. When something really does not go well, the very knowledge that your data is protected is the best feeling that a sysadmin can experience.

Thank you very much for your time and success to your team!

Thanks you!

PS Heroes of Formula 1 at MIPT!

On November 17, there will be a meeting with Formula 1 driver Daniil Kvyat, one of the youngest riders of the Scuderia Toro Rosso team, and team leader Franz Tost. MIPT graduates Sergey Belousov (CEO of Acronis) and Stanislav Protasov (co-founder and head of software development at Acronis) decided to make such a gift for the 70th anniversary of their alma mater and all physicists.

During the unique public talk, Sergey and Daniel will talk about the role of modern information technologies in the world of Formula 1, the technologies that came into our life from motor sports and the future of the automotive industry and IT. And also about the role that the blockchain can play in all this in the near future.

Guests will have the opportunity to feel like a Formula 1 pilot. In the hall of the KZ there will be installed simulators of combat cars Scuderia Toro Rosso, where everyone will be able to experience incredible emotions from speeds on the tracks around the world, be it Singapore, Malaysia, Tokyo or Sochi.

Professional photographers will help keep bright frames, and the most successful photos can be printed on magnets. For those who get hungry or come to a meeting right from the training, a pleasant coffee-zone Theory Cuisine and Bar will be organized: tea, coffee and delicious muffins according to plan.

Come, try your hand and find out that modern racing and IT are closer than you think!

»The meeting will take place on November 17, 2016 at the KP MIPT.
»The event starts at 18:00 (agreed, set a little earlier for people to gather).
»Admission is free, prior registration is required .
"But do not forget that the number of seats by invitation in the hall is limited.