📜 ⬆️ ⬇️

The driver of the rover Curiosity meets Habra

It happened! The long-awaited answers of the “driver” of MSL Curiosity to the questions that Habr asked him. Paolo Bellutta has worked with Opportunity and Spirit, so he has a wealth of experience, and most importantly, he is not shy about talking about him.

This wonderful translation is by Singerofthefall . Paolo sent the text in bulk, so we divided the answers in half, and the translator himself will publish the second part. Therefore, you can thank him now, and you can thank him later, when he finishes his work on the second part and puts it out. [1]
[1] In brackets translator's notes.
I laid out the full English text on the Google Plan, and anyone who wishes can refer to it, but believe me, this is not necessary because the translation is excellent.

Let's begin our interview:


')
Q: What is your work schedule?
A: After the first 90 days, during which we lived Martian time, we switched to the usual schedule. Usually, a business day lasts from 8 am to 8 pm Pacific Standard Time (PST). We work every day, including Saturday and Sunday, but rested, for example, on Thanksgiving Day and Christmas. Before the holidays we prepared the teams for a few days in advance, so we managed to stay at home, but the rover had to work without a break. Soon we will switch to the 6-day working week, and after the confrontation of Mars [the moment when the Sun is exactly on the line between the Earth and Mars, will be in April ] we will most likely switch to the usual five-day one. Although the rover will still work every day, he [1] has no days off :-(

[1] [ Paolo speaks of the mars rover in the feminine - she - apparently by analogy with the ships to which this pronoun is also used in the English language. Although now they practically animate it and treat it not as a ship, but as a girl :)

Elevator doors in JPL NASA.


Since this is unusual for a Russian audience, I will use the pronoun “he” - a rover ]

Q: How is the code checked for the motion program for the next day? How many people check the code before sending it? Are emulators used to test the motion program before being sent to the rover? What do they do if they find errors in the sent movement program?
A: Great questions! For the preparation of teams, we use software specifically developed for these purposes in JPL. We have a specialized editor called RoSE (Robot Sequence Editor - the robot's [ action ] sequence editor), which finds the simplest errors - typos in the command names, errors in the range of parameter values, and so on. A simulator called Hyperdrive is connected to the editor. He receives images that were made in previous Soly, and shows a three-dimensional image of the surrounding rover terrain. Then the simulator receives a list of commands and shows what the rover will do and how it will interact with the environment. You can also simulate basic telemetry, such as location and direction.

For the most important tasks, such as drilling with a manipulator, we sometimes use a prototype.


When it seems to us that the command set contains no errors, we give it to other drivers for checking. Usually, at least three people work for each shift (a driving specialist, a specialist in the use of a robotic arm, and a specialist in the use of tools), but sometimes there are more people left and they all check the prepared teams. During the shift, we also have five formal checks that are conducted by a group of a dozen people, and not all of them are drivers of the rover, because during daily work we have to use various tools, and we need to make sure that none of the teams can cause a failure of the equipment, or, what is more, damage to the rover. We also have software for additional verification of commands, and a long list of things that need to be checked each time.

We try not to make mistakes, but sometimes they happen, and quite a large part of the software on the rover should check that the commands are not meaningless, and that we are not trying to do any stupid things like a laser shot at the rover itself. So, even if we make a mistake here on Earth, the rover on Mars will detect it.

Q: How is the testing on the ground (is there a copy of the rover in the desert on the home planet?)
A: We have two devices. The first one is called VSTB (Vehicle Surface Testbed), and all equipment is installed on it, including the cameras and the arm, so we can manage it exactly the same way as the rover on Mars. Every time we need to ensure the special reliability of the program, we use it. Here - photojournal.jpl.nasa.gov/catalog/PIA15876 - there is his photo. The second unit, called the Scarecrow (Scarecrow), is equipped only with wheels and an engine. We use it to test moving across different types of surfaces.

We in JPL also have two polygons, one of them is in the building, it is called the In-Situ Instrument Laboratory:
image

And the other - in the open air - Mars Yard, where we can conduct more tests on the movement.

( at 0:33 in the frame Paolo flashes - cable pulls )

In May 2012, before landing the rover, we drove the Scarecrow into the desert near Death Valley in California to see how it behaves on the sand, as we were worried that the rover could land on the sand dunes.

Q: What do programmers train on?
A: We teach each other. Someone has experience in management, someone helped create the rover, someone participated in writing software. Believe me, all this requires a lot of experience and practice.


Q: Is it planned, after the Curiosity rover has completed the entire planned research program, to provide part of the rover's resources and time (if it is working at that time) to students, to carry out teaching scientific experiments?
A: In fact, a huge number of scientists and students from various universities are working on this mission. Their task is to decide which experiments we should carry out and follow the progress of their implementation. Honestly, I don’t think that we will ever finish all our experiments at all, as their list is constantly growing - as soon as we find the answer to one of the questions, we immediately have 10 others!

Q: Does NASA plan to visit the rover for landing sites of previous Mars exploration vehicles? Most of them have already failed, but it would be interesting to establish the true cause on the spot. Moreover, the photographs of these once legendary devices would have a very high aesthetic, and also material value.
A: At the moment, our main goal is to explore as much of the planet as possible, so going back to where we have already been would not be very useful, even if we come back with more tools. Understanding the reason why our old robots are out of order is still less important than collecting new scientific data. So, although I share your desire to see these devices again, I do not think that this will happen in the near future.

Q: How do you solve the problem of a 7 minute signal delay? And is this a big problem?
A: The signal delay due to distance varies from 5 to 20 minutes one way. Now this time is close to the maximum, and is about 19 minutes. Of course, the delay is too great for us to control the rover interactively. Think about it - if you see an obstacle in front of the rover and immediately send a command to stop, then even with the least possible delay you will be 10 minutes late! Therefore, we manage it by sending a sequence of commands every day. In addition, we could not control the rover interactively also because the Earth is not always visible from the Gale crater, in which Curiosity is located. And finally, even if you do not take all this into account, it costs a lot to call the rover - about $ 10,000 per hour!

In general, the delay before receiving data is usually even greater, since we transmit the signal through the Mars Odyssey and Mars Reconnaissance Orbiter satellites. It usually takes at least several hours to get the data from the rovers.

Q: Is there any tool or device that is very lacking in “Curiosity” in your work?
A: More scientific instruments, a more powerful computer, more storage space for data, more options for transferring data, more power, higher resolution cameras, higher mast ...

Q: Have there ever been cases when you wrote a program for a day, but already during its execution, it became clear that it was not a critical error that crept in to the program, which is too late to correct, and if not late, how do you fix it?
A: Of course! At MER, we found some bugs in the years after landing. One of the critical bugs was found on the Spirit Mars rover on the 18th Saul, and the other we found in 2007, when we toured the Crater Victoria. We have a team of specialists (including myself) that analyzes rover telemetry. Every time we find something that seems strange to us, we go to the test bench and try to recreate the conditions and see if the problem will manifest. When we find the cause, we can either change one of several thousand software parameters, or send a patch, which is a small piece of code designed to solve the problem. Well and, obviously, after the problem is solved, we try to avoid conditions in which its repetition is possible.

Q: What is the most difficult in your work?
A: Well, besides all the paperwork, which is part of the management of Curiosity ... ;-) It’s hard to imagine all kinds of problems. Of course, we have a lot of experience managing the rover on Mars, but we still have to always be on our guard, in case something goes wrong. For example, when we move a rover, we always try to leave at least one path, which, if necessary, it will definitely be able to go back. We also try to avoid actions that, even with a very low probability, can lead to disastrous consequences.

Q: What are your curiosities or unusual situations?
A: One of the most amusing incidents occurred in 2006 during the MER. In general, this mission was designed only for 90 Solov, and when software was written, only three digits were taken for storage of the Sol number. As the 999th Sol approached, we prepared a software update, and uploaded a new version to the rovers. It was a titanic job - it took us a whole year to write the update and testing, plus it took several months to download the update to the rovers. The new on-board software [2] also required changes in the ground-based software [3], since it was incompatible with its old version. Therefore, we had to reload both rovers into the same Saul, so that it did not happen that one of them was already working on the new software, and the other was still on the old one. So, this day finally comes. Spirit gets the honor of rebooting first, we send a command to reboot with the new software, the rover sends the answer, everything is great. Opportunity at this time is on the other side of the planet, and we have to wait 12 hours before we can send him the right team. And just before that, the backbone [ trunk line ] in JPL is turned off, and without it, we cannot send commands from our servers to the DSN station [Deep Space Network]. So, time is running out, and backbone still does not work. What to do?! Fortunately, one of the computers in the control center had a straight line with a DSN station (in Canberra, it seems), and this computer also had a ... floppy drive! Fortunately, one of the MER computers also has a floppy drive, but who has a floppy disk in 2006? But there was nothing else left, and we started running around the lab looking for a floppy disk. We eventually found the diskette in some forgotten drawer of the table, quickly reformatted it, copied the commands, and ran to the control center to send them. In the end, we even have two minutes left!

[2] [ Paolo uses the term flight software, which in aviation and astronautics usually means “flight control software”. In this case, refers to the part of the software that is located on the rover itself. ]
[3] [ Paolo uses the term ground software, and means the part of the software that is used on Earth ]

Q: How many people manage the rover?
A: Well, it depends on what you mean by "govern." Now we have 16 drivers, but the team of people who operate various equipment (cameras, SAM, CheMin, ChemCam ...) has, I think, almost 100 people.

Q: Is there a built-in "anti-fool" on the rover?
A: You know, as they say, there can be no complete “protection against a fool”, because fools are too creative.

Q: If a programmed movement of a manipulator arm can carry, for example, some antenna or just crash into the rover body, will such a program be automatically blocked (for example, after checking on a computer model), or do you have to monitor it manually?
A: The rover software checks if any parts of it can collide with others, so in this case the command will simply not be executed, an error will be generated, and the rover will stop the manipulator. If any of the following commands depends on the position of the manipulator (for example, a shot from a laser), then these actions will not be executed either.

But even considering that this software protects the rover, we still double (and three times, and four times) check that this could not happen. We also try to make sure that this cannot happen even if the manipulator, or other parts of the rover, are not exactly where they should be. Sometimes it is impossible to predict the exact conditions on Mars. Of course, we have cameras, and a 3D map of the area, but this data can always contain inaccuracies associated with measurement errors. Therefore, we need to make sure that the programmed actions will be performed safely even if the error is high enough.


Q: How is the relationship with the rover? What antennas / modems are used for this? What are low-level protocols? How do higher level protocols cope with errors when transmitting information?
A: We send commands from the Earth directly to the rover's antenna, which is called HGA (High Gain Antenna), using one of the DSN stations. Communication is in the X-band (7-8GHz). Here in this PDF you can find all the information you can imagine.

The data that the rover collects for the day is sent back using the UHF antenna [4] - it looks like a black cylinder to the right of the RTG [it is black at the terrestrial copy, and on Mars it is in silver foil, RTG is RTG, Note Zelenyikot ] one of the satellites (MRO or ODY), after which the satellites transmit data to DSN stations (again in the X-band, if I'm not mistaken). Usually the amount of data we received from MER was about 100 megabits. Much more data comes from MSL - an average of about 500 megabits, but once we had a huge transfer of 1200 megabits.

[4] [ Ultra High Frequency ]

Q: What are the main features and differences of photographing in Martian conditions compared to earth? What additional knowledge should a Martian "photographer" have? For example, does the dominance or degradation of certain colors occur differently compared to what we are accustomed to see on Earth, just as these differences are noticeable in underwater photography? First of all, photographs that are familiar to the human eye, and not photographs reflecting the invisible part of the spectrum, are of interest.

A: Mike Caplinger [ one of the people responsible for the cameras ] could provide this question with much more information, but I will certainly try to answer.

The atmosphere of Mars is much thinner than that of the Earth, and much more predictable. There is a certain amount of dust in the atmosphere, so at long distances you will see some deterioration of the picture; sometimes the atmosphere looks like the dirty air in the bay of los angeles. Being farther from the Sun than the Earth, Mars receives a little less light, but in general, it seems to me, there are no significant differences. In fact, making photos on Earth is harder than on Mars, since here you will find much greater differences in color, brightness and contrast.

Q: Was the rover shooting program yourself more difficult than other daily programs?


[ It seems Paolo did not understand the question here. We tried to ask about the self-portrait, and he answers about this video - Note. Zelenyikot ]:


A: Cameras designed by Mars Space Science Systems are designed to capture video at a frequency of up to 15 hertz (or frames per second, if you wish), and they have built-in memory, so for us this does not pose any problems. The only thing that can potentially be a problem is the length of the video, the frame rate, and to make sure that the video starts at the right moment, and the cameras are turned in the right direction. Again, Mike Caplinger could answer this question in more detail than I do.

Q: Are the algorithms provided for independent recovery from an abnormal situation, or does the program get interrupted when it occurs and the device enters standby mode?

A: There are several levels of action in the event of a contingency. If an error is detected in those commands that we sent, then the rover will simply refuse to carry them out. If a problem occurs during the execution of a command (as in the example with the hand I mentioned above), then the rover will send an error message, and simply refuse to use one or more devices. Finally, if there is any bug in the software, then the rover will reboot and go into “safe mode”. In this case, it will perform a pre-programmed set of actions, including sending an error message, and will wait for instructions from Earth to continue. In case the software hangs, we have watchdog timers that can restart the rover and put it into safe mode. So we do not have a blue screen of death.

Q: What parameters of the rover are you tracking for this in "real time" mode or close to it?
A: I hope the answer to the question above answers this question.

Q: When does the security system of the rover independently terminate the program?
A: It seems that we had one reboot of hardware on MSL, but I don’t remember what the reason was. MER rebooted many times, including repeated cases caused by radiation effects on the processor and memory. Depending on the reason for the reset, such situations are considered as minor (as in the case of cosmic radiation), or as significant (as, for example, the Spirit anomaly on the 18th Sol).

Q: Are there any algorithms for independent recovery from abnormal situations?
A: As I just said, it depends on the reasons that caused the abnormal situation.

Q: Does Curiosity use the data protocol / DTN architecture and, if so, what are the minimum delays in packet transmission?
A: The minimum delay is 5 minutes, the maximum is 20, but since we do not work with the rovers in real time, this delay does not affect the control. We do not use DTN, but we have some technologies that help establish communications over such long distances. Usually, for one Sol, we collect much more data than we can transfer, so we have to choose which data will be sent to Earth in each Sol. Each piece of data receives a priority, and the most important ones will be transferred first, and less important ones will be sent later (or not sent at all). Of course, if we suddenly decide that the importance of the data was incorrectly assessed, then we can change our priorities. For example, if on the thumbnail we see that the camera was turned in the wrong direction, then we either lower the priority or delete the full-sized image completely, and vice versa: if we have an image with the best exposure, then its priority will be increased. Moreover, the data is divided into packets - this is important for transferring large amounts of information, for example, the same images. That is why some of the images you saw had holes. Some of the data could be lost during transmission, or could be transmitted with errors. Of course, in case of errors, we can try to transfer the data again.
<.....>

As you can see, this part of the interview was more professional. Here we learned Paolo as a JPL specialist. I distributed the questions so that at first they were serious, and then not quite. The second part of the serious ones was also enough, but she will tell more about Paolo as a person, as well as about the relationship between Curiosity and the Martians, people in black who hire NASA, and whether the driver of the rover dreams of Mars.

If you really want to continue, I can still offer to watch this small interview with Paolo on the NASA website, and really have patience and wait a couple of days when the translation work is complete. Thanks again, Singerofthefall for his work.

Continued.

Source: https://habr.com/ru/post/170865/


All Articles