How we flew away and returned with difficulty: a detailed report on our participation in the KROK flying robot competition
On September 3, 2012, after reading the news about the competitions of flying robots on the Habrahabr website, we understood that it was waiting! By that time, we had already thought for a few months that it was time to translate our many years of interest in robotics into a professional track, and we were looking for an excuse for this.
Therefore, questions about participation did not arise - discussions began. Immediately decided to use a ready-made drone, focusing on the software. At first glance, the conditions seemed simple, so there were no illusions that we could win, there were many teams, only one prize. We decided that we would use only the camera, thus demonstrating our competence in the field of computer vision.
Looking ahead, I want to separately note the high level of organization of competitions. This has never happened in Russia. We are glad that we took part in this event and we are glad that we could win.
')
Almost all the sensors lie: the lidar of the roads, does not see the volume and at the same time becomes blind in clear weather, the sonars interfere with each other and catch incomprehensible reflections, the IR rangefinders and Kinect work only at close distances.
Flying on sensors is possible only for narrow, one might say, laboratory tasks. In addition, almost all existing sensors have limitations on the radius of action.
The camera in this regard is universal. In terms of weight and power consumption, it is comparable to sonars, in terms of speed - with lidars. For distance and resolution, the cameras have no competitors. Virtually all wildlife uses optics, and why should we not dwell on this decision. Of course, effective algorithms are needed to work with the camera, but that’s what we are doing in our professional interests.
At first, in preparation for the competitions, there was practically no free time - all team members worked in different places. Met for discussions or in a cafe, or at someone's home. Since we already had experience with ground robots, we decided to train on them. Pololu 3Pi was equipped with a camera, a communication module, black adhesive tape was stuck on the paper (later black paper strips were glued) in the form of a polygon layout, small copies of the markers were printed and they began to write algorithms.
So imperceptibly crept the time of passage of the first checkpoint. Under the terms of the competition, we had to send a video presentation of the team. They decided to make a video on the background of the Holy Trinity St. Sergius Lavra - the main attraction of Sergiev Posad. Gathered in the morning, before work (we still worked in different places). Temperature: -20. Flourishing. Drifts. With the third double shot something . We looked at them - they laughed, but they didn’t take them back - they decided to make the others happy too.
What was our joy when we received a letter from the organizers of the competition, even if it contained only one line: “Your materials are accepted!”.
At this control point, the organizers laid out videos as they were received, and we were in the first and second ten. Apparently, the other participants of the competition liked our simple plot - a couple of teams in their videos obviously parodied us.
After seeing everything - and this is more than 8 hours! - videos of other participants, we came to the conclusion that many are not aware of the complexity of the task and do not have an understanding of what and how they will do. The following control points confirmed this - out of 237 participants who passed KT-1, 204 remained after KT-2 (to pass it, it was necessary to simply send confirmation that the participant continues to participate in the competition), and after the third - only 72 (here it was necessary show the current and automatically controlled drone).
Experiments with a ground robot proved that it was possible in principle to carry out a mission only around the camera, but it was necessary to proceed to the flights.
At this stage, we already had a Parrot Ar.Drone 2.0 quadrocopter and managed to fly it a bit. The next step we had to do directly to carry out the flight task. It was obvious that if we did not recognize the crosses (take-off and landing markers), there would be nothing to do at the competitions.
Choosing C # as the main development language and armed with a ready-to-use library of interaction with Ar.Drone (written by our compatriot Ruslan Balanuhin), we started developing an automatic control system.
At the first stage, we set ourselves the task of simply hanging over the cross. That is, the drone had to take off, define a cross on the lower chamber and hover above its center, counteracting the air currents that carry it to the side.
We have ordered smaller copies of markers from advertisers (1 m in diameter). They put one cross in the center of the room 4x4 m. And then the first insights began:
All this was like a cold water tub for us ...
Refusing to select lines at this stage, we began to recognize the crosses along the contour. After a couple of weeks of “dancing with tambourines,” we achieved recognition accuracy of 70-80% at any time of the day and in any light, thereby completing the task - the drone took off and hung over the cross. Already at this stage, we realized that you can not trust one recognition - it is necessary to average the result of several recognitions.
In this picture, the red circle is the center of the cross, and the yellow is the projection of the center of the drone.
At this time (one month before the third control point) we were invited to take part in the Spring Competition of Robots, organized by the St. Petersburg Physics and Mathematics Lyceum No. 239 . One of the disciplines on them was the competition of flying robots under the rules of the RobotChallenge "Air Race" .
Such a good chance to get additional flight experience and communicate with other pilots, we could not miss. But there was no time for preparation, I had to improvise.
The goal of the competition is to fly as many correct eights as possible in 10 minutes. According to the regulations, a dotted line was drawn on the floor as a navigation aid. After some thought, we decided to use our experience of hovering over the cross. Only instead of the coordinates of the center of the cross, transfer the upper visible point of the dotted line to the drone control module. As a result, the drone, like a mule for a carrot, tried to hang over this point, and we slipped the next one.
We had only four hours of time for debugging at the real training ground on the eve of the competition, but the decision was not only elegant, but also fruitful - we flew five laps, receiving a prize for second place.
At these competitions, we were waited by another surprise - communication problems. Of the 10 minutes allotted for the attempt, we spent more than half of the time on establishing communication with the drone - he lost it during the flight, went astray. I had to stop him and start the job again. In the process of communicating with other participants, it turned out that this is the standard problem of Parrot drones.
The video shot at these competitions, we used for the passage of CT-3.
Returning home, we proceeded to the next stage - the flight between the markers. After the St. Petersburg competition, it became clear that it was necessary to fly as much as possible to real distances in actual conditions.
We agreed to use the school gym for the summer holidays. They printed life-sized crosses and were again surprised - it is impossible to identify a large cross by the lower chamber, it’s just not visible in its entirety in the lower chamber.
Once again, I had to change the concept - we returned to the definition of crosses along the lines. The algorithm was as follows - first we get all the lines, then we begin to combine the torn ones and filter them by contrast, location (we don’t need crosses on the ceiling and walls), cutting off all unnecessary. After that we find the crosshairs and we are already trying to figure out from them whether this looks like real markers. That is, we filter, we filter and once again we filter. This is terribly gluttonous in terms of resources, but it is reliable enough.
Source Image:
Found lines:
Contrast Filter:
Combine torn lines:
All intersections:
The filter does not look like a cross:
Result:
It's time for the last checkpoint. Under the terms of the competition, we had to go through it at the organizer's training ground.
On the test flight, we signed up among the first. On arrival at CROC, another surprise awaited us - a wind was added to the communication problems. Our drone tritely blown away, but at that time we did not have an algorithm for counteracting wind gusts. As a result, out of twenty attempts we managed to complete the CT-4 task only three times. Thanks to the organizers that they gave us this attempt. We passed KT-4 first.
In an attempt to set up an anti-wind algorithm, we once again ran into a communication problem. Conducting tests on the street in windy weather, we almost broke our drone, losing touch with him.
We had to do something. As we already knew, bad communication is a common problem with Parrot drones. Therefore, we used the method of its solution already found before us - the installation of an external antenna. The external antenna and the router allowed us to consider the question of communication closed. As it turned out later, many teams faced this problem only at competitions.
Three weeks before the final, the organizers finally approved the training ground. All participants were given the opportunity to visit him twice to adjust the algorithms.
After the first visit to the already completed test site, we had the feeling that we would not complete the task. We had three problems:
Attempting to fly through the maze on the manual control of the chamber led to a collision with a wall and crash. The second attempt led to the same result.
All that we did on our first visit was to shoot the landfill at different times - when the sun was behind the clouds and did not give extra shadows, and vice versa - with bright sun and contrasting shadows.
For the rest of the week, we refined the definition of the corridor using the “ vanishing point ” approach.
Arriving for the second time, three days before the competition, we were able to automatically fly the corridor in one direction, but the problem of landing and take-off remained unresolved.
We had two days left. It was not clear how to repeat the gusting wind conditions in order to find out if we won it?
As a way to fight the wind, we used active resistance to external influences during landing - trying to keep the angles of heel, pitch and yaw. And they decided to take off the propeller - at the same time gaining height and turning.
Debugged in the gym, creating gusts of wind with a sheet of fiberboard. It all looked pretty ridiculous.
On August 23, at 21:30, returning from the gym, we raised glasses for having adequately passed all stages of the competition - now we knew that we would complete the task. More precisely, we have a good chance to fulfill it - His Majesty Chance has not been canceled.
We learned to fly in the corridor without colliding with the walls. We recognized crosses well in any weather. Both crosses (I think the participants will understand me). Communication and wind problems have also been resolved.
The only thing that we did not have time is the automatic choice of the direction of the route. To determine turns, we used the right / left hand rule, which was switched after each landing. And it almost played a cruel joke with us.
Having rested against the wall of the obstacles on the way back while performing the flight, we turned around according to the rule of the right hand and flew to the already visited marker. After landing, we switched to the left hand. Having reached the last turn, they turned around again and flew back again. And only after the third landing, the rule of the right hand turned on again, and we returned safely.
We understood that 6 minutes and 45 seconds, obtained by us in the first attempt, is a losing time - it is enough for someone to just complete the task. In this case, the rule of the right / left hand, we could not cancel, and there was a chance that the drone again get off and start circling. Then at the second attempt we decided to “fly all the money” and raised the speed of the drone one and a half times. But, unfortunately, they forgot to change the speed reduction parameter, depending on the distance to the obstacle, and did not manage to brake in time, finally stopping against the wall.
It was really unexpected for us that no one else could go the distance. For a day and a half, we waited for someone to fly faster than us (the organizer's drone had the best time, but they participated in the competition off-set), and we can already relax and rest, sincerely wishing other participants to succeed. But luck was capricious, and this time it was on our side.
Below is a flight record from the drone. The recognition speed was uneven, so the time on the video does not coincide with reality.
Therefore, questions about participation did not arise - discussions began. Immediately decided to use a ready-made drone, focusing on the software. At first glance, the conditions seemed simple, so there were no illusions that we could win, there were many teams, only one prize. We decided that we would use only the camera, thus demonstrating our competence in the field of computer vision.
Looking ahead, I want to separately note the high level of organization of competitions. This has never happened in Russia. We are glad that we took part in this event and we are glad that we could win.
')
Why camera?
Almost all the sensors lie: the lidar of the roads, does not see the volume and at the same time becomes blind in clear weather, the sonars interfere with each other and catch incomprehensible reflections, the IR rangefinders and Kinect work only at close distances.
Flying on sensors is possible only for narrow, one might say, laboratory tasks. In addition, almost all existing sensors have limitations on the radius of action.
The camera in this regard is universal. In terms of weight and power consumption, it is comparable to sonars, in terms of speed - with lidars. For distance and resolution, the cameras have no competitors. Virtually all wildlife uses optics, and why should we not dwell on this decision. Of course, effective algorithms are needed to work with the camera, but that’s what we are doing in our professional interests.
Born to crawl
At first, in preparation for the competitions, there was practically no free time - all team members worked in different places. Met for discussions or in a cafe, or at someone's home. Since we already had experience with ground robots, we decided to train on them. Pololu 3Pi was equipped with a camera, a communication module, black adhesive tape was stuck on the paper (later black paper strips were glued) in the form of a polygon layout, small copies of the markers were printed and they began to write algorithms.
Three of the casket
So imperceptibly crept the time of passage of the first checkpoint. Under the terms of the competition, we had to send a video presentation of the team. They decided to make a video on the background of the Holy Trinity St. Sergius Lavra - the main attraction of Sergiev Posad. Gathered in the morning, before work (we still worked in different places). Temperature: -20. Flourishing. Drifts. With the third double shot something . We looked at them - they laughed, but they didn’t take them back - they decided to make the others happy too.
What was our joy when we received a letter from the organizers of the competition, even if it contained only one line: “Your materials are accepted!”.
At this control point, the organizers laid out videos as they were received, and we were in the first and second ten. Apparently, the other participants of the competition liked our simple plot - a couple of teams in their videos obviously parodied us.
After seeing everything - and this is more than 8 hours! - videos of other participants, we came to the conclusion that many are not aware of the complexity of the task and do not have an understanding of what and how they will do. The following control points confirmed this - out of 237 participants who passed KT-1, 204 remained after KT-2 (to pass it, it was necessary to simply send confirmation that the participant continues to participate in the competition), and after the third - only 72 (here it was necessary show the current and automatically controlled drone).
Experiments with a ground robot proved that it was possible in principle to carry out a mission only around the camera, but it was necessary to proceed to the flights.
How amazing the world is
At this stage, we already had a Parrot Ar.Drone 2.0 quadrocopter and managed to fly it a bit. The next step we had to do directly to carry out the flight task. It was obvious that if we did not recognize the crosses (take-off and landing markers), there would be nothing to do at the competitions.
Choosing C # as the main development language and armed with a ready-to-use library of interaction with Ar.Drone (written by our compatriot Ruslan Balanuhin), we started developing an automatic control system.
At the first stage, we set ourselves the task of simply hanging over the cross. That is, the drone had to take off, define a cross on the lower chamber and hover above its center, counteracting the air currents that carry it to the side.
We have ordered smaller copies of markers from advertisers (1 m in diameter). They put one cross in the center of the room 4x4 m. And then the first insights began:
- It turns out that there are a lot of crosses around us!
- It turns out that black and white are relative concepts!
- It turns out that the standard line selection algorithms for OpenCV find lines even where we could not have imagined!
- It turns out that the camera comes with broken frames!
All this was like a cold water tub for us ...
Refusing to select lines at this stage, we began to recognize the crosses along the contour. After a couple of weeks of “dancing with tambourines,” we achieved recognition accuracy of 70-80% at any time of the day and in any light, thereby completing the task - the drone took off and hung over the cross. Already at this stage, we realized that you can not trust one recognition - it is necessary to average the result of several recognitions.
In this picture, the red circle is the center of the cross, and the yellow is the projection of the center of the drone.
Carrot for drone
At this time (one month before the third control point) we were invited to take part in the Spring Competition of Robots, organized by the St. Petersburg Physics and Mathematics Lyceum No. 239 . One of the disciplines on them was the competition of flying robots under the rules of the RobotChallenge "Air Race" .
Such a good chance to get additional flight experience and communicate with other pilots, we could not miss. But there was no time for preparation, I had to improvise.
The goal of the competition is to fly as many correct eights as possible in 10 minutes. According to the regulations, a dotted line was drawn on the floor as a navigation aid. After some thought, we decided to use our experience of hovering over the cross. Only instead of the coordinates of the center of the cross, transfer the upper visible point of the dotted line to the drone control module. As a result, the drone, like a mule for a carrot, tried to hang over this point, and we slipped the next one.
We had only four hours of time for debugging at the real training ground on the eve of the competition, but the decision was not only elegant, but also fruitful - we flew five laps, receiving a prize for second place.
At these competitions, we were waited by another surprise - communication problems. Of the 10 minutes allotted for the attempt, we spent more than half of the time on establishing communication with the drone - he lost it during the flight, went astray. I had to stop him and start the job again. In the process of communicating with other participants, it turned out that this is the standard problem of Parrot drones.
The video shot at these competitions, we used for the passage of CT-3.
Filter, and everything will turn out
Returning home, we proceeded to the next stage - the flight between the markers. After the St. Petersburg competition, it became clear that it was necessary to fly as much as possible to real distances in actual conditions.
We agreed to use the school gym for the summer holidays. They printed life-sized crosses and were again surprised - it is impossible to identify a large cross by the lower chamber, it’s just not visible in its entirety in the lower chamber.
Once again, I had to change the concept - we returned to the definition of crosses along the lines. The algorithm was as follows - first we get all the lines, then we begin to combine the torn ones and filter them by contrast, location (we don’t need crosses on the ceiling and walls), cutting off all unnecessary. After that we find the crosshairs and we are already trying to figure out from them whether this looks like real markers. That is, we filter, we filter and once again we filter. This is terribly gluttonous in terms of resources, but it is reliable enough.
Source Image:
Found lines:
Contrast Filter:
Combine torn lines:
All intersections:
The filter does not look like a cross:
Result:
Take mail, telephone, telegraph
It's time for the last checkpoint. Under the terms of the competition, we had to go through it at the organizer's training ground.
On the test flight, we signed up among the first. On arrival at CROC, another surprise awaited us - a wind was added to the communication problems. Our drone tritely blown away, but at that time we did not have an algorithm for counteracting wind gusts. As a result, out of twenty attempts we managed to complete the CT-4 task only three times. Thanks to the organizers that they gave us this attempt. We passed KT-4 first.
In an attempt to set up an anti-wind algorithm, we once again ran into a communication problem. Conducting tests on the street in windy weather, we almost broke our drone, losing touch with him.
We had to do something. As we already knew, bad communication is a common problem with Parrot drones. Therefore, we used the method of its solution already found before us - the installation of an external antenna. The external antenna and the router allowed us to consider the question of communication closed. As it turned out later, many teams faced this problem only at competitions.
Three weeks before the final, the organizers finally approved the training ground. All participants were given the opportunity to visit him twice to adjust the algorithms.
Blanket for shish kebab
After the first visit to the already completed test site, we had the feeling that we would not complete the task. We had three problems:
- Definition of the corridor. The dividing partition was organized using building blocks that interfered with our wall detection algorithm.
- Landing in the wind. We carried out the landing operation, it can be said, with closed eyes: we find a cross, heal it, and we land (we fly two or three meters and sit down). However, any gust of wind blows us away from the site.
- Take off and turn in the wind. We also did this operation “on the machine” - as a result, we could be thrown to the wall by the wind (for some reason, we thought that it was forbidden to touch the walls).
Attempting to fly through the maze on the manual control of the chamber led to a collision with a wall and crash. The second attempt led to the same result.
All that we did on our first visit was to shoot the landfill at different times - when the sun was behind the clouds and did not give extra shadows, and vice versa - with bright sun and contrasting shadows.
For the rest of the week, we refined the definition of the corridor using the “ vanishing point ” approach.
Arriving for the second time, three days before the competition, we were able to automatically fly the corridor in one direction, but the problem of landing and take-off remained unresolved.
We had two days left. It was not clear how to repeat the gusting wind conditions in order to find out if we won it?
As a way to fight the wind, we used active resistance to external influences during landing - trying to keep the angles of heel, pitch and yaw. And they decided to take off the propeller - at the same time gaining height and turning.
Debugged in the gym, creating gusts of wind with a sheet of fiberboard. It all looked pretty ridiculous.
I drink to the bottom ...
On August 23, at 21:30, returning from the gym, we raised glasses for having adequately passed all stages of the competition - now we knew that we would complete the task. More precisely, we have a good chance to fulfill it - His Majesty Chance has not been canceled.
We learned to fly in the corridor without colliding with the walls. We recognized crosses well in any weather. Both crosses (I think the participants will understand me). Communication and wind problems have also been resolved.
The only thing that we did not have time is the automatic choice of the direction of the route. To determine turns, we used the right / left hand rule, which was switched after each landing. And it almost played a cruel joke with us.
Hedgehog in the fog
Having rested against the wall of the obstacles on the way back while performing the flight, we turned around according to the rule of the right hand and flew to the already visited marker. After landing, we switched to the left hand. Having reached the last turn, they turned around again and flew back again. And only after the third landing, the rule of the right hand turned on again, and we returned safely.
We understood that 6 minutes and 45 seconds, obtained by us in the first attempt, is a losing time - it is enough for someone to just complete the task. In this case, the rule of the right / left hand, we could not cancel, and there was a chance that the drone again get off and start circling. Then at the second attempt we decided to “fly all the money” and raised the speed of the drone one and a half times. But, unfortunately, they forgot to change the speed reduction parameter, depending on the distance to the obstacle, and did not manage to brake in time, finally stopping against the wall.
It was really unexpected for us that no one else could go the distance. For a day and a half, we waited for someone to fly faster than us (the organizer's drone had the best time, but they participated in the competition off-set), and we can already relax and rest, sincerely wishing other participants to succeed. But luck was capricious, and this time it was on our side.
Below is a flight record from the drone. The recognition speed was uneven, so the time on the video does not coincide with reality.
Source: https://habr.com/ru/post/193304/
All Articles