We control the webcam with the joystick

Introduction

Lyrics

Good day. Motivated by numerous posts on Habré about self-made robots, he decided to do something more or less worthwhile and interesting.

In general, I have been fond of robots for a long time, but my hands did not reach the normal design, basically I was only playing. A little thought, invented his project, looked for details, drew sketches, fantasized about the future possibilities of the robot. I ordered the parts from a well-known site, and so far, the details are overcoming the path from the heavenly king decided to implement one of the modules of the future robot from what is at hand. Or rather, not to realize the module itself, but to assemble a prototype and write software, so as not to be distracted by writing a program, and even more so as long as all the details go there is a sea of ​​free time, and the desire to do something does not give rest.

I had an Arduino Diecimila handkerchief, a few servo drives, a webcam, a joystick and an ultrasonic rangefinder on hand. Accordingly, immediately there was a desire to make a “computer vision” based on a webcam, with the possibility of both autonomous work and manual control (joystick).

What made me write this article?

Having rummaged in the Internet, I basically found all kinds of garbage, vague questions on the forums, excerpts from articles a bit far from needs. In general, I did not find a good, full-fledged article that would, from beginning to end, describe the creation of a moving webcam, with examples of code, and even more so combined with a range finder and joystick.
Then it was decided not to look for anything else, since the time for processing articles and collecting all the information in one became longer than if doing everything from scratch myself, especially since most of the articles are outdated for a long time.
')
The task is trivial, to send information from the joystick to the Arduino, which will rotate 2 servo drives with a fixed webcam at a certain angle, and, if necessary, read information from a rangefinder, sending it to SerialPort.
Having considered everything once more, I decided to start creating this prototype myself. Go!

Main part

Prototype build

The prototype was created within 5 minutes. The appearance of the prototype does not interest at all, its main goal is to work out the program part before the arrival of parts for the robot.
And I made it from the first available jar of some vitamins, two servo drives, a webcam, paper clips, electrical tape and a glue gun. It turned out the following:



The assembly is completed, the servos and the ultrasonic rangefinder are connected to the Arduino, the Arduino to the PC, and we start programming the Arduino.

We program Arduino

It all seemed very simple, since the joystick connects to the PC, the main video processing will also be on the PC, the Arduino will only receive and process information from the PC and control the servo drives. Therefore, we only need to read the Serial Port, process the incoming information in some way and somehow react to it.

Looking ahead a bit, I’ll say right away that there was an error, which I had to return to after writing a program in C #. The error was this: I, naive and full of enthusiasm, wrote a program that parses a line of the following type “90:90” coming into the Serial Port into two parts, respectively, the first part is the degrees along the X coordinate, the second part Y. With the help of the monitor the port everything was tested and worked perfectly, but when the program for controlling the joystick was written, with an enhanced port attack with strings with varying values, the Arduino simply did not have time to read everything sequentially, therefore often the lines turned into “0: 909”, “: 9090" and the like ok
Accordingly, the servos went crazy and assumed all positions except those that we need.

Therefore, without hesitation, I came to the conclusion that we need a line start character and a line end character. Again, without hesitation, the first character of the Latin alphabet, “a,” was chosen by the beginning of line symbol, the last by the end of the line was “z”, and the characters for the beginning of the axis values ​​were “x” and “y”, respectively. Total input line took the following form: "ax90y90z".

All is good, if not for the range finder. Ultrasonic range finder, he determines the distance with a bang, but there are several nuances. First, if the angle between the range finder and the wall is sharper than 45 degrees (plus or minus), then the sound is reflected from the wall on a tangent, and the value does not correspond to reality. Secondly, a rather large angle of signal emission, about 30 degrees (according to the manual), and the distance to the nearest object is measured, since the signal from the objects to which the sensor is at an angle is reflected to the other side, and we get a less real distance in a straight line , but interference still occurs, and quite often. Therefore, I added another function that takes n distance measurements, adds them and divides them into a count, set n = 10, so the noise became more smooth and less noticeable.

The code on the Arduino was immediately rewritten and took the following form:


The problem with the incorrect analysis of coordinates disappeared at all, 100 out of 100 tests were passed successfully.

The main control program (C #)

At first I wanted to write everything in C ++ under Qt, but later I still had to write in C #, but oh well.

What I wanted to get:
1. Face recognition
2. Tracking the face of a person.
3. Manual control with joystick.
4. Determination of the distance to the object.

For face recognition and image output from the webcam, without any questions, the OpenCV library was chosen, or rather its shell for C # - Emgu CV.

To read the position of the joystick at first, the Microsoft.DirectX.DirectInput library was used, which I did not like very much, and I applied the SharpDX library, and quite successfully.

What was required of the program:
1. Capture an image from a webcam and display it on the screen.
2. Recognize faces on the image, trace them and get the coordinates of the face on the image.
3. Form a string like “ax90y90z” and send it to the Serial Port to control the servos.
4. Read the position values ​​of the joystick.
5. Read from a range finder.

Having formulated the tasks, we proceed to programming.

The SharpDX library allows us to find the connected joystick and receive axis values ​​(from 0 to 65535), pressing and releasing the joystick keys from it. Servos can be rotated from 0 to 180 degrees, respectively, you need to convert the values ​​of the joystick axes from 0 to 180. I just divided the return value by 363, and I got the output values ​​from 0 to 180. Then I wrote a function that forms the servo position string and sends it to port.

Image output and face recognition are written using OpenCV and do not represent anything complicated (for us).

It is more interesting, having a range finder at hand, of course, I wanted to make a radar, and build at least some approximate picture of the terrain.

After repeating trigonometry and vectors, I wrote a procedure that calculates the coordinates of a point relative to our rangefinder with the camera on the angle of rotation of the servo drive and the distance to the object, and draws the results in PictureBox, I launch the procedure in the stream using the button, everything works, but still because of the room relief quite large interferences are obtained, but the approximate outline coincides with reality. I tried to smooth the data from the sensor, choosing only peak values ​​and drawing intervals between them, in principle, it turned out not bad, but decided to abandon it, because often the interference values ​​become peaks.

Code (just in case with detailed comments, if possible):




As a result, we get everything we wanted. The computer recognizes faces and automatically tracks them. manual joystick control works with a bang. The radar, though not entirely accurate, but works. The main functions of the module of view of the robot are worked out and it remains only to refine and improve them.

Video

That's what happened at the end.

Conclusion

Results

It turned out pretty simple. The goal is achieved, the prototype is ready. There is something to work and do in your free time, waiting for a package with components for a robot.

Future plans

The next step will be building a wheel platform for the robot, setting up a remote control (WiFi, 3G)., Hanging sensors (temperature, pressure, etc.), speech synthesis. Wishlist also has plans for a mechanical hand.

I think if there is interest in this article and its continuation, then it will surely follow! Corrections and criticism are welcome!

Thanks for attention!