Hold on to the wheel, hold on ... Our project to control the condition of drivers

Every motorist knows that drivers tend to tire. It happens that your eyes stick together, even when you go home from work, and truckers are generally one of the main professional problems: thousands of accidents happen every year due to the fact that people fall asleep at the wheel. The fact is that long-distance drivers need to constantly maintain a balance between rest and safe driving on the one hand, and fatigue with higher earnings on the other. The temptation to sleep less is great, but it is faster to take the goods and get a new order. Not to mention the fact that drivers have their own personal lives and they do not always go on a flight slept.

To make the roads safer, we launched our project to monitor the condition of drivers: according to customer requirements, we create software and hardware solutions that can monitor the driver's well-being on the road and prevent the stage of falling asleep so that he does not “wake up” in the next world.

Loving Eye

The basic idea is to monitor driver fatigue and dangerous reactions with a camera. The device first collects statistics on how a particular driver behaves while driving. And, if on the basis of these statistics, the system decides that the driver shows signs of fatigue or is about to see dreams, she signals this to him. If the driver often began to nod or react to events like a sleepy fly, then the system also reports this to the dispatcher.
')
We use video analytics to track how focused the driver is and whether he is looking at the road. The camera constantly shoots face and hands, and the system keeps track of a number of key points. We expect the driver to have his eyes open all the time, his eyes directed towards the windshield or side mirrors. If the eyes are closed, or the head leans forward and the chin falls on the chest, or the face turns away from the windshield for a long time, the system gives a signal.

Of course, this tracking and signaling do not work all the time, but only when the car is moving, it uses a GPS sensor. We also provided for the possibility of forcing the system to be turned on and off, because situations can be different. But at the same time we compare the duration of the movement of the machine and the time when the system is turned off. If necessary, the device can inform the dispatcher that the driver has been driving for a long time with the “control” turned off.

Implementation

The project consists of two parts: hardware and software.

How does the system determine that the driver will fall asleep now? With the help of the OpenCV library, which is based on machine learning. OpenCV allows you to define key points in a video stream: chin, hands, eyes (including their size and breadth of disclosure). And on the basis of these parameters, the system determines whether the driver is tired or not.

The hardware consists of a Waveshare RPi Camera (F) and a Raspberry Pi camera, assembled in a single package. It is more difficult to create hardware than it may seem at first glance: you need not to connect and solder components, but think over their combination so that the device works for a long time reliably and without failures. In fact, this is a separate integration task. And integration risks are usually the highest. Indeed, it is one thing when a software package runs on your laptop, under your OS and with your limitations, and it is quite another thing when it is transferred to a Raspberry Pi, under a different OS, where the processor is much weaker and there is less memory.

Of course, if the product is put into a series, then it will be implemented no longer on the Raspberry Pi. The piece of iron is weak, its power is not enough to process the video stream in full resolution and recognize the face, so with such a hardware stuffing you have to go to tricks.

In addition, we implemented the night mode so that the system could wake the driver even in the dark, when people most often fall asleep. That is, I had to teach a computer to process video from an infrared camera. Here we were not pioneers, a similar problem is being solved in the systems of the smart home. To determine the fatigue of the driver does not need to know the details of his appearance: hair color, eyes, skin tone. We only need to clearly determine the position of the hands and key points on the face. Strangely enough, but working with the eyes in the infrared mode turned out to be even easier than in the ordinary. If you have ever seen commercials shot with automatic cameras with animals in the wild, then you know that with a good angle, their eyes shine like laser.


This effect turned out to be very useful to us: it is easier to determine the fact of closed eyes. Of course, it took an additional adjustment of all algorithms, because the spectrum of the image is completely different, but, in general, the eyes are recognized more confidently.

But in the infrared mode, the system is worse at recognizing situations when the driver is talking on the phone, pressing his ear to his shoulder. This is also a dangerous situation, the traffic rules directly prohibit talking on the phone at the wheel.

Of course, like any machine learning model, ours also does not work with one hundred percent accuracy. But you need to understand well what exactly recognition errors depend on. For example, we have not tested our system on drivers with a non-European appearance. Perhaps in this case, the proportion of accurate eye recognition and fatigue will decrease. However, during internal testing, the system worked exactly in 8-9 cases out of 10.

* * *

Today there are a lot of good algorithms in the public domain, and we took ready-made ones, which made it possible not only to solve the problem, but also save a lot of time. Now you can quickly find a model suitable for solving some unique problem, and quickly modify it. Of course, this requires skills and experience in solving computer vision problems and quick adaptation. Then it takes just a day or two to get a prototype on a laptop. And further work depends on the task, on available capacity, storage and other factors. For example, you can quickly modify our system so that it performs a different role: for example, as a security device, which ensures that only specific people enter the room at a certain time of day. That is, you can quickly create a solution for a variety of tasks related to computer vision.