Computer control with the eyes - a practical implementation

1. Necessity and problem statement

The article is written primarily for people who find it difficult or impossible to work on a computer in a conventional way: by pressing buttons on the keyboard and moving the mouse. I will try to tell you how the system was made, which allows you to control your computer with just one glance. At once I will make a reservation that the given solution can work poorly if a person wears astigmatic glasses / glasses with lenses of different diopters .
So, there are only eyes and a computer. Nothing else can be used - neither the pedals, nor the turning of the head, nor the conversion of speech into text.

2. Selection from existing solutions

Searching the Internet for the keywords “Eye Gaze Tracking” and “Assisitive Technology” gives a lot of links, after digging into which I come to the conclusions:

• Finished devices exist, but they cost amazing money; in Russia, no one sells them, does not serve them; the software does not provide for work with the Russian language (in particular, there is no keyboard with Cyrillic).
• There is open source software aimed at creating home-made systems. Tests with your camera do not give an acceptable quality of work.
• It is possible to buy devices for developers (developer edition) at a reasonable price in the companies Tobii and TheEyeTribe. But according to the scheme “in the morning - money, in the evening - chairs”, that is, pre-order can be issued in the fall of 2013, and the device can be received at the beginning of 2014.

For reference, in section “5. Other methods that I have not tried ” provide a description of various devices about which I typed information.
')
Suddenly, in mid-December, Tobii changes the concept, moves to the development of the new EyeX device, and its old REX device sends out to developers at a price three times lower than the original one and right now, claiming that the code written for REX will work with EyeX.
I did not think long, I ordered REX, on the basis of which the system was made.

[Correction of August 22 , 2017 ] At the moment, more modern eytrekers have appeared, which can also be used (see below)

3. Implementation details

3.1. Hardware

3.1.1. Option One: Tobii Eye Tracker 4C

The most relevant model of inexpensive eytreker for the summer of 2017.

I recommend this particular device. Compared with the EyeX model:

• sold in Russia (about 11,500 rubles)
• does not require a USB 3.0 port, enough USB 2.0
• less red light in your eyes
• loads less processor
• USB wire is hard-wired in the case, and does not fall out of the mini-USB connector
• iron is newer

3.1.2. Option Two: Tobii EyeX

The previous model from Tobii. Now you can still buy, but only if the money is very tight. Get ready to gaze at the red light bulbs. Nevertheless, a very decent machine for its time.

Attention! EyeX requires USB3 to work! If your computer does not have USB3 ports, the device will not work!

You can order a device on the Tobii website .
Delivered by UPS from Sweden.

3.1.3. Other supported devices

Tobii REX is no longer for sale, but there was a very good machine with two cameras.

TheEyeTribe - no longer for sale. So-so piece of hardware. Although it made a breakthrough in the field of low-budget aytrekerov, forcing Tobii significantly reduce prices.

GazePoint GP3 - a very expensive device, at the same time inferior to cheaper devices from Tobii.

3.2 Software

Included with the devices are only demonstration programs designed to show what the device is capable of, and the means for writing their own programs (SDK). The documentation that comes with the SDK can be said to be none. But there are examples in C ++ and C #, after studying which, you can understand how and what to do.

When I tried to use software from other, more expensive Tobii models, called Tobii Gaze Interaction, nothing happened. But I liked the idea, there is implemented.
So, I had to write the software myself.

A program was written that allows you to simulate the action of the mouse with your eyes: left, right, double click, drag-and-drop, the scroll wheel, and also type the text with your eyes on a virtual keyboard.
It is better, however, to see once. I invite you to see the demonstration of the program here .


The program is available for download (including source code) on github: here .
To download, you need to press the "RAW" button - this is the custom on github.

Detailed instructions for installing the program are here . Do not be confused by the fact that only the work in the table mode is described there.

The program was useful to several people from different countries, patients with amyotrophic lateral sclerosis (ALS). Here in this clip of the Charity Foundation one of these stories is told.

4. The problem of working with glasses

The device used, as well as most other similar ones, uses to determine the direction of the glare on the pupils, which are obtained from the infrared illumination. Unfortunately, glasses add spurious glare.

Although it is possible to indicate that a person wears glasses when setting up a device for a specific user, this does not help much, the device constantly loses the position of the pupils.
I would be grateful for the proven recommendations to eliminate glare on the lenses of glasses in the infrared.

5. Other ways I haven't tried.

[outdated information, it was relevant in 2013-2014]

At once I will make a reservation that the devices are used not only to facilitate the work on the computer for people with disabilities, but also, for example, to study which parts of the web pages users pay the most attention to. That is why sometimes monstrous money is thrown up for such systems. However, even for devices for people with disabilities they often throw up amazing money.

5.1. Products of commercial companies

SMI - senso-motoric instruments

Their devices can be viewed here .
Devices can be bought in Russia (I will not give a link), but when I inquired about the cheapest price, I was told about the price of one million two hundred thousand rubles. Goodbye, SMI.

Tobii

The company has long been producing products for people with disabilities. Even Intel has invested several million dollars in them. They had an alliance with Dell when the tablet was supplied with a piece of hardware for eye tracking, with Dell providing technical support.
In Russian you can read here .
In English - on their website .
A lot of things about these devices can be viewed on youtube.
Most recently, prices start at $ 4,000. Currently (January 2014), the price of PCEye has dropped to $ 2,000, but the software is not Russified (Tobii Gaze Interaction).

Eyetech

They have also been working in this field for a long time, it seems from the end of the 90s. Their devices are more coarse, but I really liked the software, a demonstration of which can be viewed here .

At the same time, there is a clip on which a person blinks his eyes to click on the mouse button - he didn’t like it and fainted. Tobii has not seen this.
There are three devices, differing only in size, for different monitors, from a laptop to 30 inches. The price I managed to find was $ 8,700. Gentlemen are not shy.
But we will note how their software works.

DynaVox

Specialized device , you can not use standard computer programs.

EyeGaze Edge (by LC Technologies)

A monstrous device that includes a camera and a computer that processes its data. You can connect this entire unit to your computer (including wirelessly), then it will replace the mouse and keyboard, which will allow you to work with regular programs. Again 8-10 thousand dollars.
It seems to be a set of "EyeGaze Edge Pack", where the computer you use yours, just finishes them with their camera and software, but the prices for it are not found anywhere.

These are the main players in the market. One of the small, ambitious companies, which could not bring its product to mind, nevertheless, was not too lazy to criticize the leaders and brought such a sign , from which it is clear that they themselves are the coolest (although there is no product yet, but the demo shows that the position of the pupil all the time jumps). But the comparison of the rest is very significant. Although the equipment there is often outdated, it is possible to estimate the price order and who have advanced further.

5.2. Open Source Projects

Unfortunately, none of the programs mentioned below provided a sufficiently accurate definition of the direction of gaze on the existing equipment (PC104E video camera with night mode and a separate infrared LED backlight). Nevertheless, maybe someone will work better.

GazeTracker

I recommend to start with the fact that watch the clip on this page .
The project was developed at the University of Copenhagen IT. It developed somewhere until the end of 2010, after which it fell into apathy. Nevertheless, we respect many for their openness, non-commercialization and certain achievements. Initially it was aimed at the possibility of working with “do it yourself” equipment. Many homemade people used this software with their hardware, which often cost them 20-30 dollars. See, for example, this .
The current version is 2.0 beta, before that there were a number of stable versions, for example 1.6, 1.8.
In the instructions for use it is mentioned that the developers themselves used the Sony HDR-HC5 camera with one or two backlights from the same Sony, model HVL-IRM.
On the site [regarding] a live forum, you can talk, if not with the developers, then with the apologists.

Haytham

An IT university in Copenhagen did not abandon work after the GazeTracker team dropped. Now they have a much more beautiful system :

This is not one program, but a server that distributes pupil position to clients over the network. One of the clients is the Haytham_Monitor program, which controls the mouse cursor. If I had earned good money on my hardware, there would be no need to do anything. Good luck to the guys.