Eye-to-eye video communication: attempts to solve the problem of lack of eye contact

Our computers, smartphones and tablets have built-in video cameras aimed at users. Video chatting and video conferencing - one on one, one with many, many with many - have become commonplace both in business and among ordinary people, and I think most of us would agree that this connection is much better than ordinary conversation using sound alone . Seeing the face of another person enriches the conversation and increases satisfaction from it. However, I am annoyed by one feature related to the current video call: the inability of the participants in a conversation to look into each other’s eyes. There was no such problem in Star Trek, and this universe, of course, is the source of all my technological expectations.

Watch me while talking

If you used video chat, you probably understand what I mean. The camera that looks at your face is located above (and sometimes below, or to the side) from your screen. This means that the angle at which you look at the screen differs from the angle at which the camera (and your interlocutor) sees you - this effect is known as parallax [or not; because parallax is a change in the visible position of an object relative to a distant background, depending on the position of the observer / approx. trans.]. The interlocutor will get the impression that you look into his eyes only if you look into the camera. Therefore, when you see the image of your friend on the screen, it seems to you that he is looking down (or in a different direction, but not at you), and you look exactly like that on a friend's screen. Of course, you can position the camera directly in front of the screen, but then the camera will close the image of your interlocutor.

Eye contact is extremely important for meaningful conversation, and, after all, the whole point of using video instead of one audio is to see the person with whom you are talking. But if you cannot look a man in the eye, this eliminates most of the benefits of video over a regular phone call. The instructions for effective business communication usually say that you have to look into the camera while you are talking, so that the people on the other end have the impression that you are talking directly to them. But this is unnatural, it does not allow you to see their reaction to your speech. In fact, we need exactly what happened on the Federation spacecraft: video screens, which simultaneously operate cameras, so that when you look at the screen, your eyes would look the same from the other side. Naturally, engineers are already trying to achieve this effect, working in several directions.

It's all about the mirrors

One fairly simple way to maintain eye contact during a video call is to use technology borrowed from the television industry: the teleprompter . If you watch the news on TV, you will notice that the announcer is looking directly into the camera. Speakers do not learn their reports; they read them from a special video screen, which seems to appear right in front of the camera. In fact, the screen (plain flat panel) is placed face up in front of the camera from below, and the text on it is shown in mirror image. Above this screen, directly in front of the cameras, at a 45 ° angle there is a partially silvered (or double-sided) mirror. The announcer sees in him a reflection of the text below, and the camera sees only the announcer.
')


Teleprompters - technology is simple and proven; they exist more than 60 years. And when such devices are used for video communications, they are sometimes called video tunnels. But they have certain problems. One problem is in size: the equipment is, by its nature, rather cumbersome, because it requires an angled mirror in front of the camera, as well as special protection of lenses from glare. Teleprompters are also usually heavy, fragile and expensive - all of these factors make them unattractive for ordinary consumers.

I have a ProPrompter Desktop device, worth “only” $ 500, which can be put on my computer (desktop or laptop), or on a tablet. This is, in fact, a miniature teleprompter, and the video can be positioned in such a way that the image of your interlocutor (instead of reading text) will be located right in front of the camera. It is cumbersome, but working, and it turns out to be useful when I conduct remote video presentations in front of large groups, or record video on the script.

Whether you use a teleprompter, or not - there is one more problem in cases when more than two people participate in a video conversation. If I look directly at the camera, then all the people who see me on the screen, it seems that I make eye contact with them, even if they are scattered in different places. Then the participants will not have the impression that my view is moving when I transfer attention from one person to another - and I cannot say who is looking at me (or my image) on the screen. The GAZE-2 system , which is being developed at Queen's University in Kingston, Ontario, is trying to solve this problem by using several cameras in a video tunnel, along with an additional camera that goes where the computer thinks the user’s view is directed; The software switches to the camera closest to the user's gaze and rotates the image at the other end so that it coincides with what is happening.

Presence effect

Another proposed solution to the problem of determining the direction of sight, developed at Keio University in Tokyo in 1996, was the MAJIC system. She replaced the teleprompter's two-way mirror with a large curved screen of thin perforated material, which provided a reflective surface on one side, and mostly transparent on the other. Cameras behind the screen recorded the participants in one place, while ordinary video projectors showed images of other participants (located in one or several places) on the screen. A unique feature of MAJIC was that behind the image of each person on the screen each less contained a separate camera that worked with the virtual eyes of this person in this place (and, together with the speaker, reproduced his voice). In the end, it seemed that each person always looked at the participant in the conversation to which he had turned at the moment, and it was even possible to see when one participant in the conversation was looking at the other. An additional plus: the life-size projections created the feeling of actually sitting at a table opposite each other. Unfortunately, as far as I know, this system was never brought to sale, which is not surprising given its cumbersomeness and the cost of equipment.

Ten years later, a much more compact version of this system appeared. In January 2006, Apple received a patent for an eye-to-eye video system, in which a large array of microscopic (and in fact, invisible) cameras were built into the monitor along with display elements; and the software combined all those thousands or millions of images into one. This should have an effect similar to that offered by MAJIC. Time will tell when, or in what form this technology will be available to consumers.

A different, perhaps more promising approach called eye correction is being explored by researchers from large companies such as HP, Microsoft, and AT & T, among others . It all starts with one or two ordinary camcorders, mounted next to a conventional display. A special video processor digitally modifies the face image of each person in real time so that it seems that his eyes are looking directly into the camera, even if it is not so. At early demonstrations, these systems look more or less convincing - albeit a little scary - but they are not yet ready for commercial use. Also, they have not yet been adapted for satisfactory work with many participants located in one place or for selective eye contact with one of several remote participants.


It's great that progress does not stand still, but given the huge computing power in today's computing devices, I am surprised and disappointed that so far there are no software tools available to all for eye correction. It is also annoying that such a tool used to be - a Windows application called CatchEye , which worked with Skype, Google Hangouts, Facebook Messenger, and other products. However, it was removed from the market in 2017 without any explanation whatsoever. I want to think that this happened because the developer was bought by a large company like Apple or Microsoft, and now it is working hard to give this opportunity to the masses, but I may be overly optimistic. If only I could look technological giants in the eyes ...