Multitouch technology

Foreword

A small article about multitouch technology has already been published on Habré back in October 2008. In April 2011, the DMikhail habrauser described his experience in creating an interactive table . Outside is October 2012 and it's time to tell the next batch of news from the world of multitouch.

Introduction

35 years have passed since the appearance of the first multi-touch display, during which time a variety of different technologies have appeared, supporting from one to infinity touches. In this article, I will discuss some common methods for converting a conventional display to an interactive device.

Projection Capacitive Technology (Projected Capacitive Touch)

The principle of operation of this technology is described in the above article by habrauser meako, as well as on Wikipedia .
At the time of this writing, films up to 100 ”diagonally (for 4: 3 and 16: 9) with support for up to 12 independent touches are on sale. Also, Perceptive Pixel ( recently purchased by Microsoft corp. ) Presents a 82 ”multi-touch LCD display with unlimited number of touches.
The beauty of this technology lies in the fact that the touch film is glued on the reverse side of the protective glass, i.e. located between the display tool (LCD panel, projection screen, etc.) and glass, thus protected from weather conditions and mechanical damage. Some manufacturers stated that the glass thickness can reach up to 20mm. It is on this technology that most touch displays / terminals / kiosks installed in business centers, airports, train stations, subways and other public places are based.
The disadvantages of this technology can be attributed only to the lack of label recognition (fiducial marker, see below) and the limited area of ​​the sensor film. The last drawback can be partly offset by the possibility of sticking the joint-to-joint film.

Optical technologies (Optical technologies)

These technologies were widely used by DIY enthusiasts after the release of Microsoft Surface 1.0 in May 2007, and have survived to this day with little or no change.
Of all the variety of optical methods, the most widely used are the following:

• Rear Diffused Illumination (DI);
• Frustrated Total Internal Reflection (FTIR);
• Diffused Surface Illumination (DSI).

Before proceeding to a detailed description of each of the technologies I will describe the general advantages and disadvantages inherent in these methods.
')
Advantages:

• support of virtually unlimited number of simultaneous independent contacts (in fact, the number is limited by the resolution of the camera (s) and the power of the computer);
• the ability to make a screen round, triangular, hexagonal, etc. forms.

Disadvantages:

• since the technologies are based on infrared radiation of a certain wavelength (see below), the presence of direct sunlight in the installation area is unacceptable;
• installation of the projector and camera (s) requires quite a lot of space behind the screen, which is not always acceptable;
• it is necessary to develop special software that makes this system work, i.e. actually write the driver.

DI technology

The most common method (of optical), widely used in commercial installations due to its simplicity and clarity.

The diffuser is a material that scatters the flow of the projector, i.e. rear projection screen.
The substrate is an optically transparent material that gives strength to the screen.
Projector - projector.
IR illuminator is a monochrome emitter in the infrared range invisible to the human eye.
IR camera - a camera with a narrow-band filter installed, transmitting only the wavelength corresponding to the wavelength of the IR illuminator.

The principle of operation is as follows:

• The IR illuminators create a uniform IR background that passes through the substrate and the diffuser, and only a small part of the radiation is reflected back.
• when touching the diffuser with any object that reflects infrared radiation (hand, stylus, mug, etc.), in this place the camera detects a rather bright spot on a gray background.
• The signal from the camera processes the special software and, at the output, gives the coordinates (and sometimes the shape / size / pattern) of bright spots present in a predetermined area of ​​the screen.

Advantages of technology:

• the ability to recognize labels (see below);
• extreme ease of implementation;
• the ability to make a screen of any of the most complex forms, incl. non-planar;
• the ability to make a screen of any size.

Disadvantages of technology:

• the lowest image contrast from the camera (among the above three technologies);
• the possibility of false positives (when you put your finger on the screen, but without touching it, poorly implemented and configured software can recognize this as a full touch).

FTIR technology

Quite a common method, but in commercial products is extremely rare.

The diffuser is a material that scatters the flow of the projector, i.e. rear projection screen.
The substrate is an optically transparent material, the medium of propagation of infrared waves.
Projector - projector.
IR LED - LED with a wavelength lying in the IR range.
IR camera - a camera with a narrow-band filter installed, transmitting only the wavelength corresponding to the IR-LED wavelength.

Along the perimeter of the substrate, lines of IR LEDs are installed, the radiation of which penetrates into the substrate and propagates there as a wave in the optical fiber (the phenomenon is called total internal reflection). If the finger touches the screen, then due to a change in the refractive index, the wave travels past the substrate boundary and is reflected (scattered) from the finger, and this scattered radiation is detected by the camera. To enhance the contrast of the image from the camera between the diffuser and the substrate, a layer of the so-called “compliant layer”, most often made of silicone, is applied.
Scattered infrared radiation is detected by the camera and processed by special software.

Advantages of technology:

• the highest contrast of the image from the camera (among these technologies);

Disadvantages of technology:

• the need to install end IR illumination;
• does not recognize tags (see below).

DSI technology

The technology is identical to FTIR with the exception that the substrate is made of special Plexiglas.

The diffuser is a material that scatters the flow of the projector, i.e. rear projection screen.
The substrate is a special Endlighten plexiglass, the medium for the propagation of IR waves.
Projector - projector.
IR LED - LED with a wavelength lying in the IR range.
IR camera - a camera with a narrow-band filter installed, transmitting only the wavelength corresponding to the IR-LED wavelength.

The Endlighten chip is as follows: the metal “dust” is mixed in the material in such a way that when the plexiglass is illuminated from the end, the radiation is scattered on the “dust” particles and the effect of the self-emitting of the substrate occurs, while the material remains transparent to the lumen.



Thus, in addition to the recognition of clicks technology allows you to recognize labels.

Advantages of technology:

• the ability to recognize labels (see below);

Disadvantages of technology:

• the need to install end IR illumination;
• The technology is tied to a specific plexiglass manufacturer.

Tags (fiducials)

The label is a flat figure on which a pattern is applied, contrasting in the IR range. Most often it is a sticker with a black and white pattern, applied to a real object.


Sample tags.

IR radiation from the white area is reflected, and black is absorbed. Thus, in the DI and DSI technologies, the IR camera intersects the pattern, which is easily recognized by software. In the FTIR technology, such a focus will not succeed: the camera will see only the outline of the label, but not its content.
A striking example of the use of tags is Reactable:

Technical details

Diffuser

During the existence of optical multitouch technology, a large number of various films and plastics have been tried. However, the best material is reasonably considered to be the Evonik 7D006 . It is not unreasonable because exactly this material was used in Microsoft Surface 1.0.

IR illuminator

It is selected from the range of equipment security systems. On the market there are spotlights with two wavelengths: 850 nm and 940 nm.

IR camera

As far as I know, any CCD array is sensitive to near-IR, so you can use any camera, as long as the lens is not equipped with a filter that blocks infrared radiation. DIY lovers for this purpose use ordinary webcams, which are independently removed IR filters. I tried to use 3 cameras: PlayStation 3 Eye (classics of the genre), Microsoft LifeCam Cinema, Logitech C910. Most of all I liked the Logitech camera, which has the largest CCD and easy-removable filter.
In order for the camera to see only the “necessary” radiation, a narrow-band IR filter must be applied to the lens, transmitting only the wavelength corresponding to the wavelength of the IR illuminator. Buy IR filters, for example, on aliexpress.com .
Of professional cameras, Point Gray cameras are most often used.

Projector

There are no limitations on the projector, the main thing is to choose the right resolution, stream and lens. Quite a few installations use ultra-short-focus projector, which are located very close to the screen, saving space. Also, do not bypass the side of the laser-led projector, which at the moment are just beginning to gain popularity, the main advantage of which is the high service life of the light source (more than 20 thousand hours versus 3-4 thousand for lamp projectors).

Examples of commercial products

Mediascreen MonkeyBook (on DI technology):


Microsoft Surface 1.0 (on DSI technology):

UPD. MS PixelSence technology

In January 2011, Microsoft, together with Samsung, unveiled a new Microsoft Surface 2.0.



In 2012, in connection with the release of the MS Surface tablet, the Surface 2.0 table was renamed to SUR40.
In the table SUR40 uses proprietary technology MS PixelSence.



The technology is similar to the DI technology, except that instead of the IR camera, an IR sensor is used, “embedded” in each pixel of the LCD matrix. Thus, PixelSence allows you to recognize not only the fingers, but also the tags, while the thickness of the structure is minimized.