Intel Perceptual Computing SDK

Progress does not stand still, everything is changing - our understanding of mobile devices, they are becoming thinner, healing, more productive, ways of interacting with them are changing, the touch and mouse have been replaced by touch and sensors, replaced by Perceptual Computing .

What is Perceptual Computing ? This is a new technology that will allow the user to interact with their mobile device through voice, hands, face, i.e. more natural human interaction methods.



Currently, Intel is working in this direction, in October last year, the first version of the PerC SDK appeared on the company's website, which supported the following features:

• Recognition of gestures, the position of the hands, fingers, level of openness of the palm
• Tracking the position of the face, the definition of control points (lips, nose, eyes), identification ("recognition") of the face
• Tracking the position of 2-D and 3-D objects

In late November, a new version appeared ( Beta 2 ) - in addition to correcting errors and working on optimization, developers also added the ability to work with voice. This was made possible by the inclusion of the Nuance Dragon Assistant module in the SDK.

At the moment, Beta 2 is the latest version, but already includes a number of features that can help the developer add new ways of interacting with the user to their software.

To get a better idea of ​​the capabilities of the PerC SDK, you can watch a video showing its work on the reception of several applications, available on the SDK page .

Camera

To begin with, to explain how this all works, I would like to tell you about the camera.

In fact, this device can hardly be called a camera, it is rather a set of sensors. It consists of two microphones that can significantly improve the quality of speech recognition, the sensor also has built-in RGB and infrared cameras.

Specification

• Resolution RGB sensor: 720p (1280 x 720)
• Resolution IR Depth sensor: QVGA (320 x 240)
• Viewing Angle (degrees): 73
• Frequency of work (frames per second): 30
• Distance: from 15 cm to 1 meter
• Power supply: USB 2.0 (<2.5 watt)

')

System requirements

• 2nd generation Intel Core processor and above
• USB 2.0 port
• Windows 7 SP1 and higher

SDK architecture

The SDK architecture is shown in the image below.







At the top are applications that use the features of the SDK. Next come the modules responsible for the functionality, input / output module, drivers for working with the camera and the hardware platform.

Modules

As you can see, each opportunity is presented in the form of a separate module, it allows you to increase the possibility of the platform, without making significant changes to the kernel, but simply by adding a new module. The SDK provides an API that allows you to work at a high level — using embedded gesture recognition, hand position, face, voice recognition algorithms, and at a low level — creating and adding your own modules and algorithms to the SDK.

I / O module

An application that uses the SDK does not work directly with the camera, this is done by the I / O module, thereby achieving the simultaneous operation of several applications using the SDK at once, which, for example, is impossible when working with a webcam, when the application gets exclusive access restricting camera for other applications.

Driver

It works directly with the camera, uses the capabilities of the hardware platform to speed up the SDK, which allows you to win not only in speed, but also in power consumption.

Supported programming languages ​​and frameworks

PerC SDK provides the necessary files (headers, libraries) for using its capabilities in applications written in C ++, C #. The SDK also includes support for the Unity Game Engine , the openFrameworks , Processing framework. A third-party library for the Cinder framework is also available.

If desired, you can add support for any other platforms, programming languages ​​by creating libraries, wrappers in C ++, C #. As an example, I want to give my extension for Google Chrome , which allows HTML5 applications to receive notifications about recognized gestures. Its expansion requires a pre-installed PerC SDK and a camera from Creative.

Where to begin

In principle, working with the SDK is possible without the camera from Creative, although in this case only the speech recognition modules and the position and face markers will be available. In order to take advantage of all the possibilities you need to order a camera, you can do it on this page , delivery works for many countries, including Russia.

The SDK can be downloaded from the official Intel PerC SDK page , it includes all the modules for working with gestures, tracking, except for the speech recognition module, which is installed separately and is available for download there. The SDK includes many examples that are available both in the form of binaries and in the form of source texts.

Areas of use

The first thing that comes to mind is, of course, games. Here, the application is limited only by the imagination of the developer, you can allow the user to control game objects, characters, navigate in space with the help of fingers, hands, gestures.

An example of a game in which a user can control a throwing tool with his hands.







Managing gameplay and user interface on the example of the game Kung Pow Kevin







Face recognition is also useful here, for example, you can upload a user profile using face identification.

Using information about the position of the face, markers can create a “fitting room” in real time, giving the user the ability to change hairstyles, hats, glasses, makeup, etc.

In business applications, the SDK can also find application, for example, navigation in an application using gestures, voices, and again - user identification.

In educational applications for children, such as learning to count, you can ask for the right answer by showing the right number of fingers.

Another interesting application - the camera helps to do the subtraction of the background, without using any complex algorithms. Look at the picture below - the result of the infrared camera, using this information to separate the image of the user from the background is easy.







I myself liked to experiment with the camera, it is actually infrared, that's what happens if you show the camera bill.







In the dark, she sees well too.







The picture was taken in the room with no lighting at all, the laptop screen was turned off.

As I said above - the possibilities are broad, the main thing is to include fantasy. In the following articles I will talk about using the SDK using a simple application as an example.