Emotional recognition market research

Scientists have been working for many years tirelessly to powerful computing systems have been able to adequately capture and identify facial expressions. Nowadays there are great successes, and it is not only the perfection of the proposed methods and algorithms, but also the development of newer ones. Already, many companies offer proprietary software, so I wanted to learn more about the products on the recognition of emotions on the face of a person.
Under the cut a lot of text and pictures.

The recognition of emotions is part of a large reservoir of science, united in the names "Pattern Recognition" and "Processing of Visual Information". In our recognition technology the days are no longer out of reach, and formed a new trend, entailing all interested in the methods of identification and handling of found objects and their attributes. They smoothly flow from the realm of fantasy to real life.

Existing emotion recognition systems

Of commercial solutions in the emotion recognition systems market, the most complete and more interesting to consider in the context of the emotion recognition task today is FaceReader from the Dutch company Noldus Information Technology.
')

1. FaceReader

• Company - developer: Noldus Information Technology (Netherlands)
• Product Readme Information
• Current Version: 4.0



The program can correctly interpret such facial expressions as “happy”, “sad”, “angry”, “surprised”, “frightened”, “displeased” and “neutral”, as can be seen in the figure. In addition, FaceReader capable of persons of people to determine their age, gender and ethnicity. FaceReader does not need training and additional configuration.
The program implemented computer vision technology. In particular, the Active Template method, which consists in applying to the face image of a deformable template:



Also, the method is implemented Active Appearance Model, with which you can create an artificial model of the face based on control points and surface detail, and compare it with samples laid down in the memory.
The classification is done using neural network methods with a training set of 2,000 photos.

Program features:

• The average percentage of emotion recognition is 89%. For some, it is higher emotions, some less;
• face tilt can be any in the plane, its system will detect;
• The program works with downloadable video in formats with MPEG1, MPEG2, XviD, DivX4, DivX5, DivX6, DV-AVI and uncompressed AVI codecs, and you can determine emotions by frame or completely when watching the entire video. Also, FaceReader can work with static images, as well as in real time, if the user has a webcam connected;
• The program is perfectly visualized: one can always see the histogram chart, percentage rule to express emotions. And on the timeline visible manifestations of microexpressions in a certain period of time;
• FaceReader generates two text files, one is the emotion log, and the other is static, for The Observer XT, a unique visual data management system developed by the same company, associated with this program.

Disadvantages of the program:

• FaceReader is not trained to recognize children under 5 years old;
• If a person is wearing glasses, the recognition of emotions inaccurate or classification is not conducted;
• People with different skin color is perceived differently by the system, the program is not fully adapted;
• Turned face is not detected.

2. eMotion Software and GladOrSad

• Company - developer: Visual Recognition (Netherlands)
• Product Readme Information
• Current Version: Unknown

eMotion Software System is known for its founders recognized the emotion in the painting "Mona Lisa." The result showed that it was 83% happy, 9% displayed disgust, fear 6% and only 2% Mona Lisa angry.
And the system is known by the fact that it is, in fact, the first commercial pay "box" solutions. Along with this solution, the development team launched the site GladOrSad.com - respectively, Visual Recognition took the lead in the opening of a web resource dedicated to the online recognition of emotions.



EMotion Software became the first known user company Unilever's , implemented the system of recognition in the machine for the sale of ice cream - Unilever's Share Happy . People smile machine, the machine gives a smile for free ice cream!

If a person shows emotions, smiles, frowns, or makes a face, thousands of small facial muscles are in work. emotion recognition system, or ERS (Emotion-recognition system), creates a 3D - model of the person with the identification of 12 key areas, such as the corners of the eyes and the corners of his mouth.
In these programs, the tracking algorithm identifies the same emotions, there are six of them: anger, sadness, fear, surprise, disgust and happiness, and also the seventh is their confusion.
The software is not particularly demanding on the computer in terms of technical characteristics. The details of the implementation of the algorithm is unknown, since technology is kept secret, an explanatory brochure, I also found, unfortunately.

3. MMER_FEASy - the FacE Analysis System

• Company - developer: MMER-Systems (Germany)
• Product Readme Information
• Current Version: Unknown

Again, in the development of the methodology used by the overlay on the face of a specific deformable mask, Active Appearance Model methodology, which allows you to calculate the required parameters in real time. The mask operation is shown below:



The system uses three plug-ins - MMER_Lab, MMER_GPU and MMER_Locate.
MMER_Locate ensures that the person in the image, MMER_Lab classifies some features of the image, and MMER_GPU enables efficient operation of the entire system:



The program recognizes six basic emotions, also provides services in finding the faces of aged people, sex and ethnicity. Also, the system identifies the person if this ever reference photo has been loaded into the database.
The additional features of the program include its connection to other programs as a module for remote assistants, assistants, drivers, market research and domestic media - services.
The disadvantages of the program can be considered not the full coverage of the downloaded data, since You can only work with a webcam. Poor results and data unloading, where you can view only "avatarizatsiyu" person, ie, instead of a mask, this same face is substituted, but with a different facial expression.

4. FaceSecurity

• Company - developer: Cognitec (Germany)
• Product Readme Information
• Current Version: 4.6

This product consists of several manufactured components based on the FaceVACS SDK . It:
• FaceVACS-DBScan with Examiner
• FaceVACS-PortraitAcquisition
• FaceVACS-VideoScan

FaceVACS-DBScan with Examiner:

The development is designed to handle the unique data banks of any category of people, for example, co-workers.
This product is a biometric identification embodiment according to the standard samples from a database.
In the new version of the program, the system uses the new comparison algorithm B5T8, along with the old A14T8, designed to improve the identified similarity.
In addition, the new component Examiner allows automatic image conversion for comparison in the gallery. This allows operators to see developed lists of potential partners, while maintaining full audit for each step in the process.
Also, this development helps investigators to identify persons in crime scenes by photography and video surveillance by comparing images of persons in the agency repository.
FaceVACS-Examiner also provides a set of tools that help the inspectorate to identify a person in a timely manner, which allows investigators to act in accordance with the search results in the minimum amount of time after the commission of a crime.

Features:

• Cluster configuration for handling multimillion database of people;
• Flexible and convenient list management, which allows you to sort it to monitor and filter;
• Package and interactive training, identification;
• Deep and flexible management of related data;
• Reconfigured logs;
• pregeneration databases (i.e., pre-fast presetting before issuing the result);
• Available in many different search samples by database.

FaceVACS-PortraitAcquisition:

Creation and evaluation of digital portraits for photo identification documents. The operation of the component is illustrated in the figure:



This component simplifies the production of high-quality portraits for passport photos, driver's licenses and other documents that are suitable for face recognition.
The graphical user interface of the product is specially tailored to visually control and manipulate the processing of such trifles as frontal posture, uniform lighting, glasses, and squinting eyes. The software is specifically tuned to assess whether the image complies with the mandatory requirements and best practice recommendations of the ISO 19794-5 standard of the front image type. The program supports integration with Web Services (SOAP) to facilitate the issuance of the document made.

Features:

• Full compliance with ISO 19794-5 standards;
• Reliable and automated process of gathering information;
• Checking the frontal posture, glasses, uniform lighting, head size, image size, mouth opening, head rotation, checking for tinted glass, red eyes, front view of eyes, exposure, skin color, hot spots, sharpness;
• Convenient graphical user interface;
• Customizable parameters and thresholds;
• Supports the format of digital SLR cameras from Nikon (D5000) and Canon (EOS 1000D and 1100D EOS);
• Flexible crop, if necessary;
• Custom image preview (size, image type, image format);
• Flexible integration with web services.

FaceVACS-VideoScan:

The new generation of computer surveillance automatically scans incoming video streams, detects multiple faces, and checks for possible matches in the "checklist". If a match is found, operators are notified in real time.
The application includes not only the identification of undesirable people in public places, as well as the identification of high-ranking clients.

Features:

• Automatic surveillance of a person in real-time across multiple video streams;
• Real-time face compared with the reference "checklist";
• Registration with a still image or a live video stream in manual or automatic mode;
• Application of C ++ API and Web Services API;
• Possibility of scalability within a "checklist": the size, the number of video streams and the number of visible entities in the image.

Cognitec features:

• Extremely high processing of comparisons according to samples (900 000 comparisons in the database per second on an average processor power);
• Integration with Web cameras, http - cameras, digital cameras, camcorders, and support for images in common formats;
• Bulk databases, integration with Oracle, IBM DB2, MSSQL Server.

Disadvantages:

• Calculations in almost all components are applicable only to frontal faces (deviations of 15 degrees are possible, but not more);
• Light plays an important role - for example, the face image in the shadow of the components do not recognize.

Not to mention the high-profile web-solution based on FaceVACS-SDK and FaceVACS-DBScan, MyHeritage.com portal, where you can build your family tree on the basis of face recognition, as well as compare yourself with celebrities, face morphing, as well as recognize and mark yourself in the photo.
In addition to Web-based applications, Cognitec provides an API for digital billboards - billboards (billboard) for the purpose of showing ads to target audiences.
As mentioned above, Cognitec is also involved in mechanical engineering, the systems of this company are used in automobiles for analyzing the faces of drivers and fellow travelers, as well as safety, for example, by detecting the position of the head, detecting a scattered look, detecting closed eyes.
Another aspect is what sets Cognitec among other companies is the presence of its own SDK for mobile phones.

5. Products of Affective Computing Research Group

• Company - developer: Affectiva (USA)
• Product Readme Information
• Current application software version: 1.0

Rosalind Picard's company, Affectiva, is best known for the Q-Sensor wearable biosensors supplied. But not only this rich company. There is a huge experience of technology adoption among Affective computing, or emotional computing systems development goes from 1995. There are a lot of projects. This is the oldest group of developers involved in these technologies.
There is, for example, the AffQuake project based on the ID Software Quake 3 product. The point is that the game reacts to the player's emotional signals. The gamers are bombarded with sensors, and if it gets scary, the modified Quake receives the user's “physiological signals” and makes the player’s virtual embodiment fear the same way - it retreats in fear.
Or, for example, developed a toy "Emotional tiger» ( Affective Tigger ). This robot can identify the five emotional states play with him and the child to express his emotion in response. If a child jumps, squeezes and kisses a toy cheerfully, then the emotion recognition system and sensory system of the “tiger” fix this physical effect, after which happiness is demonstrated: Affective Tigger laughs and smiles.
There is an interesting solution for the Web. It is an innovative product company Affectiva, collecting data about the emotional state of people around the world wide web, Affdex . For the most part it is used for market research.
One of the ways to recognize the emotional state of the face in these developments is to record in real time with subsequent computer analysis - methods of comparison with embedded samples (SURF and based on SIFT descriptors), as well as wavelet methods. The work of these methods is used in such a program as a Pupeteer, evaluates the behavior and emotional state of students. Demonstration of the program:



In the course of this method, the six basic emotions experiment computer determines with 96 percent accuracy.
The decision is also remarkable because it recognizes, along with emotions, head movements, such as a nod or a swing, winding from side to side. Processes used Bayesian machine learning to classify emotions, as well as for the calculation of statistics and calculations of mixed states, when you can not express exactly what kind of emotion prevails.
There is not much to say about the software itself, as the technologies are closed. Development is conducted in C ++, Objective C for the iPhone. Graphically drawn through the usual tools, such as timeline (or timeline), graphs and charts, as seen in the following figure:



Among the features, it can be noted that all solutions are well drawn and adapted to the customer, and of the drawbacks, perhaps, the greatest is the presence of a computer with a non-weak processor (above Core i5) for comfortable work with applications.

Company comparison and subtotals

The solutions of only a few players in the business. Other companies offer their products intended for other little problems, but the system developed by them anyway interesting, because it can be easily upgraded to the level of emotion recognition. This software and solutions, performing tasks such as:
• verification of the person (security systems and access control);
• tracking and face tracking (CCTV systems);
• comparison of people in the image and likeness of his (search system);
• animation of the face and its transformation (morph systems);
• face transformation into 3D - models (modeling systems);
• determination of the race, age and sex of a person (gender classification system);
• much more.

And most of the companies that develop these programs provide a toolkit (SDK - Software Development Kit) to any developer.
The key criterion for the use of such products is their cost , as well as the cost of the provided SDK. It ranges from $ 5 to $ 2,000. The most expensive products are not likely to be used in small companies, where the development of this kind is not a condition of the entire company.
Below, in the table, I provide a list of such programs and program complexes. Of course, over time, the list will expand, but as of November 2011, it is relevant:


Actual activities of companies focused on web-development and technology. These are Google, Twitter, FaceBook and many others. Google Inc. uses face and emotion recognition technologies for a smarter search, Twitter assesses the mood of people writing in online blogs, FaceBook recently introduced a new feature (improvement) called Tagger to the public that automatically recognizes, marks and signs the faces of friends of a social network user.
Face.com , SDK JavaScript, PhotoTagger.
, , -. PhotoFinder , , . , — , . , . web- . , . Flickr YouTube. , . , .
, , - . , Ricoh, Fujifilm, Canon, Nikon .

, , . , :

findings

, , , , , . , , . , , , , , , . , , , , – . , .
, : - SaaS (Software as a Service – ) . .
, . , .

Thanks for attention! , , … .