Special glasses change the person’s identity for face recognition

Successful impersonation in the facial recognition system. Left: actress Reese Witherspoon, whose personality the neural network recognizes with a probability of 100%. Center: points matched for impersonation of actor Russell Crowe. Right: the victim of impersonation

Thanks to neural networks, machine vision systems have achieved record accuracy in face recognition. The personality of a person is almost unmistakably recognized even in blurred photographs and in those where the face is only partially visible. Created algorithms that take into account the context: clothing, environment, gait. So far these are only advanced technological developments. But given the global hysteria around terrorism and security (although 53 times fewer people die from terrorism than in traffic accidents), very soon such recognition programs will without any doubt be connected to millions of surveillance cameras in public places. The question is, what are the methods to deal with such surveillance?

Face recognition is very hard to fight, but scientists are still trying to find "vulnerabilities" in face recognition systems. To deceive such a system requires a specific way to change the physical appearance. There is no problem putting on a hood or putting on big dark glasses. But this is suspicious - a person with a closed face can be detained by the police or guards. Must be more intelligent ways. Some small changes can certainly bypass the computer recognition system, but they will have almost no effect on human perception in the eyes of others and will not cause anyone to suspect.

The task is similar to the one that spam distributors solve to bypass spam filters. They need the message to reach consumers in a readable form, but at the same time bypassing computer filters. So here - the face should be open and well recognized by people, but should not be recognized by the neural network.
')

The attacker's face (left), illustration of the attack (center) and the victim of impersonation (right)

Ideally, the open face of a person should be recognized by the system as an alien face, following the pattern of replacing the eye of the protagonist of the film Dissenting Opinion. In this case, the guards and the police will have no complaints at all, and the person will be able to move freely around public places that are scanned by video surveillance cameras.

It is important to understand exactly which changes will most strongly affect the algorithms of the work of neural networks in face recognition. A group of researchers from Carnegie Mellon University studied the issue in detail and recently published the results of its research on this topic. In fact, researchers are talking about a new type of attack .

Attacks that physically change appearance, but at the same time look harmless to an outside observer.

Unlike other similar attacks, in this case, the researchers focused on the factor of “harmlessness” so that the fact of the manipulation of the neural network did not cause any suspicion among strangers, including security guards, who view the video stream from surveillance cameras.

Scientists have found that in this case it is most effective to use glasses with a specially selected pattern that affects the neural network for face recognition. Testing of facial recognition systems has determined that the most vulnerable point of neural networks is a comparison of the color of neighboring pixels in specific places of the face around the eyes.

“Harmlessness” is really important, because in the event of detention, the person will be able to effectively deny the evil intent. No one can prove that these glasses are specially created to deceive the facial recognition system, and do not have a random coloring.

Researchers have proposed two options for conducting an attack: impersonation and evasion. In the first case, the task is to give the face of a person with glasses to the face of a particular other person. In the case of an attack on evasion, the challenge is to pass the face of a person with glasses on the face of any other person.

As shown by tests on real face recognition systems, it is possible to create such glasses in most cases for some individuals. For example, for the face of a white man, specially selected glasses impersonate Mila Jovovich’s personality in 87.87% of cases. Three authors of the scientific work (three white men) experienced glasses on themselves and stated that printed glasses with a specific pattern made it possible to carry out an attack on evasion in 80% of cases against the most modern commercial face recognition systems.

Of course, the success of an attack on impersonation depends largely on the attacker's race and color. For example, a South Asian woman is able to impersonate a Middle Eastern man in 88% of cases, while a Middle Eastern woman can impersonate Clive Owen (a white male) only in 16.13% of cases.

The systems were tested on the Face ++ face recognition system, as well as in the face detection system according to the most widely used method by Viola-Jones today. The Viola-Jones algorithm proposed in 2001 is implemented in a variety of popular programs, including the OpenCV computer vision library. He finds faces with high accuracy and a low number of false positives.

Scientists optimized the algorithm so that the appearance of the glasses was the most beautiful, with soft color transitions, and was suitable for printing on an ordinary printer.



Researchers estimate that printing frames and paper glasses to bypass modern facial recognition systems will cost about 22 cents. To check the effectiveness of the system for avoiding recognition, 20 random photos from the PubFig facial photo database were selected. The use of standard points made it possible to carry out an attack on evasion in 19 cases out of 20, and the use of auxiliary accessories (contact lenses, winter hat, etc.) increased the success of the attack to 100%. Scientists also note that the method successfully works not only in face recognition systems with open source, but also on systems like black box.