Chinese scientists have taught the neural network to understand what a person sees through scans of brain activity

This is how the fMRI scanner looks like.

Reading thoughts is an old dream of many people. This dream is displayed in a huge number of science fiction works, in films and fairy tales. But in fact, reading thoughts is a difficult task, for the solution of which modern technologies are needed. And not only the hardware, that is, "hardware", but also software platforms, namely, neural networks.

Relatively recently, a group of scientists taught the neural network to determine what a person sees by analyzing fMRI (Functional Magnetic Resonance Imaging) images. This is a complex problem, but it seems that scientists from China have successfully solved it.

It is difficult because it is necessary for fMRI images, in which there is a lot of noise, to learn how to determine which parts of the brain work and understand what is behind it. The images themselves when using the appropriate scanning equipment are three-dimensional. And this three-dimensional picture of the activity of certain areas of the brain needs to be translated into two-dimensional images - those that the person saw before or during the scan.
')
To obtain fMRI images, it is usually necessary to take into account the fact that the activity of one voxel (active volume) of the brain is caused by the activity of other voxels. As a result, the computer needs to ignore a number of active voxels in order to display a relatively clear image. All this complicates the task of reconstructing what a person sees. In general, this problem is solved, doctors have learned to “separate the wheat from the chaff,” yes, but the process described above reduces the quality of image recognition, seen by a volunteer whose brain is scanned.

Today, functional magnetic resonance imaging is one of the most actively developing types of neuroimaging. Since the early 1990s, functional MRI has begun to gain popularity in areas such as imaging of brain processes due to its relatively low invasiveness, lack of radiation exposure, and relatively wide availability. The method itself is based on the fact that the cerebral blood flow and the activity of neurons are interconnected. When any area of ​​the brain is active, blood flow to that area also increases.

Scientists from China who have worked on this project have decided to find new ways to analyze data from fMRI images. Experts from the Beijing Research Center began to use neural networks, which were consistently taught to determine the relationship between what a person sees and his brain activity, recorded using fMRI.

Initially, volunteers were asked to look at a simple object, during which an fMRI brain scan was performed. A simple object means the image of a number or letter in the figure. As a result, scientists obtained a set of data in the form of a brain scan and the original image. Gradually, the task was complicated, the neural network was trained on a variety of images. In total, about 1,800 original images and images of brain activity were included in the training base. Scientists spent most of their time on learning the neural network, and not on anything else.


The results shown when using the new method of Chinese specialists and various other methods created by other teams of scientists at different times

Further, the neural network gradually learned to restore the original image, which it did not show, according to the scan of the human brain. To improve the results, the neural network was trained to distinguish noise and useful data in images. As a result, computer-generated images that a person saw became clearer and more accurate. Plus, the scientists gave the neural network to compare the original image that the person saw, and the image that the computer reproduced.

The results were very interesting. The neural network learned to reproduce the original image with a high degree of accuracy. In many cases, these images were clearer than in the case of any other techniques created earlier.

“Large-scale experimental comparisons of images show us that we can truly restore human-visible images from fMRI snapshots, and do it more accurately than before,” says project manager Changde Du.

According to experts, this technology allows you to get closer to the time of creating neural interfaces for working with machines without intermediate stages. Perhaps, in the not too distant future, a person will be able to connect his nervous system to computer networks, transferring data and perceiving visual images. This will not be possible tomorrow, but perhaps faster than it’s considered.

arXiv: 1704.07575