Neurobiological methods, and cool things you can do with them: part 1

Neurobiological methods are similar to the unsung heroes of scientific journalism. Periodically, the media are illuminated with phrases like “brain-guided machines” and “mind reading” (“scientists finally learned to read your DIRTY THOUGHT !!! 11”). But how is this done? And since we have already started talking about this, what methods are used to achieve such futuristic projects as managing virtual reality with the help of our eyes or determining whether a suspect was present at the crime scene? Questions, some questions. The answers are in this post.

Eye tracking and virtual reality

By definition, eye movement tracking (EDC) measures the activity of your eyes. When and how often do you blink? Why are the pupils constricted? Does your gaze stay longer on her gorgeous breasts or on her beautiful eyes? Answering such questions, tracking eye movements helps you find out if a person is concentrated, or is relaxed and tired, if your website is convenient for users, if the patient has autism - this technology is applicable in various areas, the number of which continues to increase.

In contrast to the 1890s, today's technology does not require the immobilization of the eyes using cocaine. Today, a common method of measuring the behavior of the eyes is the light from the near-infrared part of the spectrum to the center of the eye and a comparison of the position of its reflection with the position of the pupil (the position of the reflected light remains in place, and the pupil shifts relative to it). Combining the data with the position of the head, you can extrapolate them to get the direction of the gaze and calculate which points the person is looking at. Typical measurements include fixations (where the gaze lingered on something that caught your attention), their duration, and the time it takes to look at them; saccades (sequential movement of the eyes from one detail of the object under consideration to another); the final way of looking (what you looked at, in what order and for how long). All this can be measured in two ways: by remote devices, and devices fixed on the head. The former usually join the computer screen. The latter are installed on the human head and look like Lady Gaga's futuristic outfit. Such devices allow a person to move freely.

Eye movement tracking is universal technology. Its applications range from marketing research (product development, product placement, packaging) to neurobiology (early diagnosis of Alzheimer's disease, attention and memory research, etc.). But there is one really futuristic idea that should tickle your geek's mind: virtual reality (VR). EDC is becoming more and more popular in the BP community, since it can help create a deep immersion on not the most powerful devices, and also allow the player to influence the virtual world with the help of his eyes alone.
')
Let us look at how BP and EDC lead to improved user experience. The eye has a fovea - a small cavity in the very center of the pupil, which is responsible for sharp eyesight. When we look at something, we see only a small part of our environment in detail (the one that is in the foveal field of vision). Our peripheral vision sees only color and movement, but in a rather blurry and not detailed form. To create the impression that we see much more details of our environment, the brain simply uses our experience and memory, filling in the blanks. If a blurry watermelon is in the field of your peripheral vision, your brain will simply substitute a focused image in its place, taking it from memory - and now it is clearer. And so, the developers of BP are working to create the technology of “foveal rendering” [foveated rendering], designed to significantly reduce the computational load of the computer. She must use our knowledge of brain function and interpretation of the world around. First, EDC informs the program where a person is looking. Foveal rendering, using this information, generates a picture located on the sides of the field of view, with much lower resolution and detail, and directs all computational power to increase clarity and realism when rendering the foveal region. Virtuality is updated according to the movements of your eyes so that the area of ​​your vision always remains clear. In a way similar to vision in the real world, your brain will create the illusion of a clear vision everywhere based on your experience. And although it all looks very cool, this technology is at the beginning of development, since it is rather difficult to create both an inexpensive and small device for EDC.


How it works

Technogiants did not postpone their implementation of the new approach: quite recently Google acquired one of the main startups, EDC. They are said to be working on wireless VR glasses mixing augmented reality and VR. Select a menu item, keeping your eyes on it! Look at something a little longer and enlarge the image! Stroll through the Louvre, and get information about the picture, blinking 4 times! Use the eyes instead of the mouse! Hail our new sovereign and savior, tracking eye movements! What began in 1879 as tracking the eyes while reading, has gone a way long enough to change our view of reality.

Functional magnetic resonance imaging (fMRI) and mind reading

fMRI is similar to modern art. All heard of him, but no one understands how it should be. Do not worry, soon everything will become clear. In fact, fMRI is based on the fact that blood with a high oxygen content behaves in a magnetic field differently than blood with a low oxygen content. At the same time, the more actively the brain area behaves, the more oxygen it consumes. And the more oxygen is required, the more blood enters this area. fMRI can monitor an increase in blood flow and oxygen consumption to find the active region of the brain. This type of imaging is called image formation based on blood oxygenation level dependent (BOLD). The method is non-invasive, you only need to lie in the very loudly working tube of the scanner, and it gives a good spatial resolution of the images (the temporal resolution is worse because there is a delay of 3-6 seconds between the activation of the brain area and the oxygen demand). This tool is used in a huge variety of studies.

fMRI can be used in two ways: it either helps to see brain activation when performing a simple task (for example, in experience periods of idleness change with periods of performing tasks for memory / visual recognition / moral judgments, in general, tasks like “erotica or ice cream”), or it is used to analyze the work of the brain at that moment when we are not busy with a specific task (let’s say, when we dream, looking absently out of the bus window, or smiling and nodding to our colleagues, pretending to listen to their babble about children). The first option, called fMRI, in a state of rest showed that a lot of things happen in the brain even at those moments when we are not busy: he identified several networks that control the brain and allowed us to see how the links between brain regions change in psychiatric diseases. or in different states of consciousness.


Our fMRI study showed that subjects undergoing simple memorization tasks had activity in parts of the brain associated with loud sounds, claustrophobia and loss of jewels.

When we got an idea of ​​how this works [1], what can we do about it? I resist the temptation to declare “reading thoughts” (and hand over a foil cap to you), so I’d rather use a more scientific term, “decoding thoughts based on brain activity”.

Decoding technology is not just looking for an area that responds to, say, a face. They recognize the whole brain activation pattern that matches the recognition of a particular face. Then, by setting up activation schemes for the dofigilliard of images, the algorithm of the “classification of schemes” processes them together with the associated images. As a result, the classifier learns about the connection between images and brain activity at the moment of their recognition and understands what activation scheme a certain picture is likely to cause - say, a photo of a cat or baby. The program, having processed enough examples, can begin studying fMRI scans and try to decode what exactly the person was looking at or thinking about at the time of the scan. The first experiments were simple - in early works, scientists could recognize only the category of objects that the subjects looked at at that time (shoes, boots, scissors, etc.) [2].

Soon after, the decoding mechanisms stepped far forward. At first they were used to determine which of the 120 images people look at [3] - the task is far more complicated than defining a broad category of objects. Then the researchers developed a classifier capable of producing primitive films based on the film viewed by the subject [4]. Since then, the technology has already been used for anything — building visual scenes [5], memory work (what do I think about?) [6] and recognition of intentions [7] (which button do I want to press?). But the classification of intentions is a more difficult task than image recognition. Objects are grouped by color or shape, but how to assign categories to intentions? Another problem is the possibility of generalization. So far, all decoders are working with selected brains, and the development of a standard device for reading thoughts, which can be used to fight crime, should not be expected in the next couple of years. At the current stage, as John-Dylan Haynes (a person working hard on classifier research, as well as my professor (I brag)), says, "The best way to find out someone's intentions is to ask."


How decoding works

Sources

1. www.psy.vanderbilt.edu/tonglab/sonia/Personal/fMRI_Basics.html
2. dx.doi.org/10.1016/S1053-8119 (03) 00049-1
3. gallantlab.org/_downloads/2008a.Kay.etal.pdf
4. www.ncbi.nlm.nih.gov/pubmed/21945275?dopt=Abstract&holding=npg
5. journal.frontiersin.org/article/10.3389/fnhum.2014.00059/full
6. www.jneurosci.org/content/32/38/12983?ijkey=3b948eedde2b0698790b3d2f0d7ea14f66079dee&keytype2=tf_ipsecsha
7. www.ncbi.nlm.nih.gov/pubmed/21486293