📜 ⬆️ ⬇️

Experiments with Neural Interfaces in JavaScript

The author of the material, the translation of which we are publishing today, says that for the last couple of years he has noticed a steady interest in neurotechnology. In this article, she wants to talk about her experiments with various hardware and software systems that allow you to establish a connection between the brain and the computer.



Prehistory


I do not have basic computer education (I studied advertising and marketing). I mastered programming on courses in General Assembly.

When I was looking for the first job, I started experimenting with JavaScript and with different devices. In particular, my first project was the management organization of the Sphero robotic ball through arm movement using Leap Motion .
')

Sphero ball, hand operated with Leap Motion technology

This was my first experience using JavaScript to manage something outside of the browser. I was immediately, as they say, "hooked."

Since then, I have spent a lot of time working on interactive projects. Every time, taking on a new project, I tried to find for myself more and more complex tasks. So I constantly developed and learned something new.

After experimenting with various devices, in search of another interesting task, I came across sensors of brain activity from the NeuroSky company.

The first experiments with the neuro headset


When I was interested in experiments with sensors of brain activity, I decided to purchase a NeuroSky neuro headset. She was much cheaper than other similar offers.


NeuroSky Neuro Headset

I didn’t know if my qualification was enough to write at least something for such a device (I just finished programming courses at that moment), so I decided to choose something cheaper so that if the task turns out to be for me prohibitively difficult, do not waste too much money. Fortunately, a JavaScript framework has already been created to work with the headset, so it was quite easy to start the experiments. In particular, I used the assessment of the level of my attention to control the Sphero ball and the Parrot AR.Drone quadrocopter.

During the experiments, I quickly realized that this neuro headset is not particularly accurate. She has only three sensors, so she allows to obtain rather approximate data on brain activity. The device gives access to the raw data from each sensor, which allows, for example, to visualize this data. But the fact that the headset has only three electrodes does not allow us to draw any serious conclusions about what is happening in the human brain based on the data obtained from it.

When I decided to look for other devices for reading indicators of brain activity, I found a nemo headset Emotiv Epoc . I got the feeling that this thing has more serious features in comparison with the NeuroSky headset, so I decided to buy it to continue my experiments.

Before I talk about how Emotiv Epoc works, I suggest to briefly talk about how the human brain works.

How the brain works


I can't call myself a great neuroscience expert, so my story about the brain will be rather superficial. Namely, I want to talk about a few basic things that need to be known to those who want to better understand how the neurohearnes work.
The brain consists of many billions of neurons - specialized cells that are engaged in the processing, storage and transmission of information. Different parts of the brain, consisting of neurons, are responsible for different physiological functions.


Different parts of the brain (source - macmillan.org.uk)

For example, let's talk about how the brain controls the movements. Parts of the brain, such as the primary motor cortex and the cerebellum, are responsible for movement and coordination. The signals of the corresponding neurons act on the muscles, which leads to movements.


Neuron Anatomy

As I said, an extremely simple description of the brain is presented here, but for us the most important thing is that the activity of neurons can be tracked by electroencephalography (EEG) methods, by reading indicators of brain electrical activity from the surface of the scalp.

Other technologies can also be used to monitor brain activity, but their use implies surgery. In particular, we are talking about electrocorticography - with this approach, the electrodes are superimposed directly on the cerebral cortex.

Now that we’ve figured out that the brain generates electrical signals that can be read as we work, let's talk about the Emotiv Epoc headset.

How the neuro headset works


The Emotiv company produces several types of neurogarnitures:


The Epoc headset has 14 sensors (also called “channels”) located in different places of the head.

The following figure, on the left, shows the layout of electrodes 10-20 recommended by the International Federation of Electroencephalography and Clinical Neurophysiology. Each electrode corresponds to a specific area of ​​the brain. The use of the 10-20 system allows you to follow a certain standard when creating various devices and performing scientific research of the brain.
The figure on the right shows the layout of the electrodes of the Emotiv Epoc headset. To compare it with the 10-20 system, the selection is green and orange.


Comparison of the international electrode placement system 10-20 and the Emotiv Epoc headset

The 14 Epoc channels are not so much, but the electrodes are placed evenly across the scalp. This gives hope that with Epoc it is possible to obtain fairly accurate information about brain activity.

The headset captures sensors at a frequency of 2048 samples per second (SPS). At the same time, the user has access to the signal sampling rate of 128 or 256 SPS. The device is capable of capturing brain waves with a frequency of 0.16 to 43 Hz. There are various rhythms of the brain, their brief characteristics are shown in the following figure.


Brain Wave Types

Why is it important? The fact is that depending on the application that needs to be built on the basis of an electroencephalograph, we may need to pay special attention to brain waves of a certain frequency. For example, if we need to create a program to help meditators, then we will probably be interested only in theta waves, whose frequency is 4-8 Hz.

Having dealt with the principles underlying electroencephalography, let's talk about the capabilities of Emotiv Epoc and related software.

Emotiv Epoc features


Emotiv software is not open source, a special license is required to access the raw sensor signal. Under normal conditions, the following features are available when working with Emotiv Epoc:


In order to use the recognition of mental commands, first the user needs to train the system . Learning data is saved as a file.

If you want to develop your own programs for Emotiv Epoc, you can use the API Cortex and the corresponding SDK (its support is discontinued after the release of version 3.5). If you want to use JavaScript, you can take a look at my development, the Epoc.js library.

Epoc.js library


Epoc.js is a framework designed to interact with Emotiv Epoc and Insight devices using JavaScript. This framework gives the developer access to the above features of the systems from Emotiv and allows you to interact with the emulator .
Here is the simplest project based on Epoc.js:

const epoc = require('epocjs')(); epoc.connectToLiveData('path/to/profile/file', function(event){  var action = event.blink === 1 ? 'blinking' : 'not blinking';  console.log(action); }); 

In this sample code, we include the Node.js epocjs module and create an instance of the corresponding object. Then we call the connectToLiveData method of this object, passing it the path to the file with user data obtained after training the system, and the callback function. This function is passed an event object that contains various properties available for tracking. For example, if we want the program to respond to blinking, the event.blink property is event.blink .

Each such property can be set to either 0 or 1. A unit in the property value means that the system has recorded a corresponding event. A full list of these properties can be found here .

The described library was created using the Emotiv C ++ SDK, Node.js and three modules for Node.js: Node-gyp, Bindings and Nan. When developing it, an approach was used that can now be considered obsolete. Now the use of N-API is relevant.

Having discussed the various possibilities of neuroheternologies and the ways of program work with them, I will tell you about several prototypes created by me that involve the neurointerface.

Prototypes


â–Ť1. Keyboard


This is how the keyboard is controlled by eye movements.


Eye Movement Prototype Keyboard

This was my first project created using Emotiv Epoc. I wondered if it was possible to create a simple interface using a neuro headset that allows a person to interact with a computer using eye movements. For example, when you translate the view to the right or left on the keyboard, the corresponding keys are highlighted. In order to "press" the highlighted key, you need to blink. The corresponding letter appears in the box above the keyboard.

This project looks very simple, but the most important thing is that it works.

â–Ť2. WebVR


In my second project I used mental commands. Creating it, I wanted to understand whether it is possible to control an object that is in three-dimensional space, just thinking about something.


Thought-driven web interface

Here, to create a simple three-dimensional environment, I used the Three.js library, the Epoc.js library was used to recognize mental commands, and web sockets were used to send data from the server to the client.

â–Ť3. Iot


Starting the third project, I wanted to explore the possibilities of controlling real devices with the help of mental commands. I have been interested in IoT development using JavaScript for quite some time, so it was interesting for me to know what happens if I combine the Parrot quadrocopter and the neuro headset.


Quadcopter

All the projects described above, all the prototypes created, are pretty simple designs that I created in order to test some ideas in practice and evaluate the possibilities and limitations of neural interfaces.

Neural Interface Restrictions


The word “neural interface” sounds amazing, and when it turns out that a computer can be controlled by the power of thought, it may seem that here it is - the future, but in fact, neurocomputer technologies still have quite a few limitations.

â–ŤNeediness in learning


It’s quite normal that users have to train the system, during which they record brain waves and compare them with certain commands, but for many, such a step is an obstacle to adopting a new technology. It's hard for me to imagine that someone will spend time learning neurocomputer systems, except that such a system will be really needed by someone, and the accuracy with which she recognizes mental commands will be at a very high level.

â–Ť Delays


When I developed my prototype, based on computer perception of mental commands, I found out that there was some delay between the moment when I started thinking and the program when I reacted to this thought.

I think the point here is that the machine learning algorithm used in the prototype receives data from the device in real time. In order to recognize the thought, which he had previously learned, he needs indicators collected over a certain period of time.

This has an impact on which programs can be built on the basis of a neural interface. For example, a program that helps to meditate looks quite realistic, since the delays between a change in brain state and the program’s response do not particularly affect the results of such a program. However, if you aim to create something like a wheelchair controlled by thoughts, the problem of delays becomes much more acute, calling into question a similar development.

â–ŤNon-invasive technology and accuracy of indicators


Electroencephalogram removal devices are great for everyday use in ordinary life situations. It is enough to put on a headset, putting a special gel on the sensors, and everything is ready. However, the fact that signals generated by the brain are read from the scalp, and not, say, from the surface of the brain itself, impairs the accuracy of such signals.

If we talk about the frequency of removal indicators, it is very good in existing devices. The same cannot be said about the spatial characteristics of the data obtained. EEG devices can only read signals originating in those parts of the brain that are close to the surface of the head. About what happens in the deeper structures of the brain, using a similar approach can not be known.

â–Ť Community acceptance


Neuro headset - this is not the cutest and most familiar device. I think that as long as these headsets look like they are now, they are unlikely to be worn in public places. As technology advances, it may be that devices are created that may be hidden in accessories like hats, but even here you may encounter a problem related to the fact that such devices will be inconvenient when worn for a long time.

EEG sensors should fit quite tightly to the scalp to remove the quality indicators of brain activity. And if their pressure immediately after putting on the headset can almost not be felt, over time it begins to cause discomfort. Moreover, if the sensors also need to be applied gel, it turns into an additional barrier to the wide distribution of neurohears.

As you can see, the current state of affairs in the field of neural interfaces suggests that they are unlikely to be widespread. However, if we talk about the future, then we can say that such devices have interesting perspectives.

Neural Interface Features


If you take into account the current state of technology and think about what they may become in the future, you can find several options for their application.

â–ŤHelp people with disabilities


I would like the neuro headsets to help people with disabilities live a fuller life and be more independent.

This is what I thought about when I created my first prototype - a keyboard, controlled by eye movements. This development of mine is far from the level when it could be used in practice, but working on this project, it was interesting for me to understand whether a quite affordable consumer device can really help someone. Not everyone has access to complex medical systems, and I was just amazed that the not very expensive thing, which can be freely purchased from the online store, is able to solve important and necessary tasks.

â–ŤMental practices


Mental practices, in particular - meditation - this is the scope of neurohears, which already today attracts a certain attention (for example, the Muse headset helps to meditate). It is about helping someone who wants to meditate, to do everything right.

â–ŤHelp in solving health problems


If the neuro headsets would penetrate into our lives as much as mobile phones, we could probably create applications that can respond to health problems. For example, it would be great if there were applications that, based on an analysis of brain activity, would help fight strokes, panic attacks, epileptic seizures.

â–Ť Improving labor productivity


The neuroheckair can help meditate, which means that with its help you can really figure out what time of day a person concentrates. This information, obtained by regularly wearing a headset, can help to understand when it is best to engage in some kind of activity. You can even imagine that the work schedule will be organized in accordance with the individual characteristics of a person, which will increase the productivity of his work.

Art


I like, on my own initiative, during off-hours, to explore phenomena at the intersection of art and technology. I believe that one should not underestimate the work in this direction related to neural interfaces, since, although they may seem “frivolous,” they help to better understand technologies that will prove useful in more “serious” cases of their use.

Combination of electrical brain activity sensors with other sensors


Recently, I had the idea that EEG sensors should not be considered as something completely independent. Our brain perceives the world through the senses. He is not able to see without eyes and to hear without ears. Therefore, if we want to extract the maximum benefit from the data on the electrical activity of the brain, we may need to monitor other vital signs.

The main problem here is that all this can lead to the fact that people will be literally hung with various sensors.


Are there too many sensors? (source of illustration - cognionics.net)

Perhaps no one will constantly wear the sensors shown in the previous figure.

Openbci


A few weeks ago, I acquired something new — the OpenBCI complex. My next step is to study the raw data obtained from EEG sensors and apply machine learning methods to this data. OpenBCI is an open source project, so their development seems to me perfectly suitable for this purpose. I still didn’t work much with their headset, so far I only had enough time to connect it to a computer and set it up. Here is how it all looks.


Openbci

Results


The author of this material says that he continues to study neural interfaces. Hopefully, her story will help those who are interested in this topic, but do not dare to begin practical actions, to take the first steps in the field of the use of neurohears. If all of this is interesting to you - here is another one of our publications about neuroheaders and JavaScript about Muse.

Dear readers! Do you plan to do experiments with neurohearnes?

Source: https://habr.com/ru/post/433874/


All Articles