Music and neural interfaces: Encephalophone - the neurology of creativity
More recently, the media published information about the invention of a new musical instrument, the encephalophone, which allows music to be performed using the electrical activity of the brain. In my opinion, this development is perhaps the most recent and promising innovation in the field of experimental musical instruments.
Despite the relative accordion (most publications in the media date back to July 2017), I am convinced that the topic is worthy of attention and discussion. Unfortunately, short news reports, besides the fact of the invention of the device, the names of the authors and vague descriptions of the principles of work, contain little useful information.
')
I think that the appearance of a working prototype of an encephalophone is comparable in significance with the invention, for example, Theremin. But it should also be noted that experiments on this topic and attempts to use electroencephalography as an interface for sound extraction have been carried out over the past 60 years.
According to the developers, the new tool will not only be able to change the usual notions about sound extraction and neural interfaces, but also help people with disabilities. Under the cat a story about the history of the invention, close developments, the principles of the work of the encephalophone, the results of the study and prospects for development.
The first use of EEG in music dates back to 1934, when by some Adrian and Matthews the “musicality” of the transformed encephalography signal was noticed. The name encephalophone was first used for the device developed by the neurophysiologist E. Bevers and the physicist and mathematician R. Furt in 1943. The invention of scientists was intended to diagnose neurological diseases, in particular, the localization of damaged neurons.
Like the modern musical instrument, the 43rd encephalophone was able to produce certain sounds depending on the readings of the electroencephalogram, which indicated damage to parts of the cortex, the reticular formation, and other parts of the brain. According to the New York Times on March 2, 1943, the invention of Bevers and Furt was based on the combination of electroencephalography with active sonar technology. Unlike the modern device, the first encephalophone did not allow arbitrary extraction of sounds by the “effort of thought”.
The next breakthrough that brought the creation of an interface for “mental sound extraction” closer was the electroencephalophone, created by Erkki Kurenniemi, member of the Department of Theoretical Physics of the University of Helsinki, Erkki Kurenniemi. Mainly Erkki is known for creating various musical devices and synthesizers (advanced for his time), as well as the creator of the first Finnish microprocessor.
In 1970, the talented scientist became the managing director and technical director of Digelius Electronics Finland, where he initiated the creation of a number of electromusical instruments with unusual interfaces. A series of bold experiments called Dimi (from a contraction (digital music instrument). So DIMI-O got an optical interface connected with movements and recognition of graphic images of notes. Dimi-S or Sexophone used the skin electrical conductivity as a modulator. DIMI-T was a device controlled by the electrical activity of the brain.
Electroencephalophon DIMI-T, as well as other devices of this type, was created on the basis of the EEG apparatus. The experiment can be considered relatively successful, since the frequencies of the signal that reproduced the instrument through a long workout could be tuned to harmonious combinations with notes of the main composition. According to some information, to control the instrument, Kurenniemi tried to use the beta rhythm and so on. activation reaction.
In 1965, the composer and experimental musician Lucier, together with the composer John Cage, created a device that allows you to control percussion instruments with the help of an EEG. A number of useful information for creating a modern instrument were also obtained from the experiments of David Rosenboom, who in the 1970s used the EEG as a source of musical and modulating signals.
Neuropsychologist and futurologist Ariel Garten in collaboration with inventor Stephen (Steve) Mann at the beginning of the 2000s developed a neural interface for controlling music variables under water. The instrument had an original sound, but did not give an accurate sound extraction.
An impressive contribution to the topic was made by Yuan X. and He B., by publishing in 2014 an up-to-date work “Brain interfaces using sensorimotor rhythms: current state and perspectives”, which, among others, formed the basis of the modern encephalophone.
The Brazilian composer and scientist Eduardo Rek Miranda is engaged in the creation of neural interfaces and biological feedback systems. Some of his research is aimed at creating devices that allow the electrical activity of the brain to be made in a sound-producing way. In addition, the researcher conducts experiments with neural networks that generate music.
A new “food” to continue the development of the scientist was given by Roberts, Penny, and Rezek's research on computer management using the EEG (Temporal and spatial multiplication for electroencephalogram based brain-computer interfacing.) Published in 1999. Important information was contained in the related work of 2000 by Pfursteller, Newper, Gugger, etc. (Current trends in graz brain-computer interface (BCI) research.). In these publications, the effectiveness of the application of changes in the alpha rhythm (PDR - the rhythm of the visual cortex) and mu-rhythm (the rhythm of the motor cortex) for the implementation of computer control interfaces through EEG was noted.
Using these developments and their own research on the statistical analysis of the influence of subjective factors on the EEG results, as well as research in the field of accurate recording of changes in alpha, beta and mu rhythms, Eduardo managed to create sensors that later formed the basis of the modern encephalophone.
As I have already noted, the most successful experimental encephalophone was the instrument presented this year by Thomas A. Dioule (Department of Neurology, Swedish Neurological Institute, Seattle, Washington, USA), Juan Pampin (Center for Digital and Experimental Studies, University of Washington (DXARTS), Jacob Sundstrom (University of Washington School of Music) and Felix Darvas (Department of Neurosurgery, University of Washington). The results of a study on the development and testing of a new device This feedback was published in Frontiers in Human Neuroscience, and prior to the advent of this device, experiments on the use of neural interfaces for playing music with an accurate acoustic performance were unsuccessful.
The researchers used a neural interface that uses alpha (PDR) rhythms to control (visual and occipital lobes characteristic of the waking state without physical exertion) and mu rhythms (the motor cortex accompany the state of complete rest). Relying on the mu rhythm opens up possibilities for the use of the instrument by patients with such pathologies as amyotrophic lateral sclerosis, severe spinal cord injuries, traumatic amputations, various forms of paralysis, other conditions that limit natural capabilities and motor activity.
Technically, the device is a 19-channel encephalograph, equipped with electrodes placed on a special cap and an amplifier. This unit transmits EEG results via a USB interface to a computer, which analyzes the frequency and amplitude of signals, assigning them values ​​from 1 to 8. The second computer converts the signals into notes in real time (synthesizes piano sounds).
System Principle / Thomas A. Deuel et al., Frontiers in Human Neuroscience, 2017
Interestingly, to use the tool almost does not require special training. Basically, the study was attended by 15 healthy subjects aged 25 to 60 years old who did not have experience with other musical instruments. They showed very impressive results in a controlled, precise sound extraction.
In this experience, subjects with little or no training (apart from a short 5-minute calibration), within 5 minutes, tried to accurately extract 27 notes using the intuitive “conscious efforts (modulations)”. For example, it was proposed to extract the notes of “do” and “re” of the third and fourth octaves. In the case of a three-fold successful assignment, the program synthesized a major chord and gave out the following task, and in the case of an incorrect one, an unpleasant disharmonious combination of sounds.
So, in the control mode through the PDR rhythm, the accuracy of sound extraction was 67.1%, and when using the mu rhythm 57.1%. This result is significantly higher than the generation of random notes, which would be indicated by an accuracy of no more than 19.03%. The study of trained people with experience in music demonstrated a significant increase in accuracy in controlling the alpha rhythm, but in controlling the mu rhythm the accuracy remained at the level of unprepared subjects.
Percentage accuracy for control through PDR and Mu
Scientists are actively promoting the development through live performances with the jazz project Dr. Gyrus and The Electric Sulci Featuring the Encephalophone. The party on the Ecephalophone is performed by the project manager Thomas Diouel himself. The first performance with an experienced prototype, outrageously decorated with fake wires, dates back to the year 2015 (that is, before the official release of the device).
In 2017, another jazz presentation was held with the same group.
Research and refinement of the tool continues. The developers are convinced that the capabilities of the encephalophone can be significantly expanded. The most promising direction of further development of the topic is the use of the mu-rhythm, due to the high social significance for paralyzed patients.
The most detailed features and results of the study are described here .
Traditional denim paragraph
Unfortunately, encephalophones will not be included in our catalog before the start of serial production, but meanwhile, there is a wide range of other devices for playing and creating music.
In the post used photo content:
www.ncbi.nlm.nih.gov
neuromusic.soc.plymouth.ac.uk
Frontiers in Human Neuroscience
Despite the relative accordion (most publications in the media date back to July 2017), I am convinced that the topic is worthy of attention and discussion. Unfortunately, short news reports, besides the fact of the invention of the device, the names of the authors and vague descriptions of the principles of work, contain little useful information.
')
I think that the appearance of a working prototype of an encephalophone is comparable in significance with the invention, for example, Theremin. But it should also be noted that experiments on this topic and attempts to use electroencephalography as an interface for sound extraction have been carried out over the past 60 years.
According to the developers, the new tool will not only be able to change the usual notions about sound extraction and neural interfaces, but also help people with disabilities. Under the cat a story about the history of the invention, close developments, the principles of the work of the encephalophone, the results of the study and prospects for development.
The first steps of EEG - 30th and 40th
The first use of EEG in music dates back to 1934, when by some Adrian and Matthews the “musicality” of the transformed encephalography signal was noticed. The name encephalophone was first used for the device developed by the neurophysiologist E. Bevers and the physicist and mathematician R. Furt in 1943. The invention of scientists was intended to diagnose neurological diseases, in particular, the localization of damaged neurons.
Like the modern musical instrument, the 43rd encephalophone was able to produce certain sounds depending on the readings of the electroencephalogram, which indicated damage to parts of the cortex, the reticular formation, and other parts of the brain. According to the New York Times on March 2, 1943, the invention of Bevers and Furt was based on the combination of electroencephalography with active sonar technology. Unlike the modern device, the first encephalophone did not allow arbitrary extraction of sounds by the “effort of thought”.
Electroencephalophon DIMI-T Erkki Kurenniemi
The next breakthrough that brought the creation of an interface for “mental sound extraction” closer was the electroencephalophone, created by Erkki Kurenniemi, member of the Department of Theoretical Physics of the University of Helsinki, Erkki Kurenniemi. Mainly Erkki is known for creating various musical devices and synthesizers (advanced for his time), as well as the creator of the first Finnish microprocessor.
In 1970, the talented scientist became the managing director and technical director of Digelius Electronics Finland, where he initiated the creation of a number of electromusical instruments with unusual interfaces. A series of bold experiments called Dimi (from a contraction (digital music instrument). So DIMI-O got an optical interface connected with movements and recognition of graphic images of notes. Dimi-S or Sexophone used the skin electrical conductivity as a modulator. DIMI-T was a device controlled by the electrical activity of the brain.
Electroencephalophon DIMI-T, as well as other devices of this type, was created on the basis of the EEG apparatus. The experiment can be considered relatively successful, since the frequencies of the signal that reproduced the instrument through a long workout could be tuned to harmonious combinations with notes of the main composition. According to some information, to control the instrument, Kurenniemi tried to use the beta rhythm and so on. activation reaction.
Other research and experiments
In 1965, the composer and experimental musician Lucier, together with the composer John Cage, created a device that allows you to control percussion instruments with the help of an EEG. A number of useful information for creating a modern instrument were also obtained from the experiments of David Rosenboom, who in the 1970s used the EEG as a source of musical and modulating signals.
Neuropsychologist and futurologist Ariel Garten in collaboration with inventor Stephen (Steve) Mann at the beginning of the 2000s developed a neural interface for controlling music variables under water. The instrument had an original sound, but did not give an accurate sound extraction.
An impressive contribution to the topic was made by Yuan X. and He B., by publishing in 2014 an up-to-date work “Brain interfaces using sensorimotor rhythms: current state and perspectives”, which, among others, formed the basis of the modern encephalophone.
Contribution of Eduardo Rivers Miranda
The Brazilian composer and scientist Eduardo Rek Miranda is engaged in the creation of neural interfaces and biological feedback systems. Some of his research is aimed at creating devices that allow the electrical activity of the brain to be made in a sound-producing way. In addition, the researcher conducts experiments with neural networks that generate music.
A new “food” to continue the development of the scientist was given by Roberts, Penny, and Rezek's research on computer management using the EEG (Temporal and spatial multiplication for electroencephalogram based brain-computer interfacing.) Published in 1999. Important information was contained in the related work of 2000 by Pfursteller, Newper, Gugger, etc. (Current trends in graz brain-computer interface (BCI) research.). In these publications, the effectiveness of the application of changes in the alpha rhythm (PDR - the rhythm of the visual cortex) and mu-rhythm (the rhythm of the motor cortex) for the implementation of computer control interfaces through EEG was noted.
Using these developments and their own research on the statistical analysis of the influence of subjective factors on the EEG results, as well as research in the field of accurate recording of changes in alpha, beta and mu rhythms, Eduardo managed to create sensors that later formed the basis of the modern encephalophone.
Encephalophone 2017
As I have already noted, the most successful experimental encephalophone was the instrument presented this year by Thomas A. Dioule (Department of Neurology, Swedish Neurological Institute, Seattle, Washington, USA), Juan Pampin (Center for Digital and Experimental Studies, University of Washington (DXARTS), Jacob Sundstrom (University of Washington School of Music) and Felix Darvas (Department of Neurosurgery, University of Washington). The results of a study on the development and testing of a new device This feedback was published in Frontiers in Human Neuroscience, and prior to the advent of this device, experiments on the use of neural interfaces for playing music with an accurate acoustic performance were unsuccessful.
The researchers used a neural interface that uses alpha (PDR) rhythms to control (visual and occipital lobes characteristic of the waking state without physical exertion) and mu rhythms (the motor cortex accompany the state of complete rest). Relying on the mu rhythm opens up possibilities for the use of the instrument by patients with such pathologies as amyotrophic lateral sclerosis, severe spinal cord injuries, traumatic amputations, various forms of paralysis, other conditions that limit natural capabilities and motor activity.
Technically, the device is a 19-channel encephalograph, equipped with electrodes placed on a special cap and an amplifier. This unit transmits EEG results via a USB interface to a computer, which analyzes the frequency and amplitude of signals, assigning them values ​​from 1 to 8. The second computer converts the signals into notes in real time (synthesizes piano sounds).
System Principle / Thomas A. Deuel et al., Frontiers in Human Neuroscience, 2017
Interestingly, to use the tool almost does not require special training. Basically, the study was attended by 15 healthy subjects aged 25 to 60 years old who did not have experience with other musical instruments. They showed very impressive results in a controlled, precise sound extraction.
In this experience, subjects with little or no training (apart from a short 5-minute calibration), within 5 minutes, tried to accurately extract 27 notes using the intuitive “conscious efforts (modulations)”. For example, it was proposed to extract the notes of “do” and “re” of the third and fourth octaves. In the case of a three-fold successful assignment, the program synthesized a major chord and gave out the following task, and in the case of an incorrect one, an unpleasant disharmonious combination of sounds.
So, in the control mode through the PDR rhythm, the accuracy of sound extraction was 67.1%, and when using the mu rhythm 57.1%. This result is significantly higher than the generation of random notes, which would be indicated by an accuracy of no more than 19.03%. The study of trained people with experience in music demonstrated a significant increase in accuracy in controlling the alpha rhythm, but in controlling the mu rhythm the accuracy remained at the level of unprepared subjects.
Percentage accuracy for control through PDR and Mu
Scientists are actively promoting the development through live performances with the jazz project Dr. Gyrus and The Electric Sulci Featuring the Encephalophone. The party on the Ecephalophone is performed by the project manager Thomas Diouel himself. The first performance with an experienced prototype, outrageously decorated with fake wires, dates back to the year 2015 (that is, before the official release of the device).
In 2017, another jazz presentation was held with the same group.
Total
Research and refinement of the tool continues. The developers are convinced that the capabilities of the encephalophone can be significantly expanded. The most promising direction of further development of the topic is the use of the mu-rhythm, due to the high social significance for paralyzed patients.
The most detailed features and results of the study are described here .
Traditional denim paragraph
Unfortunately, encephalophones will not be included in our catalog before the start of serial production, but meanwhile, there is a wide range of other devices for playing and creating music.
In the post used photo content:
www.ncbi.nlm.nih.gov
neuromusic.soc.plymouth.ac.uk
Frontiers in Human Neuroscience
Source: https://habr.com/ru/post/373927/
All Articles