Biomechanics and artificial intelligence in medicine. Lecture at YaC 2017

In this lecture, Egor Tokunov, the head of Intellect and Innovations, talks about how neural networks can influence a person’s motor skills and how they can help in the rehabilitation of patients with motor disorders.

Yegor spoke at the digital health section at the Yet another Conference 2017 .


')

You all know the movie "The Matrix." Inspired by the viewing, we thought: “Why don't we make a system where a person just in a short time, without using any explanations, can take and get some kind of new motor skill?”.

We lacked two parameters. The first is what can we transmit to a person in the brain? And the second is how to transfer it directly to the brain? However, we thought and found a solution.



I will tell you about the path that we walked and which began almost in the 90s, when we were engaged with athletes on the basis of the Russian State University of Physical Culture and prepared them for participation in competitions. We received a huge amount of information about how the movement of athletes. By obtaining this data using motion capture systems and analyzing them, we could load them into special calculation tools and obtain optimized movements, which we needed to bring athletes' equipment to. However, in the process of working with coaches, we realized that the data transmission channel to the coach, when we try to interpret something from the figures we received, and the coach, in turn, tries to convey this to the athlete, is very inefficient.



Therefore, we thought that if we understand how to manage the process directly in the athlete's brain, then, going to the clear and natural language of the rules that guide people, we can most simply and most fully explain to him what is required of him, to achieve results.

We had an experiment. In 2003, a dissertation was defended on the use of a neural network and fuzzy controllers for motion control. As an experimental sample was used a three-link with flexible rods, which emulated the muscles. The developed neural network was trained on human movements. Then there was the following task: on the basis of this training, after receiving some kind of influence in the upper part of the three-mates, restore balance. A trained neural network did an excellent job with this task. Then it was necessary to understand how the three-link control is being built. For this, a system of fuzzy regulators was built inside the neural network, which, in contrast to the modern classical mechanism of using artificial intelligence, was not a black box. It was written in such a way that we could later look at these rules.



We also thought how we can convey this information to the person, bypassing some explanation. Understanding and seeing the experiments that our colleagues put ... 70% of the information that a person uses to carry out his movements comes from visual perception. In this case, a classic example is presented of using the so-called redirected walking in virtual reality, which allowed showing the unreal world to a person and constantly turning it around to make a person walk, say, in a circle. At the same time, the person had the full impression that he was walking along an endless straight line.

We thought even wider. If we connect some tactile stimuli, the effect may become even stronger. Also studying the literature, we found many examples confirming this. In the classic form, here is an example when, in addition to the visual illusion, a person received more tactile irritation. Several attempts at repeating this exercise, linking, say, the sixth finger to the little finger by tactile sensations, as well as asking the person to move the sixth finger, gave scientists the opportunity to record electrical impulses and myography from the little finger muscles.



To test whether our guesses are correct and whether this system is applicable on real movement, we developed primitive artificial intelligence, which at that time could process this data and translate changes into simple graphic primitives in real time. For this purpose, a program was developed that included a test on the Stabilan stabilographic platform, where the test subject's task was to repeat the deviation of the center of mass in accordance with the marker he saw on the screen. In this example, the red bar is the standard marker, and the blue bar is the subject who was standing on the platform. His task was to repeat the movement behind the red marker. In the process, we tested the work of the neural network, which really showed that the forecast that we received when the network was working, exactly corresponded to the errors and corrections that were given by the person during his movement.

We also tried to add an extra effect here. In carrying out this movement, we began to exacerbate the human error by performing the tracking movement. The more we aggravated and the earlier we showed a mistake to a person in real time, the faster and more accurately he began to adjust his training. Thus, we achieved the following effect: without explaining anything, we forced the person to perform some ideal movement, which corresponded to the optimal calculated trajectory.



Also on this slide is a picture of the very first neural network that we used. Its peculiarity lies precisely in the fact that Layer 2 was applied here, which is on the slide and which was exactly the system of fuzzy rules. Drawing rules that were based on a language understandable for a person and contained such instructions as “more”, “stronger”, “slower”, “faster”, and knowing the given weights from the next level, we could understand which rules this person often just uses to perform one movement or another. Such a scheme gave us the opportunity to convey information to a person in the language in which he himself performed and built these movements.

With the development of new means of capturing and obtaining data on the movement, we began to think in the direction of a deeper human immersion in order to achieve the strongest effect from the impact of our system. First, we used optical motion capture systems and passive markers. Somewhere in 2015, with the advent of modern virtual reality helmets, we started using the motion-tracking and tracking system SteamVR Tracking. Another difference was that these systems are virtually identical in accuracy, but the second system gave much greater accessibility to the end user.



In the course of our work, we tried a huge number of tools - we wanted to understand what methods of influences and data we needed to train neural networks and obtain the most optimal algorithm for human exposure. AnyBody Modeling System programs were used to study how muscles work and to calculate their parameters. Visual 3D is a program that usually comes bundled with biomechanical motion capture systems and allows you to calculate the mechanics of movements, power, speed in joints, etc. And, the OpenSim program for creating and connecting all of this together for a person who is conditionally in our team called homunculus. Inside OpenSim, we connected all the knowledge and tried to simulate how the influence exerted would affect the change in the motor program.

We looked in the direction of how to optimize the system in terms of pedagogy. Through experiments, they realized that modern and classical methods of reward and punishment are also very effective. And, moreover, having data on the forecast of human movement, we can make the promotion very targeted and show them in advance in real time.

Conducting experiments with modern helmets, we saw that virtual reality and the level of immersion that it provides, raises the effectiveness of neuroplasticity several times.

As a result, we have combined all previous research and development into a single system, which allowed, by showing a man a directional visual illusions, to make him move exactly as we want. We also carried out preliminary computational optimizations of the given target movements for the parameter we needed - for example, in sports, these were labor costs, speed of movement, forces and energy. Optimization made it possible to rebuild the motor program by practically direct downloading of this program to the brain by several repetitions of this exercise, either using virtual reality tools or using some other means of classical visualization: projectors, TV.



We transferred our product to the rehabilitation system, since all similar mechanisms work perfectly for it, if a person has cognitive functions.



The system currently consists of three main components: this equipment, which is some kind of motion capture system, a computer, and a visualization system. Here special software is applied, which includes a cloudy artificial intellectual system for the primary acquisition of data on the method of human movements and his motor handwriting. It can also output another neural network, which is then used in real time to predict movements. Both the methodology and the rehabilitation exercises themselves were developed by us in close cooperation with the Scientific Center for Neurology of the Russian Academy of Sciences.

The solution is now divided into three product levels. Each has its own advantages and disadvantages. In particular, the Kinect prefix is ​​used in the Home solution, so there is no need to install additional sensors on the patient, but the accuracy of the predictions and operation of Kinect is somewhat lower. Which, however, makes it very affordable.

Solution Pro is an intermediate product where, in combination with a marker-free capture system, we can use a virtual reality helmet. This is such an optimal level, where a sufficient level of immersion is achieved, and, in principle, sufficient accuracy.

The third solution is B2B, which uses total immersion in virtual reality. Additional sensors are put on a person, allowing him to obtain a model of movements with very high accuracy and — using the visual illusions developed by us and the listed algorithms — to bring the rehabilitation process to the greatest efficiency.



Our colleagues with whom we communicated say that the system will find its ideal application, including in case of cerebral palsy, during rehabilitation after oncological operations, traumatic brain injuries and ADHD. We are now moving into these areas.

But we also did not forget about our origins. We came from athletes and developed several examples and cases to develop and improve motor skills. In particular, here is the case when we conducted an experiment and tested 70 people in a virtual dash. Working with our system showed a 50 percent increase in shooting accuracy compared to people in the test and control groups who performed this exercise simply by repeating. This set included about 600 shots, of which the first 100 were used to train the neural network, 400 as the training process, and the last 100 to determine the control result.



Summarize. Where are we going? At the moment, we have confirmed that efficiency compared to a simple repetition system is 50% higher. We think that with the improvement of our algorithms and directed illusions, as well as with increasing neural network layers and connecting complex movements, this training or rehabilitation system will be more efficient than working with a specialist. Now we are working on complex movements, development. The third level, which we are just thinking about: how to connect our system with some cognitive functions and a decision-making system. Thanks for attention.