Magical gestures as a challenge for an electronic engineer
The magic wand is cool, but it's much more awesome to do magic with just the movement of your hands. To make this possible, in 2014 we began to recognize gestures and made a glove on accelerometers for a role-playing game. The project turned out to be difficult and with far-reaching consequences.
So, the game “There is such a craft” in the world of Andrzej Sapkowski . Most of the characters are teenagers with the makings of witches who study in the castle of Kaer-Morhen. They have to gut monsters, boil elixirs, fence by special rules - and practice magic gestures. Someone is dying, someone is swinging to an adult witch.
Signs Aard and Yrden in computer games around the world of Andrzej Sapkowski.
')
It is logical to put on the player’s left hand (in the right sword) a device for reading the movements of the fingers - for example, a glove. The simplest solution would be to use strain gauges in it. But in 2013-2014, the strain gauge cost 110 rubles. This amount, multiplied by five fingers and 25 devices, gave very heavy expenses. Unfortunately, medium-sized production teaches well to consider: we had to spend no more than 2 thousand rubles for one glove, and could not afford a quarter of this amount to drop on the sensors.
We turned to accelerometers. Accelerometer cost quite realistic 30 rubles. For the first experiments, only one sensor was used, fixed on the sword - and it soon became clear that it was impossible to work with dynamic movement, as initially wanted. In fact, the accelerometer gave only information about the presence or absence of a wave.
Working with static positions instead of dynamics was chosen because in statics we could clearly draw the line between where we need the position and where we do not. In the dynamics of the acceleration, which arise during the rotation of the accelerometer could be similar to anything.
So, it became clear that it is necessary to work with static positions. But what? The human hand has more than 20 degrees of freedom, is it worth it to stick an accelerometer on each phalanx? The sensor costs money, and the fingers are all of different lengths and flexibility ...
Using the accelerometer, in the language of aircraft modellers, we can determine only two types of movements - roll (swing left-right) and pitch (up-and-down head). And the yaw (turns left and right) - no longer, because the force of gravity remains unchanged.
The accelerometer does not see the difference between these two gestures. Problem!
In the end, we decided to put one sensor on the back of the hand and one on each finger, in the region of the third phalanx. It gave some room for making gestures.
The first prototype looked like a Discovery board with sensors connected. The python script that processed the data (six three-degree sensors - a vector of 18 numbers corresponded to each position of the glove), we will not show anyone - it is not that perfect, since 2014 we have learned a lot.
The very first prototype - it perfectly perceives the gestures of Vladimir Yashunsky, his main developer.
A reference value was recorded and compared with the resulting data. No signs of trouble.
Two months before the game, the series was ready - 25 sensors sealed on wires, glued to construction gloves, connected to 25 boards with batteries and LEDs.
Work process - soldering LEDs.
Each accelerometer was put on hot melt glue. So, occupied a slightly different position. New gloves with a sin in half agreed to accept the gestures of Vladimir, but only him.
Prototype and series.
What to do? Before the game, two months, gloves announced, players learn gestures. Theoretically, it was possible to make your own firmware for each glove. Practically - terribly not wanted. I also didn’t want to expand the allowable range of movements, otherwise gestures would start to be confused with each other. I had to organize the collection of statistics
We invited 10 people (with different lengths and flexibility of fingers) and asked everyone in every sign to make all the gestures. The volunteer made a gesture. Vladimir was convinced that the system recognized him correctly, confirmed that the gesture was made admissible and wrote down the value in a folder. Needless to say that the averaged values ​​were a little similar to those given by Vladimir himself?
Having collected statistics, for each gesture, the expectation and variance were calculated.
Ten gestures were invented. But a single gesture, leading to a positive result, is fraught with incidents. It is quite a shame to raise a hand to greet a person, and inadvertently incinerate him.
It was decided to retreat from the text of Sapkovsky and include three gestures in each Sign.
The first gesture is the initiating gesture of the Sign, it is unique. On the second one, strength is gained, this gesture is the same. And it ends with a third - executive - gesture. There were three kinds.
There is such a rule - a person is inclined to blame someone else for all his failures. And if failure seems to be electronics-related, they will blame electronics for the problem. For us, this means this: if you can interpret the action in favor of the player, you have to do it. But on the path of goodness are false-positive operation, but they also will not forgive.
To control the correctness of the execution of the Mark, they first wanted to write a ring buffer. But it was impossible to prevent a large number of false positives or, conversely, failures. Therefore, we went the other way, through the state machine. If the glove is only turned on, the first gesture may be only one of the five initiating gestures. They cause a weak vibration, and the “power set” made first is ignored.
Maximum user loyalty was manifested as:
After the first correct gesture, the user has 10 seconds to make the second correct gesture. In the interim, he can do any nonsense with his hands.
This is not counted as an error:
See, the user correctly initiated Aard (open palm), switched to gaining strength, but hesitated and made the wrong gesture of descent. With ring buffering, this would interrupt the execution of the mark, the user would be upset. The state machine ignores this small blot. The user remembers the correct gesture of descent and the glove works.
And somewhere glove still could throw dirty trick.
Did you want to push the comrade away with Aard, and inadvertently burned with a similar sign Igni?
Between the correct performances of Aard (open hand) and Igni (pinch) there are some more unkempt options - 5 accelerometers are pointing up, and the one on the back of the hand is not. This is something fraught with surprises for the witch.
It was decided that the glove should communicate with the user, and it will communicate with vibration (vibration we use often, it allows you to create devices without technogenic screens, without unmasking sound and light - but still able to communicate with users).
When performing the initiating gesture, the glove vibrates weakly, on the second gesture it vibrates more strongly. They explained to the users: this is because you feel how a lump of force fills your palm. A correctly executed mark was indicated by a flash of the corresponding color, everyone in the game knew how to react to this.
All the game technicians — the monsters with which the witchmen fought — knew how to react to light gloves. And I had to face, for example, with such a manticore on jolly jumpers.
Photo by Lilia Barladian
The wicked swamp creature the witch girl let in close, in order to surely cast aside Aard.
Photo by Lilia Barladian
Most of the magic with a glove was supposed to occur at night, but just in case it used regular output LEDs (brighter than RGB). In addition, on the output LEDs there is no risk of mixing colors (those who worked with RGB noticed that in certain angles it is clear that yellow light is a combination of red and green, and so on).
The witch cannot make a gesture. Under the reinforced adhesive tape, you can see the battery from a mobile phone, above - a vibration motor.
Photo of Catherine Grigorova
Well, when the system has one entry point, after which it begins to recognize something. Ok google And what to do when there are five such points? Users regularly felt a slight vibration, the power raging in them.
Witcher in a glove. On the chest, the medallion is simple electronics, a color indication of drunk potions.
Photo by Elena Mauritius
Then we came to the forest. There came out a few problems. First, no one wore top gloves — leather — they were uncomfortable in them. So you can see the uniform cyberpunk in the photos, but what is much more unpleasant is that the wires died from this. Several vibration motors died, they were soldered right on the range. Perhaps the main problem of the glove was the year of its creation. We didn’t have a 3D printer, it didn’t pack anything, everything kept on hot melt.
Witcher.
Photo by Elena Mauritius
However, the game went without global jambs.
Returning to Moscow, washing the gloves in the washing machine (with the controller and the battery disconnected, of course), having come to our senses and considered losses, we thought about modernization.
It would be possible to use not an IDE cable (it is not intended for wearable electronics at all and was used from bitter poverty), but a telephone “redhead cable”. And letting him on the drawstring in a glove, so the problem with washing would not have gotten up at all. And connect the IR LED to combine the glove with the laser tag. AND…
And then came the sale of myobrazlet . We played with him and realized that we would not further develop the glove. Unpromising. Reading the signal from the muscles is much more convenient than the signal from the fingers. (However, for role-playing use myobraslt is not suitable because of the price). Gloves went for another game in Primorye and constantly toured exhibitions.
We believe that the experience was useful, although sometimes painful - it came in handy when creating a bracelet for recognizing a dynamic gesture, this is our fresh project. Mindful of how we worn out with the sensors, soldered to the wires and distributed on the fabric, in the bracelet there were two soldered to the usual reliable charge. The glove worked in statics with a large amount of input data (18 numbers from 6 sensors), the position of the bracelet is described by 2 numbers (using the Mahoney algorithm, we convert 9 values ​​obtained from the sensors).
However, there are those who are interested in experimenting with a glove. In 2015, the glove was shown to the participants of the design school, and some considered it a challenge. Daniil Velovaty (then a schoolboy, now a student of MIPT) put together his own version. Unlike us, he realized the data transfer from the device and surprised visitors to the exhibitions that he remotely turned on the kettle with a sign. Daniel's sign language is also his own.
So, the game “There is such a craft” in the world of Andrzej Sapkowski . Most of the characters are teenagers with the makings of witches who study in the castle of Kaer-Morhen. They have to gut monsters, boil elixirs, fence by special rules - and practice magic gestures. Someone is dying, someone is swinging to an adult witch.
Signs Aard and Yrden in computer games around the world of Andrzej Sapkowski.
')
It is logical to put on the player’s left hand (in the right sword) a device for reading the movements of the fingers - for example, a glove. The simplest solution would be to use strain gauges in it. But in 2013-2014, the strain gauge cost 110 rubles. This amount, multiplied by five fingers and 25 devices, gave very heavy expenses. Unfortunately, medium-sized production teaches well to consider: we had to spend no more than 2 thousand rubles for one glove, and could not afford a quarter of this amount to drop on the sensors.
Component selection and sign language creation
We turned to accelerometers. Accelerometer cost quite realistic 30 rubles. For the first experiments, only one sensor was used, fixed on the sword - and it soon became clear that it was impossible to work with dynamic movement, as initially wanted. In fact, the accelerometer gave only information about the presence or absence of a wave.
Working with static positions instead of dynamics was chosen because in statics we could clearly draw the line between where we need the position and where we do not. In the dynamics of the acceleration, which arise during the rotation of the accelerometer could be similar to anything.
So, it became clear that it is necessary to work with static positions. But what? The human hand has more than 20 degrees of freedom, is it worth it to stick an accelerometer on each phalanx? The sensor costs money, and the fingers are all of different lengths and flexibility ...
Using the accelerometer, in the language of aircraft modellers, we can determine only two types of movements - roll (swing left-right) and pitch (up-and-down head). And the yaw (turns left and right) - no longer, because the force of gravity remains unchanged.
The accelerometer does not see the difference between these two gestures. Problem!
In the end, we decided to put one sensor on the back of the hand and one on each finger, in the region of the third phalanx. It gave some room for making gestures.
The first prototype looked like a Discovery board with sensors connected. The python script that processed the data (six three-degree sensors - a vector of 18 numbers corresponded to each position of the glove), we will not show anyone - it is not that perfect, since 2014 we have learned a lot.
The very first prototype - it perfectly perceives the gestures of Vladimir Yashunsky, his main developer.
Serial production and unexpected plot twist
A reference value was recorded and compared with the resulting data. No signs of trouble.
Two months before the game, the series was ready - 25 sensors sealed on wires, glued to construction gloves, connected to 25 boards with batteries and LEDs.
Work process - soldering LEDs.
Each accelerometer was put on hot melt glue. So, occupied a slightly different position. New gloves with a sin in half agreed to accept the gestures of Vladimir, but only him.
Prototype and series.
What to do? Before the game, two months, gloves announced, players learn gestures. Theoretically, it was possible to make your own firmware for each glove. Practically - terribly not wanted. I also didn’t want to expand the allowable range of movements, otherwise gestures would start to be confused with each other. I had to organize the collection of statistics
We invited 10 people (with different lengths and flexibility of fingers) and asked everyone in every sign to make all the gestures. The volunteer made a gesture. Vladimir was convinced that the system recognized him correctly, confirmed that the gesture was made admissible and wrote down the value in a folder. Needless to say that the averaged values ​​were a little similar to those given by Vladimir himself?
Having collected statistics, for each gesture, the expectation and variance were calculated.
Glove eyes of the user
Ten gestures were invented. But a single gesture, leading to a positive result, is fraught with incidents. It is quite a shame to raise a hand to greet a person, and inadvertently incinerate him.
It was decided to retreat from the text of Sapkovsky and include three gestures in each Sign.
The first gesture is the initiating gesture of the Sign, it is unique. On the second one, strength is gained, this gesture is the same. And it ends with a third - executive - gesture. There were three kinds.
Retreat. Man and Electronics
There is such a rule - a person is inclined to blame someone else for all his failures. And if failure seems to be electronics-related, they will blame electronics for the problem. For us, this means this: if you can interpret the action in favor of the player, you have to do it. But on the path of goodness are false-positive operation, but they also will not forgive.
To control the correctness of the execution of the Mark, they first wanted to write a ring buffer. But it was impossible to prevent a large number of false positives or, conversely, failures. Therefore, we went the other way, through the state machine. If the glove is only turned on, the first gesture may be only one of the five initiating gestures. They cause a weak vibration, and the “power set” made first is ignored.
Maximum user loyalty was manifested as:
After the first correct gesture, the user has 10 seconds to make the second correct gesture. In the interim, he can do any nonsense with his hands.
This is not counted as an error:
See, the user correctly initiated Aard (open palm), switched to gaining strength, but hesitated and made the wrong gesture of descent. With ring buffering, this would interrupt the execution of the mark, the user would be upset. The state machine ignores this small blot. The user remembers the correct gesture of descent and the glove works.
And somewhere glove still could throw dirty trick.
Did you want to push the comrade away with Aard, and inadvertently burned with a similar sign Igni?
Between the correct performances of Aard (open hand) and Igni (pinch) there are some more unkempt options - 5 accelerometers are pointing up, and the one on the back of the hand is not. This is something fraught with surprises for the witch.
It was decided that the glove should communicate with the user, and it will communicate with vibration (vibration we use often, it allows you to create devices without technogenic screens, without unmasking sound and light - but still able to communicate with users).
When performing the initiating gesture, the glove vibrates weakly, on the second gesture it vibrates more strongly. They explained to the users: this is because you feel how a lump of force fills your palm. A correctly executed mark was indicated by a flash of the corresponding color, everyone in the game knew how to react to this.
All the game technicians — the monsters with which the witchmen fought — knew how to react to light gloves. And I had to face, for example, with such a manticore on jolly jumpers.
Photo by Lilia Barladian
The wicked swamp creature the witch girl let in close, in order to surely cast aside Aard.
Photo by Lilia Barladian
Most of the magic with a glove was supposed to occur at night, but just in case it used regular output LEDs (brighter than RGB). In addition, on the output LEDs there is no risk of mixing colors (those who worked with RGB noticed that in certain angles it is clear that yellow light is a combination of red and green, and so on).
The witch cannot make a gesture. Under the reinforced adhesive tape, you can see the battery from a mobile phone, above - a vibration motor.
Photo of Catherine Grigorova
Lyrical digression
Well, when the system has one entry point, after which it begins to recognize something. Ok google And what to do when there are five such points? Users regularly felt a slight vibration, the power raging in them.
Witcher in a glove. On the chest, the medallion is simple electronics, a color indication of drunk potions.
Photo by Elena Mauritius
Then we came to the forest. There came out a few problems. First, no one wore top gloves — leather — they were uncomfortable in them. So you can see the uniform cyberpunk in the photos, but what is much more unpleasant is that the wires died from this. Several vibration motors died, they were soldered right on the range. Perhaps the main problem of the glove was the year of its creation. We didn’t have a 3D printer, it didn’t pack anything, everything kept on hot melt.
Witcher.
Photo by Elena Mauritius
However, the game went without global jambs.
What's next?
Returning to Moscow, washing the gloves in the washing machine (with the controller and the battery disconnected, of course), having come to our senses and considered losses, we thought about modernization.
It would be possible to use not an IDE cable (it is not intended for wearable electronics at all and was used from bitter poverty), but a telephone “redhead cable”. And letting him on the drawstring in a glove, so the problem with washing would not have gotten up at all. And connect the IR LED to combine the glove with the laser tag. AND…
And then came the sale of myobrazlet . We played with him and realized that we would not further develop the glove. Unpromising. Reading the signal from the muscles is much more convenient than the signal from the fingers. (However, for role-playing use myobraslt is not suitable because of the price). Gloves went for another game in Primorye and constantly toured exhibitions.
We believe that the experience was useful, although sometimes painful - it came in handy when creating a bracelet for recognizing a dynamic gesture, this is our fresh project. Mindful of how we worn out with the sensors, soldered to the wires and distributed on the fabric, in the bracelet there were two soldered to the usual reliable charge. The glove worked in statics with a large amount of input data (18 numbers from 6 sensors), the position of the bracelet is described by 2 numbers (using the Mahoney algorithm, we convert 9 values ​​obtained from the sensors).
However, there are those who are interested in experimenting with a glove. In 2015, the glove was shown to the participants of the design school, and some considered it a challenge. Daniil Velovaty (then a schoolboy, now a student of MIPT) put together his own version. Unlike us, he realized the data transfer from the device and surprised visitors to the exhibitions that he remotely turned on the kettle with a sign. Daniel's sign language is also his own.
Source: https://habr.com/ru/post/403757/
All Articles