What awaits us in Microsoft Kinect 2.0?
The first version of Kinect, a touchscreen gaming controller from Microsoft, ranked second in the ranking of the 10 Most Innovative Technical Products of 2011, according to Popular Mechanics magazine. By February 2013, the number of devices sold reached 24 million. At the same time, the first 8 million were sold within 60 days from the moment the device went on sale, making Kinect the “fastest selling user device” in the Guinness Book of Records.
But time passes, and the release of the new version of Kinect is not far off. I liked the latest TechCrunch article (in the original “ How Microsoft built the cameras in the upcoming Kinect ”) about improvements to the new Kinect. The text was written by a man who specifically visited the Microsoft campus in Mountain View to get familiar with the new controller, and also spoke with the engineers who designed it.
Under the cat 5 arguments, why the new Kinect has every chance to repeat the success of the first version.
')
Figure 1. View modes of the new Kinect.
Thought 1: The new Kinect was steeper than expected, well ahead of its predecessor in technical terms. The increased viewing angle, the greater number of pixels in the sensor, and the improved resolution allow the child to recognize the movement of the wrist at a distance of 3.5 meters.
Thought 2: Several new viewing modes have been added to the second version. Despite the fact that these modes are not available to ordinary users, they can be extremely useful for application developers to improve the accuracy of tracking movements and reduce errors.
For example, Infrared View Mode and new human body modeling tools can be used to track muscle movements and the relative orientation of body parts.
And the Deep Image Mode , acting as a radar, where each of the 220 thousand sensor pixels records data independently, allows you to create, for example, a surprisingly accurate and detailed display of the room.
Thought 3: The camera settings that are invariant to light (which remain unchanged with changes in light) are also added to the new Kinect. With these settings, Kinect gives the same result regardless of the characteristics of the lighting room. In practice, this means that you can use Kinect in the dark or in a room with light noises. For example, two spotlights aimed directly at the sensor will not affect the performance of the device. The author of the article immediately decided to test it, and according to his conclusion, this mechanism really works, as the manufacturer promises.
Figure 2. The new Kinect settings, invariant to the light, allow it to be used in the dark.
Thus, when processing data from Kinect, developers do not have to worry about how the user is lit, and the data will not be distorted due to an unexpected change in the lighting in the room, for example, turning on the highlight.
The number of recognizable joints of the human body is also significantly increased, which can be used to more accurately track the movements of users' hands.
Thought 4: The minimum size of an object that could recognize the first-generation Kinect was 7.5 cm. The new Kinect simultaneously recognizes objects as small as 2.5 cm in size by 60%. The number of people simultaneously tracked in the room is also increased to 6 compared to 2 in the previous version.
Thought 5: The first Kinect became the fastest selling user device in history. Its existence has helped prolong interest in the Xbox 360, even when the console is morally obsolete. Microsoft releases the new Kinect along with the new Xbox One. Both devices will be available in the US from mid-November 2013 and will compete for users with the upcoming PlayStation 4 console from Sony.
Microsoft wanted to create a camera operating on the basis of “flight time,” that is, the time required for the light to travel from its source to the object and back. Given that this is happening fairly quickly, the new Kinect needs to process giant data streams in real time, which in itself is a complex engineering task. Representatives of the Israeli branch of Microsoft and the central office in Silicon Valley, have teamed up to implement the academic idea of ​​“flight time” into a commercial product.
Figure 3. The idea of ​​“flight time” formed the basis for the operation of the camera of the new Kinect.
By combining their best minds, Microsoft managed to solve the problem of data collection, but other problems arose related to data overload due to large volumes and blurring of object contours.
In short, two conditions were required at the same time:
Initially, the development team was far from the goal. The guys have developed algorithms that optimize the load on the processor and perform image processing so that distant objects do not merge and do not erode while moving. As engineers say, the implementation of these software tasks was made possible by pre-calibration of the camera. If the hardware is not pre-configured, the algorithms would learn from imperfect or incorrect data. It is advisable to train the algorithms on the final data, and not on data with “noise” or test data.
The equipment of the new Kinect is multi-component, that is, there is an aggregation node that collects sensor data and forms separate streams for each component of the signal (Microsoft does not go into details, but there is an assumption that this is a separate chip). Microsoft also declined to talk about where the process of “cleaning” data is performed. There is an assumption that to reduce the input, this process, at least partially, occurs on the console itself.
The end result is a multi-component data stream that can be used by application developers based on Xbox or PC.
In Russia, the new console will not be available until next year.
But time passes, and the release of the new version of Kinect is not far off. I liked the latest TechCrunch article (in the original “ How Microsoft built the cameras in the upcoming Kinect ”) about improvements to the new Kinect. The text was written by a man who specifically visited the Microsoft campus in Mountain View to get familiar with the new controller, and also spoke with the engineers who designed it.
Under the cat 5 arguments, why the new Kinect has every chance to repeat the success of the first version.
')
Figure 1. View modes of the new Kinect.
Thought 1: The new Kinect was steeper than expected, well ahead of its predecessor in technical terms. The increased viewing angle, the greater number of pixels in the sensor, and the improved resolution allow the child to recognize the movement of the wrist at a distance of 3.5 meters.
Thought 2: Several new viewing modes have been added to the second version. Despite the fact that these modes are not available to ordinary users, they can be extremely useful for application developers to improve the accuracy of tracking movements and reduce errors.
For example, Infrared View Mode and new human body modeling tools can be used to track muscle movements and the relative orientation of body parts.
And the Deep Image Mode , acting as a radar, where each of the 220 thousand sensor pixels records data independently, allows you to create, for example, a surprisingly accurate and detailed display of the room.
Thought 3: The camera settings that are invariant to light (which remain unchanged with changes in light) are also added to the new Kinect. With these settings, Kinect gives the same result regardless of the characteristics of the lighting room. In practice, this means that you can use Kinect in the dark or in a room with light noises. For example, two spotlights aimed directly at the sensor will not affect the performance of the device. The author of the article immediately decided to test it, and according to his conclusion, this mechanism really works, as the manufacturer promises.
Figure 2. The new Kinect settings, invariant to the light, allow it to be used in the dark.
Thus, when processing data from Kinect, developers do not have to worry about how the user is lit, and the data will not be distorted due to an unexpected change in the lighting in the room, for example, turning on the highlight.
The number of recognizable joints of the human body is also significantly increased, which can be used to more accurately track the movements of users' hands.
Thought 4: The minimum size of an object that could recognize the first-generation Kinect was 7.5 cm. The new Kinect simultaneously recognizes objects as small as 2.5 cm in size by 60%. The number of people simultaneously tracked in the room is also increased to 6 compared to 2 in the previous version.
Thought 5: The first Kinect became the fastest selling user device in history. Its existence has helped prolong interest in the Xbox 360, even when the console is morally obsolete. Microsoft releases the new Kinect along with the new Xbox One. Both devices will be available in the US from mid-November 2013 and will compete for users with the upcoming PlayStation 4 console from Sony.
Problems encountered
Microsoft wanted to create a camera operating on the basis of “flight time,” that is, the time required for the light to travel from its source to the object and back. Given that this is happening fairly quickly, the new Kinect needs to process giant data streams in real time, which in itself is a complex engineering task. Representatives of the Israeli branch of Microsoft and the central office in Silicon Valley, have teamed up to implement the academic idea of ​​“flight time” into a commercial product.
Figure 3. The idea of ​​“flight time” formed the basis for the operation of the camera of the new Kinect.
By combining their best minds, Microsoft managed to solve the problem of data collection, but other problems arose related to data overload due to large volumes and blurring of object contours.
In short, two conditions were required at the same time:
- Processing about 6.5 million pixels every second;
- Low load on the Xbox One to maintain its high performance;
Initially, the development team was far from the goal. The guys have developed algorithms that optimize the load on the processor and perform image processing so that distant objects do not merge and do not erode while moving. As engineers say, the implementation of these software tasks was made possible by pre-calibration of the camera. If the hardware is not pre-configured, the algorithms would learn from imperfect or incorrect data. It is advisable to train the algorithms on the final data, and not on data with “noise” or test data.
The equipment of the new Kinect is multi-component, that is, there is an aggregation node that collects sensor data and forms separate streams for each component of the signal (Microsoft does not go into details, but there is an assumption that this is a separate chip). Microsoft also declined to talk about where the process of “cleaning” data is performed. There is an assumption that to reduce the input, this process, at least partially, occurs on the console itself.
The end result is a multi-component data stream that can be used by application developers based on Xbox or PC.
In Russia, the new console will not be available until next year.
Useful articles
- Original article (in English)
Source: https://habr.com/ru/post/196636/
All Articles