The main thing is to sit shoe. How 3D-technology helps to choose shoes by foot

Can I use Intel RealSense technology for ... fitting shoes? “Why not”, they decided in the company Volumental, “because the leg is the same three-dimensional object as the others, and therefore it can be 3D-scanned”. And, imagine, applied, while achieving excellent accuracy - up to 2 mm! This curious example shows how wide the possibilities of using RealSense are and how many untested possibilities await their time for their researchers.

If you ever bought children's shoes, then you know that finding the right size shoes is very difficult. The child grows, and the purchased shoes quickly become small. In addition, it is not so easy to persuade a child to just sit on the spot to try on several pairs of shoes. After that, you still need to understand how accurately and truthfully the child answered, whether the new sneakers fit him, especially if they depict a character from a favorite cartoon.
But what if you could quickly and accurately measure your child's feet before trying shoes on, so that you can choose among pairs of exactly the right size?

Volumental and Intel RealSense technology

Volumental plans to implement this approach using Intel RealSense technology. Three-dimensional scanning to accurately determine the size of the body will be used not only to measure the feet and buy shoes. Caroline Walerud, head of Volumental, says: "We believe that all clothes and shoes should be made to measure, to the exact dimensions of each person." Volumental experts believe that three-dimensional scanning of body parts will allow wider use of the possibilities of individual adjustment of various products, not only shoes and clothes, but also medical devices. And now, thanks to Intel RealSense technology, these opportunities are available for the mainstream market.
')
Volumental is a young company working in the field of three-dimensional scanning of the human body. The company was formed on the basis of a computer vision research project organized at the Royal Institute of Technology, Stockholm. When Intel announced the release of Intel RealSenseTM hardware and software, Volumental immediately turned its attention to this technology and decided to use it as the basis for 3D modeling and visualization software. However, for the transition from a common idea to a developed project, it was necessary to solve a lot of problems.

Intel released the Intel RealSense F200 user-friendly camera, which is embedded in Ultrabook, transformers and all-in-one computers, while Volumental wrote code to measure and create three-dimensional foot models, first for children (since fitting children's shoes is more difficult), and then adults. Volumental specialists conducted testing with users in order to understand how customers will “naturally” try to measure their feet using a tablet or laptop. Users intuitively wanted to drive the device around their feet to scan (Fig. 1) instead of just placing the device on the floor.


Figure 1. Testing with users


Figure 2. Stepladder mode and vertical position with the screen turned out

After watching the training video in which the device was on the floor, users agreed that it was more convenient. It turned out that it was most convenient when the laptop-transformer was installed in the “stepladder” mode or simply with a vertically raised screen, turned outward from the keyboard (Fig. 2). It is important that users can scan both bare feet and feet in their socks, since it was necessary to create a product, when working with which people would not feel awkward, everything had to be extremely simple and intuitive. Adding instructions on how to properly install the device for each of the three poses made it possible to get good results at home.

In retail stores, where in shoe departments you need to give buyers the opportunity to self-service, but avoid theft of devices, they suggested using stationary kiosks to scan their feet, as shown in fig. 3. Schemes on the bottom platform show the buyer how and where to stand.


Figure 3. Kiosk for the selection of shoe size

With the help of Intel RealSense depth and color cameras, Volumental developers created an application that takes three-dimensional images of bare feet or toes in socks. The application scans the legs from three angles, recognizes the legs and their direction (Fig. 4).


Figure 4. Three snapshots


Figure 5. Results on screen

The Intel RealSense 3D camera provides both a color image and the corresponding depth information for each pixel. The Volumental software analyzes the data of these images and provides data on the length, width and circumference of the foot, creating a useful three-dimensional model of the user's feet. The resulting model contains thousands of measurements, it can be used to recommend the right size of the finished shoe, and to order shoes to measure.

In fig. 5 shows an example of a three-dimensional model of the buyer's feet. Shoe stores can provide each customer with catalogs of shoes only the right size.

How it works

With the help of pre-beta hardware and Intel RealSense SDK 2014, companies from Volumental, Intel, and PLAE (children's footwear manufacturer in San Francisco) showed a demonstration of the capabilities of the system for measuring children's feet at the 2014 Intel Developer Forum in San Francisco. In early 2015, Volumental and Intel showed their latest development - a free-standing stationary kiosk with a computer and an Intel RealSense F200 foot-level camera. Such kiosks are planned to be installed in the shoe departments of some retail stores in 2015.

The program ShoeSizer combines computer vision and machine learning technologies to create accurate three-dimensional models of legs in seconds. The system works even if the legs move (not too much), because the algorithms of tracking the three-dimensional trajectory are used and an accurate result is achieved.

Software algorithms are based on machine learning; large database of legs was used to train the system in order to achieve the creation of accurate models even with insufficiently clear images or in the absence of data. Algorithms and processing techniques are the intellectual property of the company Volumental, but Intel experts have helped to improve the performance of the program through multi-threaded processing using OpenMP *. Finished models contain much more information about foot size than just the right shoe size.

Other tasks

Touch screen systems require the use of hands, which is inconvenient if the system should be at foot level. Nevertheless, the developers appreciated and exploited other features of Intel RealSense technology, including the ability to use voice commands. Kiosk accepts user voice commands to start scanning.

It was also necessary to take into account that there is a huge variety of socks of different types, different colors, with different patterns and texture of the fabric. It took very thorough testing to determine that the system works best with white cotton socks, but you can also wear socks of other colors or with patterns. Black socks with a dense texture are the hardest to recognize. The code of the Volumental program had to be modified to support a wider range of images. In addition, even the color of the floor and its texture is important, since the floor plane is taken into account when measuring. In this regard, one of the advantages of a stationary measurement kiosk is that the buyer does not put his foot on the floor, but on the kiosk site, the color and texture of which are known in advance to the developers of the system.

Not only shoes

Caroline Wailrad, Head of Volumental, states: “This is only the beginning for Volumental. We are going to manufacture products based on this model and develop widespread commercial use. We are pleased to take the opportunity to be the first to launch such solutions on the market. We believe that easy recording of foot size data is the first step towards a completely new approach to shoe manufacturing, which will more closely match the individual size of customers. In the future, the principles of production will change dramatically: the individual wishes of customers will be taken into account to a much greater extent than they are now. ”

The technology was originally developed for retail shoe stores, but it also has certain advantages in areas such as prosthetics and the creation of medical devices. With this technology, leading footwear companies will be able to offer their customers shoes that perfectly fit in size. Currently, Volumental is working on several projects with prosthetics partners and custom-made footwear in addition to large retail shoe stores. With Intel's new cameras, you'll soon see 3D scanners for shoppers in all stores.