Navigation in the store: through augmented reality to the desired shelf

Recently, we launched a pilot project: in 10 of our Moscow stores, you can quickly orient yourself in the sales rooms and immediately find shelves with the right product groups using the small augmented reality application M.Go. As far as we know, this is the first such project in the world retail practice. I want to tell in more detail about this application, how it is arranged and what this project can grow into if buyers like it.

About 8-10 years ago, various augmented reality projects appeared on smartphones for the first time. They allowed to find a certain flat marker similar to a QR code on an image, static or video. The application could recognize its size and position in space. Since then, this technology has been actively developed. The loudest example of a product with an augmented reality is the game Pokemon GO, which blew up all the charts several years ago. There are also solutions on the market today that greatly facilitate the development of AR products to third-party developers.

How does M.Go work

When you start the application automatically determines your location and offers to choose the store in which you are. After that, M.Go will ask you to calibrate on the marker. They are pasted on the floor in all 10 stores that today support the application (two markers in each, except for a large two-story store on Sadovaya-Spasskaya - in it you will find already three markers):
')

It is necessary to point the camera of the smartphone on this sticker on the floor, the application recognizes it and “understands” exactly where you stand.


And after a couple of seconds, a list of categories of goods and services available in this store will appear on the screen.


Click on the desired icon, and two pointers will appear on the screen. One “hangs” in the distance above the place where you need to go (it can disappear from the screen if the camera is pointing the other way), and the second pointer is always at the bottom of the screen - this is an arrow showing the direction, like a navigator.






Why are there several markers in each store? The fact is that the application works correctly while it is active. As soon as you minimize it or another application becomes active (for example, someone calls you), navigation inside the store will be interrupted. And to use M.Go again, you need to be calibrated. And so that customers do not have to return to the beginning every time, we stuck on an additional marker in stores.

How does M.Go

When we started to create an application, we relied on our vision and common sense. After all, there were no other similar decisions in the field of retail. There were only successful examples of using technology stack to create applications with augmented reality. We had a few closed pilots. They tested the work of technology AR and different interface options, which is especially important for such applications.

The first option was with the arrow showing the direction. In the second version there was a flying bot that moved along a predetermined route. We chose the option with the arrow, so you can come to the right department of the store, choose another category of goods and go further. And in the version with the bot, it was necessary to return to the calibration marker again, select the next category of goods and only after that go along the new route. Inconvenient. In addition, our stores and so arranged quite simply, it is not a labyrinth of grocery hypermarkets, in which you get tired to get to the exit.

Today, the market has two libraries ARKit and ARCore, created in the depths of Apple and Google. Libraries can recognize not only markers, but also the space in which the user's device is located. Each frame from the camera is translated into black and white mode with high contrast, so that only the contour lines remain on the image. The system finds the key points (the intersection of the contours in the image) and turns them into a “cloud of points”. These are spatial markers.

This technology formed the basis of our application.

On top of the ARCore and ARKit engines, we made a special architecture for working specifically with shop premises. They have their own specifics - the product lines are fairly uniform and not always visually unique. The longer a customer goes through the store, the more positioning error accumulates. We adjust it in different ways. One of the solutions is additional markers that allow recalibration without returning to the beginning.

By the way, we first made the calibration markers red and white, in company colors. They were very beautiful, but very quickly got dirty. I had to choose a different combination of colors, after which the markers even became more reliably recognized by the application.

According to the processor load, M.Go is comparable to any instant messenger. The main consumer of energy is the camera. M.Go can work on most iOS and Android devices, starting with Android 7.0 and iOS 11. This restriction is imposed on the SDK by Google and Apple themselves.

Why does our AR-application "know" only 10 stores? The implementation of support for each store requires a lot of work:

• create a digital planogram (store map) indicating the location of all categories of goods,
• agree on where the markers will be placed
• test how accurately the application is “oriented”.
  

But the project is still pilot, we want to understand whether people need such functionality.

What's next?

If the pilot is successful, we will embed AR functionality in our main application and add new features. There are many options. For example, to simplify the change in the placement of goods in the store. So that the administrator in the tablet simply points to the store layout, where certain categories now lie. And this data would be synchronized with the application installed on users.

You can create a complete universal system for store managers to:

• divide the goods into subcategories in more detail,
• indicate the balance of goods
• add current promotions.

In the client application, we plan to add the seller’s call button to a specific zone, as well as introduce gamification elements: catch the discounts flying through the hall or look for virtual bonuses behind the fridge. In addition, the solution will allow you to find products that are not on display in the sales area, with a detailed description and a 3D model.