How we distinguished a trolley from a foreman - video analytics for the hypermarket checkout zone (and a continuation about the cat terminator)
In one construction hypermarket there are 18 cash desks, and we must be able to do so that the cashiers open them in time so that the queue is not more than 4 people. Well, so that the extra cash is not idle open. This recognition of people (counting buyers) with video, analytics on weather and other factors and prediction of the flow. Plus a lot of other funny statistics.
An example of a queue in front of the cash register is a picture cropped and zamylennaya at the request of security guards, in fact we see a queue longer than in the photo.
In retail, the first question was how to distinguish the foreman from the cart. And it was not at all funny.
')
Yes, and those who were worried about the terminator cat from the previous post - he was caught. Details at the end.
Before that, we were doing something similar for commuter train ticket offices and the train itself. In the case of cash registers, the queue length was perfectly recognized, and cashiers began to work much more optimally. But on the platforms, it turned out to be cheaper to record hares not with cameras, but just to put the phone in place.
At the entrance we have data from the cameras, looking vertically down at the box office. The image from them is recognized by Synesis CasRetail software, and we already have not a video stream, but data on what time what object came and what time it left the zone of action.
Finally, we have a complete complete story analytics that you can tell.
Here is an example of the log (did not decipher, and so it should be clear):
We could not take data from cash registers on checks - they are directly connected to Europe via VPN, and, alas, there are two personal laws that need to be overcome. In general, if we donât want to wait a year (and we donât), this data is not in practice.
The software processes the 640x480 image and, by the way, for the task of counting customers, more is not needed. The system copes easily ... Yes, downsample almost 3 times and this increases the speed of image processing. It does not need a lot of megapixels, as a person can occupy only 30-40 pixels in a frame.
The problem of determining the number of people from the log was not solved at all, and this is why:
We came to the conclusion that right at the checkout, such a task is not solved quickly, without digging into a low level of recognition, in any way. Industrialized, of course, you can profile carts and other objects, but we remembered Uncle Dijkstra with his history with railway cars and decided to watch more.
After a few days of observing the habits of the carts, it turned out that they cling to people. And the group of people who made one check almost simultaneously leaves the cashier area. That is, the task was reduced to counting clusters of objects that go beyond the virtual cashier line (in the picture - tripwire_11) - and this is solved very simply. By the way, the line is needed in order to correctly understand when a person left the cash register. because if he left the zone, he could simply leave the queue back into the hall (for example, he was freaked out, did not wait).
That is, the system gives real-time statistics âtwo objects â one customerâ and from this it shows the queue length. And with a delay of 1 minute, it validates itself and specifies how many buyers were in fact based on how they behaved at the cash register.
It was super cool. We solved almost all the problems with such a system with an error of 2-5% (depending on the number of carts, cleaners, guards, and other deviations from the statistical average).
So we:
Now the system is already six months old, and it works with a good level of scheduling, but then we did not have historical data. Therefore, to begin with, we calculated loading the cash registers in a couple of weeks. The conclusions were paradoxical:
Iâll say a special thing about the latter: it turns out that people do not know how to correctly distribute at the cash desk. And it is quite possible that some cash registers are idle and some are loaded. Skillful distribution allows you to save a lot on the fact, and the store learned it.
The results of the analysis of the dynamics of the maximum and average queue lengths by service areas on one of the dates
Then we did for the operational manager dashboard, where you can see the data at each checkout every half hour with average and maximum load. Two penalty features - few people and many people. It turns out the accumulated statistics: total simple cashiers, overloads, and so on.
The minimum change of cashier from the outstaff (external company providing cashiers) is 4 hours. The sooner the cashier is ordered, the cheaper. Therefore, further, the task comes down, firstly, to forecasting (you need to know how many cashiers and when necessary, the algorithm, you need to say, is not simple - for those in the subject - it is a blending of 6 models from classic ARIMA to RandomForest (for kaggle lovers - and without xgBoost, you can accurately predict. And secondly, to optimize based on the forecast data, namely, to minimize the total cost of staff, while observing the required level of service and a lot of technical constraints (for example, a cashier's outtaff cannot be pulled less than half a day) .
A month later, we solved this problem without taking seasonal peaks into account:
All this together is simply arranged as modifiers (each profile is a graph, that is, a matrix), and a superposition of vectors every half hour gives the predicted load. And then it is imposed a limit on the duration of the shift and the permissible error - and a schedule is drawn up.
The system can learn, and verify itself according to the plan-fact scheme. Plus, you can fasten email-notifications, collect statistics.
In general, the system works. Marketers are happy. To her now the last customer from the story with the cat is also looking closely. There are, among other things, interest among their European colleagues for a number of objects, we count now and for them.
And these are the results of our test or âcomicâ service called âtraffic jams in the canteenâ. It also works fine for a couple of years. In order to use it, just go to the corporate social network, where in real time you can see the current workload of our office canteen, as well as the forecast of the time needed to defend the queue at the cashier and dine, depending on when to approach. And what is most interesting is that quite recently such a system wanted to introduce two external customers. One of them, for example, as it turned out, had been struggling unsuccessfully for 2 years with huge lines in the dining room.
I recall in the last post about facial recognition at the entrance to the object was a cat, causing a loss of the store at 50-100 thousand rubles per month. This tireless hunter was marked on video surveillance with a shadow jumping from above to the fattest bags of food and revealing them in search of warm juicy entrails. And he was charged a lot of expensive alcohol.
The cat left the chase for 4 years. Even when they gave him (alive) 5 thousand rubles for a reward, they could not catch him for several months. But after the last post the animal has comprehended habraeffekt: employees were organized and tracked down a wild animal. The cat caught now lives in the military unit nearest to the store (as a talisman) and gets used to people. Here is his sly face:
An example of a queue in front of the cash register is a picture cropped and zamylennaya at the request of security guards, in fact we see a queue longer than in the photo.
In retail, the first question was how to distinguish the foreman from the cart. And it was not at all funny.
')
Yes, and those who were worried about the terminator cat from the previous post - he was caught. Details at the end.
Before that, we were doing something similar for commuter train ticket offices and the train itself. In the case of cash registers, the queue length was perfectly recognized, and cashiers began to work much more optimally. But on the platforms, it turned out to be cheaper to record hares not with cameras, but just to put the phone in place.
Customer clusters
At the entrance we have data from the cameras, looking vertically down at the box office. The image from them is recognized by Synesis CasRetail software, and we already have not a video stream, but data on what time what object came and what time it left the zone of action.
Finally, we have a complete complete story analytics that you can tell.
Here is an example of the log (did not decipher, and so it should be clear):
{ "camera" : "22f961d5-9cfc-4214-8870-65edb79fe373", "start" : "2016-09-11T14:28:03+03:00", "end" : "2016-09-11T14:29:00+03:00", "rules" : [ // { "counter" : 12, "rule" : "36da29cf-7858-4bc6-9b11-55ff129aeaa7", "extendedStatistics" : [ "2015-02-11T14:28:04+03:00", "2015-02-11T14:28:04+03:00", "2015-02-11T14:28:12+03:00"] // (optional) }, // { "counter" : 0, "histogram":[57.61533203125003,0.0, .., 0.0,0.0], // total 256 items "lifetimeHistogram" : [], "lifetimeHistogramStep" : 60, "lostObjectsCounter" : 2, "bornObjectsCounter" : 0, "objectsAtEndInterval" : 12, "rule" : "ea473652-2f7d-49fc-bce0-e6d93e472c17" } ] } We could not take data from cash registers on checks - they are directly connected to Europe via VPN, and, alas, there are two personal laws that need to be overcome. In general, if we donât want to wait a year (and we donât), this data is not in practice.
The software processes the 640x480 image and, by the way, for the task of counting customers, more is not needed. The system copes easily ... Yes, downsample almost 3 times and this increases the speed of image processing. It does not need a lot of megapixels, as a person can occupy only 30-40 pixels in a frame.
The problem of determining the number of people from the log was not solved at all, and this is why:
- The buyer may come in crowds (for example, family). In fact there will be 3-4 people, but really - one check. And the data about the check in a European country, and if you do not want to wait a year ... well, you understand.
- A foreman may come, collect building materials in 3-4 carts and leave. The system will fix the group of objects, but again the buyer will be alone.
- Mother with a stroller and a cart - these are 3 incomprehensible objects.
We came to the conclusion that right at the checkout, such a task is not solved quickly, without digging into a low level of recognition, in any way. Industrialized, of course, you can profile carts and other objects, but we remembered Uncle Dijkstra with his history with railway cars and decided to watch more.
After a few days of observing the habits of the carts, it turned out that they cling to people. And the group of people who made one check almost simultaneously leaves the cashier area. That is, the task was reduced to counting clusters of objects that go beyond the virtual cashier line (in the picture - tripwire_11) - and this is solved very simply. By the way, the line is needed in order to correctly understand when a person left the cash register. because if he left the zone, he could simply leave the queue back into the hall (for example, he was freaked out, did not wait).
That is, the system gives real-time statistics âtwo objects â one customerâ and from this it shows the queue length. And with a delay of 1 minute, it validates itself and specifies how many buyers were in fact based on how they behaved at the cash register.
It was super cool. We solved almost all the problems with such a system with an error of 2-5% (depending on the number of carts, cleaners, guards, and other deviations from the statistical average).
Recognition Surprises
So we:
- We know the incoming flow of people for every day and we know it in real time.
- We know the length of the queue at each checkout.
- We are able to make real-time notifications about the need to open a cashier (somewhere there is a queue longer than 4-5 people).
- We are able to close cash registers if they are empty.
- We begin to collect data for analytics to predict the schedule of cash.
Now the system is already six months old, and it works with a good level of scheduling, but then we did not have historical data. Therefore, to begin with, we calculated loading the cash registers in a couple of weeks. The conclusions were paradoxical:
- Yes, there are times when more cashiers are needed.
- Yes, there are times when the store is losing due to idle cash registers (hundreds of thousands of rubles per month).
- The most fun - when the peaks at the box office 10-18 cash desk 1-3 may not be loaded.
Iâll say a special thing about the latter: it turns out that people do not know how to correctly distribute at the cash desk. And it is quite possible that some cash registers are idle and some are loaded. Skillful distribution allows you to save a lot on the fact, and the store learned it.
The results of the analysis of the dynamics of the maximum and average queue lengths by service areas on one of the dates
Project progress
Then we did for the operational manager dashboard, where you can see the data at each checkout every half hour with average and maximum load. Two penalty features - few people and many people. It turns out the accumulated statistics: total simple cashiers, overloads, and so on.
The minimum change of cashier from the outstaff (external company providing cashiers) is 4 hours. The sooner the cashier is ordered, the cheaper. Therefore, further, the task comes down, firstly, to forecasting (you need to know how many cashiers and when necessary, the algorithm, you need to say, is not simple - for those in the subject - it is a blending of 6 models from classic ARIMA to RandomForest (for kaggle lovers - and without xgBoost, you can accurately predict. And secondly, to optimize based on the forecast data, namely, to minimize the total cost of staff, while observing the required level of service and a lot of technical constraints (for example, a cashier's outtaff cannot be pulled less than half a day) .
A month later, we solved this problem without taking seasonal peaks into account:
- Historical data by day give accurate downloads by hour.
- There are profiles for each day of the week.
- There are special profiles for the situations of âprecipitationâ, âholidayâ, âday before the weekendâ, âday before 3 days offâ - these are very important situations for analytics. Rain, for example, gives minus 18 percent to customers, and the day before 3 days off - plus 14%.
- There is weather data.
- There are correlates from other projects (famous schedules of colleagues from around the world) - there are just 3-4 measurements per day, but they are very useful sometimes when there is no data (for example, they will be useful for the New Year).
- We know the conversion, that is, with what probability coming into the store will become a buyer. This is just a multiplier to the output stream.
- For people entering through the ticket office, we also know approximate coefficients - they had to be counted by hand, but they introduce a very small error.
All this together is simply arranged as modifiers (each profile is a graph, that is, a matrix), and a superposition of vectors every half hour gives the predicted load. And then it is imposed a limit on the duration of the shift and the permissible error - and a schedule is drawn up.
The system can learn, and verify itself according to the plan-fact scheme. Plus, you can fasten email-notifications, collect statistics.
Farther
In general, the system works. Marketers are happy. To her now the last customer from the story with the cat is also looking closely. There are, among other things, interest among their European colleagues for a number of objects, we count now and for them.
And these are the results of our test or âcomicâ service called âtraffic jams in the canteenâ. It also works fine for a couple of years. In order to use it, just go to the corporate social network, where in real time you can see the current workload of our office canteen, as well as the forecast of the time needed to defend the queue at the cashier and dine, depending on when to approach. And what is most interesting is that quite recently such a system wanted to introduce two external customers. One of them, for example, as it turned out, had been struggling unsuccessfully for 2 years with huge lines in the dining room.
About a cat
I recall in the last post about facial recognition at the entrance to the object was a cat, causing a loss of the store at 50-100 thousand rubles per month. This tireless hunter was marked on video surveillance with a shadow jumping from above to the fattest bags of food and revealing them in search of warm juicy entrails. And he was charged a lot of expensive alcohol.
The cat left the chase for 4 years. Even when they gave him (alive) 5 thousand rubles for a reward, they could not catch him for several months. But after the last post the animal has comprehended habraeffekt: employees were organized and tracked down a wild animal. The cat caught now lives in the military unit nearest to the store (as a talisman) and gets used to people. Here is his sly face:
More projects
- Recognition of employees at the turnstile
- Number Recognition (2012)
- Voice fingerprints
- Paranoid Voice Recognition
- Turnstiles with 3D infrared cameras for quick face recognition
- My mail is Brahew@croc.ru.
Source: https://habr.com/ru/post/314318/
All Articles