How eBay did a barcode scanner on WebAssembly
Since its announcement, the WebAssembly technology has immediately attracted the attention of front-end developers. The web community enthusiastically accepted the idea of ​​running code written in languages ​​other than JavaScript in the browser. Most importantly, WebAssembly guarantees a much higher speed than JavaScript.
Our engineers closely followed the development of the standard. As soon as WebAssembly 1.0 support was implemented in all major browsers, the developers immediately wanted to try it out.
But there was a problem. Although many applications benefit from WebAssembly, the scope of e-commerce technology is still primitive. We could not immediately find the correct version of its use. There were a few suggestions, but in all versions of JavaScript fit better. When we evaluate new technologies in eBay, the first question is: “What is the potential benefit for our customers?” If there is no clarity here, we don’t go on to the next step. It is very easy to get carried away with new fashionable technology, even if it doesn’t matter to customers and only complicates the existing workflow. User experience is always more important than developer experience. But with WebAssembly different. This technology has a huge potential, we just could not find the right use case. However, in the end still found.
In native eBay applications on iOS and Android, there is a UPC barcode scanning feature for automatic form entry. It works only in applications and requires intensive image processing on the device in order to recognize bar code digits in the stream of images from the camera. The resulting code is then sent to the server service, which, in turn, fills out the form. This means that the image processing logic on the device must be very efficient. For native applications, we compiled our own C ++ library into native code for iOS and Android. It recognizes bar codes exceptionally well. We are gradually switching to native APIs in iOS and Android, but our C ++ library is still reliable.
')
Barcode scanner - an intuitive feature for sellers, it greatly simplifies filling out the form. Unfortunately, this feature did not work on the mobile version of the site, and the sellers had to manually enter the UPC, which is inconvenient.
We used to look for a barcode scanning option on the web. Two years ago, they even released a prototype based on the BarcodeReader open JavaScript library. The problem was that it worked well only in 20% of cases. The remaining 80% of the time the scanner worked extremely slowly or did not work at all. In most cases it was a time out. Quite expectedly: JavaScript can be compared in speed with native code only if it is on a “hot path”, i.e. it is highly optimized by JIT compilers. The trick is that JavaScript engines use multiple heuristics to determine if a path is “hot”, without guaranteeing results. This inconsistency, obviously, led to the frustration of users, and we had to disable this feature. But now everything is different. With the rapid development of the web platform, the question arose: "Is it possible to implement a reliable barcode scanner on the web?"
One option is to wait for the Shape Detection API to exit with built-in image detection, including barcodes . But these interfaces are still at a very early development stage and far from cross-browser compatibility. And even in this case, work is not guaranteed on all platforms. Therefore, we will have to consider other options.
This is where WebAssembly comes into play. If the barcode scanner is implemented on WebAssembly, then it will be guaranteed to work. Strong typing and the structure of the WebAssembly byte-code make it possible to always keep the “hot path” of execution. In addition, we already have a C ++ library for native applications. C ++ libraries are ideal candidates for compiling to WebAssembly. We thought the problem was solved. It turned out not quite.
The working prototype architecture for a barcode scanner on WebAssembly was fairly simple.
WebAssembly Workflow
The first step in any WebAssembly project is to define a clear compilation pipeline. Emscripten has become the de facto standard for compiling WebAssembly, but it is important to have a consistent environment that produces a deterministic result. Our frontend is based on Node.js, so you need to find a solution that is compatible with the npm workflow. Fortunately, at about that time Surma Das published an article called “Emscripten and npm” . The Docker- based approach to compiling WebAssembly makes sense because it eliminates a lot of overhead. As recommended in the article, we took the Docker image of Emscripten from trzeci . That compilation in WebAssembly became possible, the native C ++ library had to be corrected a little. Basically, we acted at random by trial and error. In the end, we managed to compile it, as well as set up a neat WebAssembly workflow within the existing assembly pipeline.
Scanner performance is measured by the number of frames processed by the Wasm API per second. The Wasm API takes a frame from the camera's video stream, performs calculations and returns a response. This is done on an ongoing basis until a bar code is detected. Performance is measured in FPS.
Our WebAssembly implementation showed an amazing 50 FPS speed during testing. However, it worked only in 60% of cases, and in the rest it took off on a timeout. Even with such a high FPS, they could not quickly detect the bar code for the remaining 40% of scans, producing a warning message at the end. For comparison, the previous JavaScript implementation usually worked at a speed of 1 FPS. Yes, WebAssembly is much faster (50 times), but for some reason it does not work in almost half the time. It should also be noted that in some situations, JavaScript worked very well and immediately found the barcode. One of the obvious options was to increase the timeout, but this would only increase the frustration of users, and so we do not solve the real problem. Therefore, we abandoned this idea.
At first, we could not understand why the native C ++ library, which worked perfectly in native applications, did not show the same result on the web. After extensive testing and debugging, we found that the recognition speed depends on the focusing angle of the object and the background shadow. But how then does everything work in native applications? The fact is that in native applications, we use the built-in API for autofocusing and allow the user to focus manually by pointing the barcode with his finger. Therefore, native applications always provide the library with high-quality clear images.
Realizing the essence of what is happening, we decided to try another native library: a fairly popular and stable open source ZBar barcode scanner. More importantly, it works well with blurred and grainy images. Why not try? Since we already had the WebAssembly workflow, the compilation and deployment of ZBar to WebAssembly went smoothly. Performance was decent, around 15 FPS, although not as good as our own C ++ library. But the success rate was close to 80% for the same timeout. An obvious improvement compared to our C ++ library, but still not 100%.
The result did not satisfy us yet, but we noticed something unexpected. Where Zbar crashed on time-out, our own C ++ library did the work very quickly. It was a pleasant surprise. It seems that libraries processed images of different quality in different ways. This led us to the idea.
Probably, you already understood. Why not create two threads of workers: one for Zbar and the other for our C ++ library, and not run them in parallel. Whoever won (who first sends a valid barcode) sends the result to the main stream, and both workers stop. We implemented such a scenario and started testing ourselves, trying to imitate as many scripts as possible. This setting showed 95% of successful scans. Much better than previous results, but still not 100%.
One of the strange suggestions was to add the original JavaScipt library to the competition. It will be three streams. We honestly did not think that this would change anything. But such a test did not require any effort, because we standardized the working interface. To our surprise, with three streams, the success rate was really close to 100%. This again was completely unexpected. As mentioned earlier, JavaScript worked very well in some situations. Apparently, he closed the gap. So popular wisdom is right - “JavaScript always wins . ” If no joke, the following illustration provides an overview of the final architecture that we implemented.
Barcode Scanner Web Architecture
The following figure shows the high-level function diagram:
Barcode Scanner Functional Diagram
The resources necessary for the operation of the scanner are preloaded after rendering the main page. This way, the landing page loads quickly and is ready for interaction. WebAssembly resources (wasm files and glue code scripts) and the JavaScript scanner library are preloaded and cached using XMLHttpRequest after loading the main page. It is important here that they are not executed immediately in order to leave the main thread free for the user to interact with the page. Execution occurs only when the user clicks on the barcode icon. If the user clicks on the icon before loading the resources, they will be loaded on demand and immediately executed. The barcode scanner event handler and the worker controller are loaded along with the page, but they are very small.
After rigorous testing and internal use by employees, we launched A / B testing on users. The test group was shown the scanner icon (screenshot below), and the control group was not.
Final product
To assess success, we introduced the metric “Draft Completion Rate”. This is the time between the start of editing a draft and submitting the form. The metric should show how a barcode scanner helps people fill out forms. The test lasted several weeks, and the results were very pleasant. They are completely consistent with our original hypothesis. Draft completion time decreased by 30% for flow with barcode scanner.
A / B test results
We also added profiling to evaluate the effectiveness of all types of scanners. As expected, Zbar made the largest contribution (53% of successful scans), then our C ++ library (34%) and, finally, the JavaScript library from 13%.
The experience of implementing WebAssembly has become very informative for us. Engineers are very pleased with the emergence of new technologies and immediately want to try them out. If the technology is also useful for customers, then this is a double joy. We repeat the idea expressed at the beginning of the article. Technology is developing at a very fast pace. Every day something new appears. But few technologies matter to customers, and WebAssembly is one of them. Our biggest conclusion from this exercise is to say “no” in 99 situations and “yes” in the only case where it is really important for customers.
In the future, we plan to expand the use of a barcode scanner and implement it on the buyers side so that they can scan product codes offline for their search and purchase on eBay. Also consider the option to extend the function using the Shape Detection API and other functions in the browser. But we are glad that we have found the right way to use WebAssembly on eBay and successfully applied the technology in e-commerce.
Special thanks to Surme Das and Lin Clark for the many articles on WebAssembly. They really helped us break the deadlock several times.
Our engineers closely followed the development of the standard. As soon as WebAssembly 1.0 support was implemented in all major browsers, the developers immediately wanted to try it out.
But there was a problem. Although many applications benefit from WebAssembly, the scope of e-commerce technology is still primitive. We could not immediately find the correct version of its use. There were a few suggestions, but in all versions of JavaScript fit better. When we evaluate new technologies in eBay, the first question is: “What is the potential benefit for our customers?” If there is no clarity here, we don’t go on to the next step. It is very easy to get carried away with new fashionable technology, even if it doesn’t matter to customers and only complicates the existing workflow. User experience is always more important than developer experience. But with WebAssembly different. This technology has a huge potential, we just could not find the right use case. However, in the end still found.
Barcode Scanner
In native eBay applications on iOS and Android, there is a UPC barcode scanning feature for automatic form entry. It works only in applications and requires intensive image processing on the device in order to recognize bar code digits in the stream of images from the camera. The resulting code is then sent to the server service, which, in turn, fills out the form. This means that the image processing logic on the device must be very efficient. For native applications, we compiled our own C ++ library into native code for iOS and Android. It recognizes bar codes exceptionally well. We are gradually switching to native APIs in iOS and Android, but our C ++ library is still reliable.
')
Barcode scanner - an intuitive feature for sellers, it greatly simplifies filling out the form. Unfortunately, this feature did not work on the mobile version of the site, and the sellers had to manually enter the UPC, which is inconvenient.
Web Barcode Scanner
We used to look for a barcode scanning option on the web. Two years ago, they even released a prototype based on the BarcodeReader open JavaScript library. The problem was that it worked well only in 20% of cases. The remaining 80% of the time the scanner worked extremely slowly or did not work at all. In most cases it was a time out. Quite expectedly: JavaScript can be compared in speed with native code only if it is on a “hot path”, i.e. it is highly optimized by JIT compilers. The trick is that JavaScript engines use multiple heuristics to determine if a path is “hot”, without guaranteeing results. This inconsistency, obviously, led to the frustration of users, and we had to disable this feature. But now everything is different. With the rapid development of the web platform, the question arose: "Is it possible to implement a reliable barcode scanner on the web?"
One option is to wait for the Shape Detection API to exit with built-in image detection, including barcodes . But these interfaces are still at a very early development stage and far from cross-browser compatibility. And even in this case, work is not guaranteed on all platforms. Therefore, we will have to consider other options.
This is where WebAssembly comes into play. If the barcode scanner is implemented on WebAssembly, then it will be guaranteed to work. Strong typing and the structure of the WebAssembly byte-code make it possible to always keep the “hot path” of execution. In addition, we already have a C ++ library for native applications. C ++ libraries are ideal candidates for compiling to WebAssembly. We thought the problem was solved. It turned out not quite.
Architecture
The working prototype architecture for a barcode scanner on WebAssembly was fairly simple.
- Compile the C ++ library using Emscripten . It will issue the link code and the .wasm file.
- Select a worker thread from the main thread. The JavaScript code for the worker imports the generated JavaScript junction code, which in turn creates the .wasm file.
- The main stream sends a snapshot from the stream from the camera to the worker's stream, and it will call the corresponding WASM API through the linking code. The API response is transmitted to the main thread. The response can be a UPC string (which is transmitted to the backend) or an empty string if no barcode is detected.
- For an empty response, the above step is repeated until a bar code is detected. This cycle runs for the specified time interval in seconds. Once the threshold is reached, we will display the warning message “Invalid product code. Try a different barcode or search by text . ” Either the user did not focus the camera on a real bar code, or the scanner is not effective enough. We track timeout statistics as an indicator of the quality of the scanner.
WebAssembly Workflow
Compilation
The first step in any WebAssembly project is to define a clear compilation pipeline. Emscripten has become the de facto standard for compiling WebAssembly, but it is important to have a consistent environment that produces a deterministic result. Our frontend is based on Node.js, so you need to find a solution that is compatible with the npm workflow. Fortunately, at about that time Surma Das published an article called “Emscripten and npm” . The Docker- based approach to compiling WebAssembly makes sense because it eliminates a lot of overhead. As recommended in the article, we took the Docker image of Emscripten from trzeci . That compilation in WebAssembly became possible, the native C ++ library had to be corrected a little. Basically, we acted at random by trial and error. In the end, we managed to compile it, as well as set up a neat WebAssembly workflow within the existing assembly pipeline.
It works fast, but ...
Scanner performance is measured by the number of frames processed by the Wasm API per second. The Wasm API takes a frame from the camera's video stream, performs calculations and returns a response. This is done on an ongoing basis until a bar code is detected. Performance is measured in FPS.
Our WebAssembly implementation showed an amazing 50 FPS speed during testing. However, it worked only in 60% of cases, and in the rest it took off on a timeout. Even with such a high FPS, they could not quickly detect the bar code for the remaining 40% of scans, producing a warning message at the end. For comparison, the previous JavaScript implementation usually worked at a speed of 1 FPS. Yes, WebAssembly is much faster (50 times), but for some reason it does not work in almost half the time. It should also be noted that in some situations, JavaScript worked very well and immediately found the barcode. One of the obvious options was to increase the timeout, but this would only increase the frustration of users, and so we do not solve the real problem. Therefore, we abandoned this idea.
At first, we could not understand why the native C ++ library, which worked perfectly in native applications, did not show the same result on the web. After extensive testing and debugging, we found that the recognition speed depends on the focusing angle of the object and the background shadow. But how then does everything work in native applications? The fact is that in native applications, we use the built-in API for autofocusing and allow the user to focus manually by pointing the barcode with his finger. Therefore, native applications always provide the library with high-quality clear images.
Realizing the essence of what is happening, we decided to try another native library: a fairly popular and stable open source ZBar barcode scanner. More importantly, it works well with blurred and grainy images. Why not try? Since we already had the WebAssembly workflow, the compilation and deployment of ZBar to WebAssembly went smoothly. Performance was decent, around 15 FPS, although not as good as our own C ++ library. But the success rate was close to 80% for the same timeout. An obvious improvement compared to our C ++ library, but still not 100%.
The result did not satisfy us yet, but we noticed something unexpected. Where Zbar crashed on time-out, our own C ++ library did the work very quickly. It was a pleasant surprise. It seems that libraries processed images of different quality in different ways. This led us to the idea.
Multithreading and speed racing
Probably, you already understood. Why not create two threads of workers: one for Zbar and the other for our C ++ library, and not run them in parallel. Whoever won (who first sends a valid barcode) sends the result to the main stream, and both workers stop. We implemented such a scenario and started testing ourselves, trying to imitate as many scripts as possible. This setting showed 95% of successful scans. Much better than previous results, but still not 100%.
One of the strange suggestions was to add the original JavaScipt library to the competition. It will be three streams. We honestly did not think that this would change anything. But such a test did not require any effort, because we standardized the working interface. To our surprise, with three streams, the success rate was really close to 100%. This again was completely unexpected. As mentioned earlier, JavaScript worked very well in some situations. Apparently, he closed the gap. So popular wisdom is right - “JavaScript always wins . ” If no joke, the following illustration provides an overview of the final architecture that we implemented.
Barcode Scanner Web Architecture
The following figure shows the high-level function diagram:
Barcode Scanner Functional Diagram
Resource Download Note
The resources necessary for the operation of the scanner are preloaded after rendering the main page. This way, the landing page loads quickly and is ready for interaction. WebAssembly resources (wasm files and glue code scripts) and the JavaScript scanner library are preloaded and cached using XMLHttpRequest after loading the main page. It is important here that they are not executed immediately in order to leave the main thread free for the user to interact with the page. Execution occurs only when the user clicks on the barcode icon. If the user clicks on the icon before loading the resources, they will be loaded on demand and immediately executed. The barcode scanner event handler and the worker controller are loaded along with the page, but they are very small.
results
After rigorous testing and internal use by employees, we launched A / B testing on users. The test group was shown the scanner icon (screenshot below), and the control group was not.
Final product
To assess success, we introduced the metric “Draft Completion Rate”. This is the time between the start of editing a draft and submitting the form. The metric should show how a barcode scanner helps people fill out forms. The test lasted several weeks, and the results were very pleasant. They are completely consistent with our original hypothesis. Draft completion time decreased by 30% for flow with barcode scanner.
A / B test results
We also added profiling to evaluate the effectiveness of all types of scanners. As expected, Zbar made the largest contribution (53% of successful scans), then our C ++ library (34%) and, finally, the JavaScript library from 13%.
Conclusion
The experience of implementing WebAssembly has become very informative for us. Engineers are very pleased with the emergence of new technologies and immediately want to try them out. If the technology is also useful for customers, then this is a double joy. We repeat the idea expressed at the beginning of the article. Technology is developing at a very fast pace. Every day something new appears. But few technologies matter to customers, and WebAssembly is one of them. Our biggest conclusion from this exercise is to say “no” in 99 situations and “yes” in the only case where it is really important for customers.
In the future, we plan to expand the use of a barcode scanner and implement it on the buyers side so that they can scan product codes offline for their search and purchase on eBay. Also consider the option to extend the function using the Shape Detection API and other functions in the browser. But we are glad that we have found the right way to use WebAssembly on eBay and successfully applied the technology in e-commerce.
Special thanks to Surme Das and Lin Clark for the many articles on WebAssembly. They really helped us break the deadlock several times.
Source: https://habr.com/ru/post/453712/
All Articles