Mobile OCR. How it all began

The transfer of OCR technologies (optical recognition technologies) from a PC to mobile devices has been discussed, perhaps, from the moment they appeared. Back in the late 90s, when smart devices could be counted on the fingers, we thought about creating a program that extracts data from a business card, the Business Card Reader, or BCR for short. This idea, as they say, was in the air, but neither the capabilities of the cameras, nor the computing power of the devices then allowed it to be realized. There was not much choice: either a regular phone with a good camera “for ordinary people”, or an advanced business handheld “for professionals” (and why do businessmen need cameras on the device - photograph themselves or something?)

But as time went on, people's demands grew, and more or less decent cameras began to appear in all devices. As soon as the capabilities of the devices were “pulled up”, the developers ported our recognition technology, taking into account a lot of limitations inherent in mobile operating systems: library size (try cramming OCR with an image base of 500 kilobytes on the media), speed on unproductive processors. I also had to take into account that it was not the ideal images from the scanner that were to be recognized, but photographs, taken often with distortions, in poor lighting conditions, etc.

And so, at the end of the 90s, ABBYY’s first text recognition application was released, and it was designed for a portable scanner, not for a phone. This scanner manufactured by the Swedish company C Technologies AB was a small device, remotely resembling a marker or pen, and allowed to scan books, magazines, newspapers line by line.
')
The mobile SDK was built into this pen, the user led the pen along the line, the device recognized the text and immediately showed it on a small screen - then the text could be transferred to the computer. This mechanism of work now seems to be an anachronism, just look at the old device of that time, but in those times it seemed like magic!



In order to make a version for C Pen from a large FineReader, it took one person-year (in fact, one developer from our team did it all then).

C Technologies AB still exists and produces portable scanners that use modern solutions from ABBYY.



After such a successful start, the team of developers and analysts had no doubt that mobile OCR technology should be implemented in a commercial product. In 2005, for the first time by the South Korean company Diotec (the company developed applications that were pre-installed on Samsung phones), we first made an SDK that worked on a mobile phone. Thanks to this collaboration, several Samsung cell phones in Europe and Indonesia have been equipped with our technology. In addition to the main text recognition libraries in the Mobile SDK, libraries were added, with which the program determined which line contains the phone number, which name and surname (even if they are typed, for example, in Swedish), and which address - address. This was the functionality that allows you to recognize business cards and store data in the desired fields of the phone's notebook.

In 2007, we launched a separate product, ABBYY Mobile OCR SDK, a toolkit that allowed third-party developers to incorporate OCR in their mobile devices and knew how to recognize business cards. The phone on which we developed the first version of the Mobile SDK, the Nokia 6630 . It already had a suitable operating system, but it was not yet suitable for recognizing the camera, so the image files were poured into it through a cord or Bluetooth.



History has kept the instructions for the first display of the product at the exhibition.



The toolkit could be used on the most popular then mobile platforms, including Windows Mobile, Symbian or Linux, and on any mobile devices that could somehow receive images: smartphones, handheld computers, portable scanners, digital cameras. And in early 2008, the first product came out that users themselves could install on their phones, the ABBYY Business Card Reader under Symbian. Even then, the program for work did not require a permanent connection to the Internet, it supported 3 interface languages ​​and 16 text recognition languages.







Unfortunately, the mess with the versions of Symbian, a huge number of manufactured mobile devices (except Nokia, the devices were produced by Samsung, Motorola and Sony, and the latter even wrote their own modification of Symbian - UIQ) did not allow us to develop the application as we would like. Some of the phone manufacturers supported touch input (for example, Sony), someone until a certain point is not (Nokia), each company tried to build on the system something of their own, which will be loved by users. For each device, it was necessary to develop and maintain a separate version of BCR, it took time of developers, and did not give the opportunity to move forward.

Everything changed, as usual, the first iPhone. We wanted to try to make a Business Card Reader for the new OS , but at that moment it was not obvious to us whether the new smartphone would be popular, so we decided to make the MVP (minimal valuable product) - a product with minimal effort. The development was given to the team of outsourcers who had both experience and devices for testing. So the first version of the BCR for iOS was done by the outsourcers from the company Media Moji, the application worked on the iPhone 3G. The "treshka" had problems with autofocus, so business cards were not recognized very well on it. In iPhone 3GS, the problem was fixed, and the next version of BCR already recognized the data as it should.



Of course, as soon as the demand for the product was confirmed, the development moved to the state. By the way, right now we are looking for a new head of the iOS development team. The requirements are simple - you are a cool specialist with experience of 3 years and a good portfolio in the background. If interested, details are here .

You ask - what about Android? As we have said, the company was initially cautious about new mobile platforms, and we started developing for Android only when we became confident that this platform would be popular (until this happened, our blog’s readers left about two hundred comments with questions when our mobile products for Android will be released). As a result, the first BCR for this platform was released a whole year later than under iOS.



The growing popularity of iOS and Android devices has made life easier for all mobile application developers, including us. At the time of Windows Mobile and Symbian, there were a lot of mobile app stores on the Internet. It was a big headache, because different online stores had different licensing systems, and if we wanted to be present with our applications, we had to meet these requirements and make changes to distributions. We have multiplied huge tables with versions of distributions (only one Business Card Reader existed in 50 different versions), the support of such a number of versions required working hands and time. The emergence of centralized mobile app stores - AppStore and Android Market (now Google Play) - solved this problem, and we were able to focus on developing the capabilities of the application.

We will not list here everything that we have invented since then in BCR - the article is not about that. We will tell only about how you were looking for ways to implement contactless transfer of business cards between devices. The topic of the electronic exchange of business cards has been discussed for five years now, the developers are trying to find out who is in that much - for example, in one of the implementations (not ours), in order to exchange business cards, it was necessary to knock smartphones on each other. We also made our own version of contactless transfer over Wi-Fi based on AllJoyn technology from Qualcomm (June 2013). But not everything went as smoothly as it should. Visually, the process of self-transferring business cards between two smartphones looked very impressive, and even worked between iOS and Android devices, but there were a lot of restrictions:

• for the exchange of business cards for both users had to stand BCR
• both users must have been on the same Wi-Fi network
• this network was supposed to be public and allow such data exchange.

To tell the truth, even in the same grid, the exchange did not work every time, and we did not develop this functionality further.



As a result, we realized that the easiest and most popular way to exchange is good old e-mail. And now such an exchange scenario has been implemented: after scanning an alien business card, the user immediately sees on the screen a large “Send your business card” button. He presses on it, and in the old manner, an email is sent by email with a welcome text similar to this:



Template text, of course, can be pre-configured. Attached to the letter is an image of a business card and a VCF file. It seems to us that for the exchange of electronic business cards this is by far the most lively scenario.

Of course, developers like to fasten all kinds of "magic" to business card reading applications - and we are no exception. For example, recently (May 2015) in our application appeared avtozakhvat - now you do not need to click on the camera's lowering, you just need to point it at the business card - focusing, defining borders, cropping the background, improving the image and recognition occur “automatically” in a split second. Soon BCR will add profile data from some social networks and several other cool chips.

For the past ten years, everyone has been predicting the death of paper business cards, but the years go by, and people continue to exchange these pieces of paper. Perhaps in 5-10 years there will be a single format of electronic exchange of contact data with bots via blockchain with big data and deep machine learning, and we, of course, are already preparing for this, but until then the topic of photographing and recognizing business cards will remain in demand.

Well, about how we tried to make a mobile FineReader and what came of it, read the second series. Do not switch.