About EMC Captiva InputAccel and Kofax Capture Industrial Mass Input, Image Processing and OCR Systems
In the realities of Russian legislation, due to the lack of legal norms for conducting business electronically (paperless), accounting must create and store a very large number of paper documents (primary financial documents (accounts, invoices, invoices, etc.), transaction journals and operations, personnel documents and others).
To reduce processing and storage costs, medium and large companies resort to using automated systems for mass input and processing. The following software versions are widely known and used on the market: EMC, Kofax, Abbyy, Cognitive Technologies.
Over the last year of work, I managed to comprehend many secrets of this interesting area of ​​business automation using EMC Captiva and Kofax Capture systems from a complete zero, and I want to tell about them in this article.
Both manufacturers promote their products under the definition of "Mass Input Systems", and not as text recognition systems, and this is no accident. The thing is that text recognition as such is just a small task, of all that these systems can perform.
To begin with, the processing of Kofax and Captiva is the same principle: step by step. One processing step is conditionally launching one separate .exe, which performs certain specified actions. To create a so-called "process", there are special designers in which you define a sequence of steps and routing rules.
In the general scenario, the process of processing with recognition looks like this:
The figure shows that the document is sequentially processed from its preparation (separation of sheets, removal of clips), then capture (scanning), recognition and indexing (extracting certain parts of the text), then checking by the user (immediately partial or full manual indexing), forming The new output format (if necessary) and export from the processing system.
Since the source can be not only a scanner, but also a file system, facsimile systems, e-mail, web services and other systems, there may be other steps in processing (image enhancement, rotation, smoothing, converting), checking the indexed values ​​can be automatic, and the export can occur as a file system, in any ERP, CRM and other systems, sent by e-mail and other means, it is advisable to provide the structure of processing in the following form:
Logically, the processing systems look like separate running processes, interacting with the central server, which is responsible for routing the pre-configured process:
The system is built according to the modular architecture scheme and consists of the following main components (For example, Kofax Capture):
The database consists of a directly relational database and a directory on a file system, usually extended over a network, in which step-by-step document processing artifacts are stored.
The server performs the routing of documents in steps and modules.
Unmanaged modules run in the background without user interaction, for example, a recognition module.
Client modules, such as 'Scanning' or 'Indexing', are the main user interface of the system.
Administration is done in the Batch Administrator module for Kofax or the Administrative console for Captiva.
')
For both systems, scaling takes place, both vertical and horizontal. Services can be run on one computer or spaced apart in order to increase performance, run multiple times (under available licenses), which is why Recognition service is separated from other services on this scheme, since required most resources.
The quality of the input image can be low (the minimum typical requirement is 300dpi, 1 bit per pexel is black & white), contain artifacts, blemishes, blurriness and other noises, therefore, usually preprocessing is used, which can significantly change the image quality and improve recognition quality. EMC uses PixTools components, Kofax uses its development of VirtualReScan.
Original image before processing
Image after processing
An important step is the Classification (determining the type and form of a pre-configured document). The document can be determined either by graphic content, or by the presence of keywords or in a mixed way. It is also possible to classify “in code” —for example, when a document was received through a web service through an accounting system and its type is known in advance.
For recognition, different recognition engines can be used, which are supplied with the basic kit with the products, however, for the most part, the engine from Abbyy is used both there and there. It gives high quality recognition of Russian printed text. Handwriting input is difficult to recognize, so usually such documents, if they exist, are not recognized, and a limited number of fields are indexed by the operator.
It is important to note that the basic set of Captiva InputAccel, as well as Kofax Capture, allows indexing only strictly formal documents, such as bank questionnaires, where text fields are preset and when retrieved, we will always reliably know where to look for where. For processing weakly and unstructured documents, you must use the optional modules Captiva Dispatcher or Kofax Transformation Modules. In this case, full-page text recognition takes place and in most cases the principle of finding fields by regular expressions in combination with the position of “anchor” sections is used (anchor words can also be detected by regular expressions or rigid patterns). The same modules are necessary if you are going to process the tabular parts of documents, which, by the way, is the developer’s worst dream and, if possible, we try to abandon the tables.
Comparison of system components
Licensing of products is about the same, all licenses are competing, that is, on the number of active connections. You buy the volume of pages per year that you intend to process, such a license is renewable (the counter is reset once a year) and non-renewable. It is also necessary to additionally purchase licenses for modules, for example, in the Captiva delivery, only one scanner location and all the rest must be purchased separately.
In Russia, for the most part, the EMC Captiva system is prevalent, Kofax is used quite rarely.
If you are interested in continuing, I can describe in detail the process of developing and customizing the processes and recognition patterns based on the EMC Captiva platform.
To reduce processing and storage costs, medium and large companies resort to using automated systems for mass input and processing. The following software versions are widely known and used on the market: EMC, Kofax, Abbyy, Cognitive Technologies.
Over the last year of work, I managed to comprehend many secrets of this interesting area of ​​business automation using EMC Captiva and Kofax Capture systems from a complete zero, and I want to tell about them in this article.
Both manufacturers promote their products under the definition of "Mass Input Systems", and not as text recognition systems, and this is no accident. The thing is that text recognition as such is just a small task, of all that these systems can perform.
To begin with, the processing of Kofax and Captiva is the same principle: step by step. One processing step is conditionally launching one separate .exe, which performs certain specified actions. To create a so-called "process", there are special designers in which you define a sequence of steps and routing rules.
In the general scenario, the process of processing with recognition looks like this:
The figure shows that the document is sequentially processed from its preparation (separation of sheets, removal of clips), then capture (scanning), recognition and indexing (extracting certain parts of the text), then checking by the user (immediately partial or full manual indexing), forming The new output format (if necessary) and export from the processing system.
Since the source can be not only a scanner, but also a file system, facsimile systems, e-mail, web services and other systems, there may be other steps in processing (image enhancement, rotation, smoothing, converting), checking the indexed values ​​can be automatic, and the export can occur as a file system, in any ERP, CRM and other systems, sent by e-mail and other means, it is advisable to provide the structure of processing in the following form:
Logically, the processing systems look like separate running processes, interacting with the central server, which is responsible for routing the pre-configured process:
The system is built according to the modular architecture scheme and consists of the following main components (For example, Kofax Capture):
- Database
- Kofax Capture Server
- Unmanaged processing modules
- Client modules
The database consists of a directly relational database and a directory on a file system, usually extended over a network, in which step-by-step document processing artifacts are stored.
The server performs the routing of documents in steps and modules.
Unmanaged modules run in the background without user interaction, for example, a recognition module.
Client modules, such as 'Scanning' or 'Indexing', are the main user interface of the system.
Administration is done in the Batch Administrator module for Kofax or the Administrative console for Captiva.
')
For both systems, scaling takes place, both vertical and horizontal. Services can be run on one computer or spaced apart in order to increase performance, run multiple times (under available licenses), which is why Recognition service is separated from other services on this scheme, since required most resources.
The quality of the input image can be low (the minimum typical requirement is 300dpi, 1 bit per pexel is black & white), contain artifacts, blemishes, blurriness and other noises, therefore, usually preprocessing is used, which can significantly change the image quality and improve recognition quality. EMC uses PixTools components, Kofax uses its development of VirtualReScan.
Original image before processing
Image after processing
An important step is the Classification (determining the type and form of a pre-configured document). The document can be determined either by graphic content, or by the presence of keywords or in a mixed way. It is also possible to classify “in code” —for example, when a document was received through a web service through an accounting system and its type is known in advance.
For recognition, different recognition engines can be used, which are supplied with the basic kit with the products, however, for the most part, the engine from Abbyy is used both there and there. It gives high quality recognition of Russian printed text. Handwriting input is difficult to recognize, so usually such documents, if they exist, are not recognized, and a limited number of fields are indexed by the operator.
It is important to note that the basic set of Captiva InputAccel, as well as Kofax Capture, allows indexing only strictly formal documents, such as bank questionnaires, where text fields are preset and when retrieved, we will always reliably know where to look for where. For processing weakly and unstructured documents, you must use the optional modules Captiva Dispatcher or Kofax Transformation Modules. In this case, full-page text recognition takes place and in most cases the principle of finding fields by regular expressions in combination with the position of “anchor” sections is used (anchor words can also be detected by regular expressions or rigid patterns). The same modules are necessary if you are going to process the tabular parts of documents, which, by the way, is the developer’s worst dream and, if possible, we try to abandon the tables.
Comparison of system components
| Emc captiva | Kofax | What is he doing |
|---|---|---|
| InputAccel Server | Server process, manages the life cycle of the process (batch) | |
| KNS, not necessarily if the whole process will go on the same machine, does not require IIS | Networking | |
| InputAccel | Kofax capture | Basic processes, the ability to recognize only strictly structured documents, i.e. strict forms, questionnaires, etc. |
| Dispatcher | Ktm | Setting up recognition patterns for semi-structured documents and non-template texts (all Russian financial initial) |
| FreeForm Designer | No, built into KTM | Designer for setting complex recognition rules |
| Administation Console, Web application, requires IIS | No, there is no need built into Capture | Application for server settings, process and batch management, licensing, etc. |
| eInput | Kfs | Ability to work through the browser |
Licensing of products is about the same, all licenses are competing, that is, on the number of active connections. You buy the volume of pages per year that you intend to process, such a license is renewable (the counter is reset once a year) and non-renewable. It is also necessary to additionally purchase licenses for modules, for example, in the Captiva delivery, only one scanner location and all the rest must be purchased separately.
In Russia, for the most part, the EMC Captiva system is prevalent, Kofax is used quite rarely.
If you are interested in continuing, I can describe in detail the process of developing and customizing the processes and recognition patterns based on the EMC Captiva platform.
Source: https://habr.com/ru/post/144796/
All Articles