OpenVX: computer vision standard
Author: Victor Erukhimov, Executive Director, Itseez, Chairman of the OpenVX Working Group
The Khronos Group on November 18, 2013 presented a preliminary specification of the OpenVX 1.0 standard for computer vision. Since Itseez was one of the initiators of this activity and actively participated in the creation of the specification, we decided to tell about this standard to the audience of Habrahabr.
When a few years ago, Itseez engineers began to engage in computer vision on mobile platforms, they were amazed at how slow the algorithms of mobile phone processors are compared to desktop computers, which everyone was used to. The simple function of changing the image resolution (cv :: resize) could work longer than 20 milliseconds, leaving almost no time for processing video in real time (a function that processes 30 frames per second should work for approximately 33 ms per frame). For example, using NEON optimization, cv :: resize can be accelerated more than 7 times on ARM (see [ 1 ]). It became obvious that a more effective interaction of the code with the SoC is needed, and not only with the central processor, but with all the available accelerators, including GPU, DSP , ASIC , FPGA . No company will solve this problem alone, so Itseez together with NVIDIA turned to Khronos Group with a proposal to create a standard for computer vision, which is a Hardware Abstraction Layer (HAL), that is, an interface whose functions will be called high-level algorithms, and The release will be optimized for computational architecture. Examples of standards from Khronos - OpenGL and OpenCL , solve this problem for graphics and general calculations, respectively. The proposal was supported by a large number of companies, and a working group was created, Itseez employee was elected as the chairman. Industry leaders are involved in developing the standard (see below). With such powerful support that the idea of OpenVX received, you can really change the industry by making the development of computer vision applications easier than ever!
')
The standard was created in order to speed up computer vision algorithms primarily on mobile and embedded architectures.
OpenVX is a C API of two levels: immediate mode and graph mode. The first is the individual functions, the structure is very similar to the primitives from OpenCV . All functions have an equivalent in OpenCV, mostly in the imgproc module.
The upper level allows you to describe the algorithm of computer vision in the form of a directed graph, where each node corresponds to a function. For each function immediate mode, there is an equivalent graph node. The graph is completely set before execution. The user can define his function through the C API callback and include it in the graph (user nodes, see user nodes in the Stereo Machine Vision example below). Both API levels work with containers for images, the internal structure of which is not specified - the user operates with identifiers:
Thus, these containers are opaque (opaque), which gives greater freedom of implementation. The distribution of images and graph nodes by streams and accelerators is almost not controlled by the standard at all, and, accordingly, is left to the discretion of the developer implementing the standard. As part of this architecture, you can store images and execute OpenVX functions on accelerators, which can greatly speed up the algorithm or make it more energy efficient. In general, the Graph API opens a lot of room for optimizations. For example, you can perform several operations in parallel on different cores or different accelerators. Under certain conditions, it is possible to execute two consecutive nodes of the graph simultaneously in one stream, performing calculations on the fly, or process the image in tiling, in order to increase the cache efficiency. Tiling support for user nodes is implemented using an extension of the OpenVX 1.0 standard. Tiling Extension is an interface approved by the OpenVX group, but not mandatory for implementation for OpenVX 1.0. In the future, perhaps, Tiling API will become an obligatory part of the next versions of the standard.
From the very beginning, OpenVX was designed to interact effectively with OpenCV: data exchange with no overhead, if possible, very close specifications of functions. Perhaps in the future, OpenCV will use OpenVX for acceleration on a number of platforms. But since OpenVX 1.0 is still in development, and OpenCV 3.0 will be architecturally different from previous versions, it’s difficult to describe the interaction model now.
During the work on OpenVX, we received many questions about the interaction between OpenVX and OpenCL . For example, why can't OpenVX just be an OpenCL kernel set (kernels)? We decided to make OpenVX not requiring dependence on OpenCL, because there are accelerators that do not support OpenCL, but are able to perform part of the functions of OpenVX. However, OpenVX is designed to enable OpenVX implementation on OpenCL.
A lot of developers of computer vision technologies are now working with one platform, choosing an algorithm and optimizing its implementation for one system. Given the widespread support of OpenVX for different platforms, efforts to adapt the algorithm for another platform will be minimal. In the near future, this will lead to the growth of cool computer vision applications for mobile phones and tablets. In the longer term, OpenVX will encourage chip manufacturers to create accelerators oriented towards OpenVX. This will lead to a radical acceleration of computer vision technologies and create conditions for smarter applications that no one even dreams of now, such as augmented reality on wearable devices and the functions of safe driving available on economy cars. .
A preliminary version of the specification is now published ( direct link to the document ). Its goal is to collect community feedback and, if possible, take them into account in the final OpenVX 1.0 specification, which is expected to be released no later than mid-2014. Now you can leave messages on the forum , in the future access to bugzilla will open. We encourage all future OpenVX users to look at the standard! And, of course, we invite all enthusiasts of computer vision to send us your resume to us at jobs@itseez.com !
Good luck to everyone and fast algorithms!
[1] Kari Pulli, Anatoly Baksheev, Kirill Kornyakov, Victor Eruhimov: Real-time computer vision with OpenCV. Commun. ACM (CACM) 55 (6): 61-69 (2012) .
The Khronos Group on November 18, 2013 presented a preliminary specification of the OpenVX 1.0 standard for computer vision. Since Itseez was one of the initiators of this activity and actively participated in the creation of the specification, we decided to tell about this standard to the audience of Habrahabr.Why do I need OpenVX
When a few years ago, Itseez engineers began to engage in computer vision on mobile platforms, they were amazed at how slow the algorithms of mobile phone processors are compared to desktop computers, which everyone was used to. The simple function of changing the image resolution (cv :: resize) could work longer than 20 milliseconds, leaving almost no time for processing video in real time (a function that processes 30 frames per second should work for approximately 33 ms per frame). For example, using NEON optimization, cv :: resize can be accelerated more than 7 times on ARM (see [ 1 ]). It became obvious that a more effective interaction of the code with the SoC is needed, and not only with the central processor, but with all the available accelerators, including GPU, DSP , ASIC , FPGA . No company will solve this problem alone, so Itseez together with NVIDIA turned to Khronos Group with a proposal to create a standard for computer vision, which is a Hardware Abstraction Layer (HAL), that is, an interface whose functions will be called high-level algorithms, and The release will be optimized for computational architecture. Examples of standards from Khronos - OpenGL and OpenCL , solve this problem for graphics and general calculations, respectively. The proposal was supported by a large number of companies, and a working group was created, Itseez employee was elected as the chairman. Industry leaders are involved in developing the standard (see below). With such powerful support that the idea of OpenVX received, you can really change the industry by making the development of computer vision applications easier than ever!
')
The standard was created in order to speed up computer vision algorithms primarily on mobile and embedded architectures.
OpenVX architecture
OpenVX is a C API of two levels: immediate mode and graph mode. The first is the individual functions, the structure is very similar to the primitives from OpenCV . All functions have an equivalent in OpenCV, mostly in the imgproc module.
The upper level allows you to describe the algorithm of computer vision in the form of a directed graph, where each node corresponds to a function. For each function immediate mode, there is an equivalent graph node. The graph is completely set before execution. The user can define his function through the C API callback and include it in the graph (user nodes, see user nodes in the Stereo Machine Vision example below). Both API levels work with containers for images, the internal structure of which is not specified - the user operates with identifiers:
typedef uintptr_t vx_reference; typedef vx_reference vx_image; Thus, these containers are opaque (opaque), which gives greater freedom of implementation. The distribution of images and graph nodes by streams and accelerators is almost not controlled by the standard at all, and, accordingly, is left to the discretion of the developer implementing the standard. As part of this architecture, you can store images and execute OpenVX functions on accelerators, which can greatly speed up the algorithm or make it more energy efficient. In general, the Graph API opens a lot of room for optimizations. For example, you can perform several operations in parallel on different cores or different accelerators. Under certain conditions, it is possible to execute two consecutive nodes of the graph simultaneously in one stream, performing calculations on the fly, or process the image in tiling, in order to increase the cache efficiency. Tiling support for user nodes is implemented using an extension of the OpenVX 1.0 standard. Tiling Extension is an interface approved by the OpenVX group, but not mandatory for implementation for OpenVX 1.0. In the future, perhaps, Tiling API will become an obligatory part of the next versions of the standard.
Openvx and other tools open-two-cap-letters
From the very beginning, OpenVX was designed to interact effectively with OpenCV: data exchange with no overhead, if possible, very close specifications of functions. Perhaps in the future, OpenCV will use OpenVX for acceleration on a number of platforms. But since OpenVX 1.0 is still in development, and OpenCV 3.0 will be architecturally different from previous versions, it’s difficult to describe the interaction model now.
During the work on OpenVX, we received many questions about the interaction between OpenVX and OpenCL . For example, why can't OpenVX just be an OpenCL kernel set (kernels)? We decided to make OpenVX not requiring dependence on OpenCL, because there are accelerators that do not support OpenCL, but are able to perform part of the functions of OpenVX. However, OpenVX is designed to enable OpenVX implementation on OpenCL.
How OpenVX can affect the industry
A lot of developers of computer vision technologies are now working with one platform, choosing an algorithm and optimizing its implementation for one system. Given the widespread support of OpenVX for different platforms, efforts to adapt the algorithm for another platform will be minimal. In the near future, this will lead to the growth of cool computer vision applications for mobile phones and tablets. In the longer term, OpenVX will encourage chip manufacturers to create accelerators oriented towards OpenVX. This will lead to a radical acceleration of computer vision technologies and create conditions for smarter applications that no one even dreams of now, such as augmented reality on wearable devices and the functions of safe driving available on economy cars. .
How can I influence OpenVX?
A preliminary version of the specification is now published ( direct link to the document ). Its goal is to collect community feedback and, if possible, take them into account in the final OpenVX 1.0 specification, which is expected to be released no later than mid-2014. Now you can leave messages on the forum , in the future access to bugzilla will open. We encourage all future OpenVX users to look at the standard! And, of course, we invite all enthusiasts of computer vision to send us your resume to us at jobs@itseez.com !
Good luck to everyone and fast algorithms!
Links
[1] Kari Pulli, Anatoly Baksheev, Kirill Kornyakov, Victor Eruhimov: Real-time computer vision with OpenCV. Commun. ACM (CACM) 55 (6): 61-69 (2012) .
Source: https://habr.com/ru/post/204236/
All Articles