Machine vision. What is it and how to use it? Optical source image processing
Machine vision is a scientific direction in the field of artificial intelligence, in particular robotics, and related technologies for obtaining images of real-world objects, processing them, and using the obtained data to solve various kinds of applied tasks without the participation of (full or partial) man.
')
Image processing and image analysis are mainly focused on working with 2D images, i.e. how to convert one image to another. For example, pixel-by-pixel operations to increase contrast, operations to highlight edges, eliminate noise, or geometric transformations, such as rotating an image. These operations assume that image processing / analysis acts independently of the content of the images themselves.
Computer vision focuses on processing three-dimensional scenes designed for one or more images. For example, by restoring a structure or other information about a 3D scene from one or more images. Computer vision often depends on more or less complex assumptions about what is represented in the images.
There is also an area called visualization, which was originally associated with the process of creating images, but sometimes dealt with processing and analysis. For example, radiography works with the analysis of video data for medical use.
Finally, pattern recognition is an area that uses various methods to obtain information from video data, mainly based on a statistical approach. Much of this area is devoted to the practical application of these methods.
Thus, we can conclude that the concept of “machine vision” today includes: computer vision, recognition of visual images, analysis and image processing, etc.
The classical problem in computer vision, image processing and computer vision is the definition of whether the video data contains some characteristic object, feature or activity.
This task can be reliably and easily solved by man, but it has not been solved satisfactorily in computer vision in the general case so far: random objects in random situations.
One or more predefined or studied objects or classes of objects can be recognized (usually together with their two-dimensional position in the image or three-dimensional position in the scene).
An individual instance of an object belonging to a class is recognized.
Examples: the identification of a specific human face or fingerprint or car.
Video data are checked for a specific condition.
Detection based on relatively simple and fast calculations is sometimes used to find small areas in the analyzed image, which are then analyzed using techniques that are more demanding of resources to get the correct interpretation.
Search for images by content: finding all the images in a large set of images that have content defined in various ways.
Position evaluation: determining the position or orientation of a particular object relative to the camera.
Optical character recognition: character recognition on images of typed or handwritten text (usually for translation into the text format most suitable for editing or indexing. For example, ASCII).
Restoration of 3D shape from 2D images is performed using stereo-construction of the depth map, reconstruction of the normal field and depth map from the halftone image, reconstruction of the depth map from the texture and shape determination from the movement
An example of the restoration of 3D shapes in 2D image
Several tasks related to motion estimation, in which a sequence of images (video data) are processed to find an estimate of the speed of each point of the image or 3D scene. Examples of such tasks are: determining the three-dimensional movement of the camera, tracking, that is, following the movement of an object (for example, cars or people)
Given two or more images of the scene, or video. Scene restoration has the task to recreate the three-dimensional model of the scene. In the simplest case, the model may be a set of points of three-dimensional space. More complex methods reproduce the full three-dimensional model.
The task of image restoration is noise removal (sensor noise, motion blur, etc.).
The simplest approach to solving this problem are various types of filters, such as low or medium frequency filters.
A higher level of noise removal is achieved during the initial analysis of video data for the presence of various structures, such as lines or borders, and then controlling the filtering process based on this data.
Optical flow analysis (finding the movement of pixels between two images).
Several tasks related to motion estimation, in which a sequence of images (video data) are processed to find an estimate of the speed of each point of the image or 3D scene.
Examples of such tasks are: definition of three-dimensional camera movement, tracking, i.e. following the movement of an object (for example, cars or people).
Counts the number of light or dark pixels.
Using the pixel counter, the user can select a rectangular area on the screen in a place of interest, for example, where he expects to see the faces of people passing by. The camera will immediately respond with information about the number of pixels represented by the sides of the rectangle.
The pixel counter allows you to quickly check whether the mounted camera complies with regulatory requirements or customer requirements regarding pixel resolution, for example, for the faces of people entering the doors controlled by the camera, or for license plate recognition.
Converts an image in grayscale to binary (white and black pixels).
The values of each pixel are conventionally encoded as “0” and “1”. The value "0" is conventionally called the background or background and "1" - the foreground.
Often, when storing digital binary images, a bitmap is used, where one bit of information is used to represent one pixel.
Also, especially in the early stages of the development of technology, the two possible colors were black and white, which is not mandatory.
Used for searching and / or counting parts.
The goal of segmentation is to simplify and / or change the representation of the image so that it is easier and easier to analyze.
Image segmentation is usually used to highlight objects and borders (lines, curves, etc.) in images. More precisely, image segmentation is the process of assigning such labels to each pixel of an image that pixels with the same labels have common visual characteristics.
The result of image segmentation is a set of segments that together cover the entire image, or a set of contours, separated from the image. All pixels in a segment are similar in some characteristic or computed property, for example, in color, brightness, or texture. Neighboring segments differ significantly in this characteristic.
Barcode - graphic information applied to the surface, marking or packaging of products, representing the possibility of reading it with technical means - a sequence of black and white stripes or other geometric shapes.
In machine vision, bar codes are used to decode 1D and 2D codes designed to be read or scanned by machines.
Optical character recognition: automated text reading, such as serial numbers.
Recognition is used to convert books and documents into electronic form, to automate business accounting systems, or to publish text on a web page.
OCR allows you to edit text, search for words or phrases, store it in a more compact form, display or print material without losing quality, analyze information, and also apply electronic translation, formatting, or conversion to speech to text.
Used computer vision for non-destructive quality control of superconducting materials.
Introduction Solving the tasks of ensuring integrated security (both the antiterrorist and mechanical security of facilities and the technological security of engineering systems) currently requires the system organization of control, the current state of facilities. One of the most promising ways to control the current state of objects is optical and optical-electronic methods based on video image processing technologies of an optical source. These include: programs for working with images; newest image processing; equipment for image acquisition, analysis and processing, i.e. a complex of tools and methods relating to the field of computer and computer vision. Computer vision is a common set of methods that allow computers to see and recognize three- or two-dimensional objects, both engineering and non-engineering. To work with computer vision, digital or analog I / O devices are required, as well as computer networks and IP location analyzers designed to monitor the production process and prepare information for making operational decisions in the shortest possible time.
Formulation of the problem. Today, the main task for the designed machine vision complexes remains the detection, recognition, identification and qualification of potential risk objects located in a random place in the operational area of the complex. Currently existing software products aimed at solving the listed problems have a number of significant drawbacks, namely: significant complexity associated with the high detail of optical images; high power consumption and a fairly narrow range of possibilities. Extending the tasks of detecting objects of potential risk to the area of searching for random objects in random situations that are in a random place is not possible with available software products, even with the use of a supercomputer.
Purpose. Development of a universal program for processing images of an optical source, with the possibility of streaming data analysis, that is, the program must be easy and fast so that it can be recorded on a small-sized computer device.
Tasks:
The relevance of the program is determined by:
Analysis of the relevance of the program.
To simplify the solution of problems of this type, I developed a mathematical model and wrote a program for a computer device for image analysis using the simplest transformations of the original images.
The program works with conversions such as binarization, brightness, image contrast, etc. The principle of the program is demonstrated by the example of the analysis of superconducting materials.
When creating composite superconductors based on Nb3Sn, the volume ratio of bronze and niobium varies, the size and number of fibers in it, the uniformity of their distribution over the cross section of the bronze matrix, the presence of diffusion barriers and stabilizing materials. For a given volume fraction of niobium in the conductor, an increase in the number of fibers leads, respectively, to a decrease in their diameter. This leads to a noticeable increase in the Nb / Cu – Sn interaction surface, which greatly accelerates the growth process of the superconducting phase. Such an increase in the number of the superconducting phase with an increase in the number of fibers in the conductor provides an increase in the critical characteristics of the superconductor. In this regard, it is necessary to have a tool to control the volume fraction of the superconducting phase in the final product (composite superconductor).
When creating the program, the importance of conducting research on the materials from which superconducting cables are created was taken into account, since if the ratio of niobium to bronze is incorrect, an explosion of wires is possible, and, consequently, human sacrifice, monetary costs and loss of time. This program allows you to determine the quality of the wires on the basis of chemical physical analysis of the object.
Program block diagram
Description of the stages of the study.
Stage 1. Sample preparation: cutting of a composite superconductor on an EDM machine; pressing the sample into the plastic matrix; polishing the sample to a mirror state; etching a sample to isolate niobium fibers on a bronze matrix. Samples of pressed composite superconducting samples were obtained;
Stage 2 Image acquisition: metallographic imaging on a scanning electron microscope.
Stage 3 Image processing: creating a tool for determining the volume fraction of the superconducting phase in a metallographic image; a set of statistically significant data on a specific type of sample. Created mathematical models of various tools for image processing; a software development was created to estimate the volume fraction of the superconducting phase; the program was facilitated by combining several mathematical functions into one; An average value of the volume fraction of niobium fibers in a bronze matrix of 24.7 ± 0.1% was obtained. A low deviation rate indicates a high repeatability of the composite wire structure.
Electron microscopic images of composite superconductors
Methods of image processing in the program.
Formulas of the program
Binarization formula (Otsu method):
Formula of erosion:
Dilatation formula:
Dilation and erosion pattern
Segmentation formulas by color thresholds:
Determining the magnitude of the gradient of brightness for each pixel of the image:
Threshold Calculation:
Used equipment
Program interface
Historical breakthroughs in machine vision
- 1955 - Oliver Selfridge. The article "Eyes and ears of a computer."
- 1958 - Frank Rosenblatt. Computer implementation of the perceptron.
- The 1960s are the first image processing systems.
- 1970s - Lawrence Roberts. The concept of machine construction of three-dimensional images of objects.
- 1979 - Hans-Helmut Nagel. Theory of the analysis of dynamic scenes.
- 1990s - The first unmanned vehicle control systems.
- 2003 - Corporate face recognition systems.
Machine Vision Components
Machine vision focuses on the use of, mainly industrial, for example, autonomous robots and visual inspection and measurement systems. This means that the technology of image sensors and control theory is associated with the processing of video data to control the robot and the processing of the received data in real time is carried out by software or hardware.
- One or more digital or analog cameras (black and white or color) with suitable optics for imaging
- Software for making images for processing. For analog cameras, this is the image digitizer.
- Processor (modern PC with multi-core processor or embedded processor, for example, DSP)
- Machine vision software that provides tools for the development of individual software applications.
- I / O equipment or communication channels for the results report
- Smart camera: one device that includes all the above items.
- Very specialized light sources (LEDs, fluorescent and halogen lamps, etc.)
- Specific software applications for image processing and detection of relevant properties.
- A sensor to synchronize detection parts (often an optical or magnetic sensor) for capturing and processing images.
- Drives of a certain form used to sort or discard defective parts.
')
Image processing and image analysis are mainly focused on working with 2D images, i.e. how to convert one image to another. For example, pixel-by-pixel operations to increase contrast, operations to highlight edges, eliminate noise, or geometric transformations, such as rotating an image. These operations assume that image processing / analysis acts independently of the content of the images themselves.
Computer vision focuses on processing three-dimensional scenes designed for one or more images. For example, by restoring a structure or other information about a 3D scene from one or more images. Computer vision often depends on more or less complex assumptions about what is represented in the images.
There is also an area called visualization, which was originally associated with the process of creating images, but sometimes dealt with processing and analysis. For example, radiography works with the analysis of video data for medical use.
Finally, pattern recognition is an area that uses various methods to obtain information from video data, mainly based on a statistical approach. Much of this area is devoted to the practical application of these methods.
Thus, we can conclude that the concept of “machine vision” today includes: computer vision, recognition of visual images, analysis and image processing, etc.
Tasks of machine vision
- Recognition
- Identification
- Detection
- Text recognising
- Restoration of 3D forms from 2D images
- Motion estimation
- Scene restoration
- Image Recovery
- Selection on images of structures of a certain type, image segmentation
- Optical flow analysis
Recognition
The classical problem in computer vision, image processing and computer vision is the definition of whether the video data contains some characteristic object, feature or activity.
This task can be reliably and easily solved by man, but it has not been solved satisfactorily in computer vision in the general case so far: random objects in random situations.
One or more predefined or studied objects or classes of objects can be recognized (usually together with their two-dimensional position in the image or three-dimensional position in the scene).
Identification
An individual instance of an object belonging to a class is recognized.
Examples: the identification of a specific human face or fingerprint or car.
Detection
Video data are checked for a specific condition.
Detection based on relatively simple and fast calculations is sometimes used to find small areas in the analyzed image, which are then analyzed using techniques that are more demanding of resources to get the correct interpretation.
Text recognising
Search for images by content: finding all the images in a large set of images that have content defined in various ways.
Position evaluation: determining the position or orientation of a particular object relative to the camera.
Optical character recognition: character recognition on images of typed or handwritten text (usually for translation into the text format most suitable for editing or indexing. For example, ASCII).
Restoration of 3D shape from 2D images is performed using stereo-construction of the depth map, reconstruction of the normal field and depth map from the halftone image, reconstruction of the depth map from the texture and shape determination from the movement
An example of the restoration of 3D shapes in 2D image
Motion estimation
Several tasks related to motion estimation, in which a sequence of images (video data) are processed to find an estimate of the speed of each point of the image or 3D scene. Examples of such tasks are: determining the three-dimensional movement of the camera, tracking, that is, following the movement of an object (for example, cars or people)
Scene restoration
Given two or more images of the scene, or video. Scene restoration has the task to recreate the three-dimensional model of the scene. In the simplest case, the model may be a set of points of three-dimensional space. More complex methods reproduce the full three-dimensional model.
Image Recovery
The task of image restoration is noise removal (sensor noise, motion blur, etc.).
The simplest approach to solving this problem are various types of filters, such as low or medium frequency filters.
A higher level of noise removal is achieved during the initial analysis of video data for the presence of various structures, such as lines or borders, and then controlling the filtering process based on this data.
Image Recovery
Optical flow analysis (finding the movement of pixels between two images).
Several tasks related to motion estimation, in which a sequence of images (video data) are processed to find an estimate of the speed of each point of the image or 3D scene.
Examples of such tasks are: definition of three-dimensional camera movement, tracking, i.e. following the movement of an object (for example, cars or people).
Image processing methods
- Pixel count
- Binarization
- Segmentation
- Barcode reading
- Optical character recognition
- Measurement
- Edge detection
- Pattern matching
Pixel count
Counts the number of light or dark pixels.
Using the pixel counter, the user can select a rectangular area on the screen in a place of interest, for example, where he expects to see the faces of people passing by. The camera will immediately respond with information about the number of pixels represented by the sides of the rectangle.
The pixel counter allows you to quickly check whether the mounted camera complies with regulatory requirements or customer requirements regarding pixel resolution, for example, for the faces of people entering the doors controlled by the camera, or for license plate recognition.
Binarization
Converts an image in grayscale to binary (white and black pixels).
The values of each pixel are conventionally encoded as “0” and “1”. The value "0" is conventionally called the background or background and "1" - the foreground.
Often, when storing digital binary images, a bitmap is used, where one bit of information is used to represent one pixel.
Also, especially in the early stages of the development of technology, the two possible colors were black and white, which is not mandatory.
Segmentation
Used for searching and / or counting parts.
The goal of segmentation is to simplify and / or change the representation of the image so that it is easier and easier to analyze.
Image segmentation is usually used to highlight objects and borders (lines, curves, etc.) in images. More precisely, image segmentation is the process of assigning such labels to each pixel of an image that pixels with the same labels have common visual characteristics.
The result of image segmentation is a set of segments that together cover the entire image, or a set of contours, separated from the image. All pixels in a segment are similar in some characteristic or computed property, for example, in color, brightness, or texture. Neighboring segments differ significantly in this characteristic.
Barcode reading
Barcode - graphic information applied to the surface, marking or packaging of products, representing the possibility of reading it with technical means - a sequence of black and white stripes or other geometric shapes.
In machine vision, bar codes are used to decode 1D and 2D codes designed to be read or scanned by machines.
Optical character recognition
Optical character recognition: automated text reading, such as serial numbers.
Recognition is used to convert books and documents into electronic form, to automate business accounting systems, or to publish text on a web page.
OCR allows you to edit text, search for words or phrases, store it in a more compact form, display or print material without losing quality, analyze information, and also apply electronic translation, formatting, or conversion to speech to text.
My program, written in LabView on working with images
Used computer vision for non-destructive quality control of superconducting materials.
Introduction Solving the tasks of ensuring integrated security (both the antiterrorist and mechanical security of facilities and the technological security of engineering systems) currently requires the system organization of control, the current state of facilities. One of the most promising ways to control the current state of objects is optical and optical-electronic methods based on video image processing technologies of an optical source. These include: programs for working with images; newest image processing; equipment for image acquisition, analysis and processing, i.e. a complex of tools and methods relating to the field of computer and computer vision. Computer vision is a common set of methods that allow computers to see and recognize three- or two-dimensional objects, both engineering and non-engineering. To work with computer vision, digital or analog I / O devices are required, as well as computer networks and IP location analyzers designed to monitor the production process and prepare information for making operational decisions in the shortest possible time.
Formulation of the problem. Today, the main task for the designed machine vision complexes remains the detection, recognition, identification and qualification of potential risk objects located in a random place in the operational area of the complex. Currently existing software products aimed at solving the listed problems have a number of significant drawbacks, namely: significant complexity associated with the high detail of optical images; high power consumption and a fairly narrow range of possibilities. Extending the tasks of detecting objects of potential risk to the area of searching for random objects in random situations that are in a random place is not possible with available software products, even with the use of a supercomputer.
Purpose. Development of a universal program for processing images of an optical source, with the possibility of streaming data analysis, that is, the program must be easy and fast so that it can be recorded on a small-sized computer device.
Tasks:
- development of a mathematical model of the program;
- writing a program;
- testing of the program in a laboratory experiment, with full preparation and conduct of the experiment;
- study of the possibility of applying the program in related areas of activity.
The relevance of the program is determined by:
- the absence of image processing software on the software market with a detailed analysis of the engineering components of the objects;
- the ever-growing demands for quality and speed of obtaining visual information that dramatically increase the demand for image processing programs;
- the existing need for high-performance programs that are reliable and simple from the user's point of view;
- the high cost of professional visual processing software.
Analysis of the relevance of the program.
- the absence of image processing software on the software market with a detailed analysis of the engineering components of the objects;
- the ever-growing demands for quality and speed of obtaining visual information that dramatically increase the demand for image processing programs;
- the existing need for high-performance programs that are reliable and simple from the user's point of view;
- there is a need for high performance programs and simple control, which is extremely difficult to achieve in our time. For example, I took Adobe Photoshop. This graphic editor has a harmonious combination of functionality and ease of use for an ordinary user, but in this program it is impossible to work with complex tools for image processing (for example, image analysis by building a mathematical dependence (function) or integral image processing);
- the high cost of professional visual processing software. If the software is of high quality, then the price for it is extremely high, down to the individual functions of one or another set of programs. The graph below shows the price / quality relationship of simple program analogues.
To simplify the solution of problems of this type, I developed a mathematical model and wrote a program for a computer device for image analysis using the simplest transformations of the original images.
The program works with conversions such as binarization, brightness, image contrast, etc. The principle of the program is demonstrated by the example of the analysis of superconducting materials.
When creating composite superconductors based on Nb3Sn, the volume ratio of bronze and niobium varies, the size and number of fibers in it, the uniformity of their distribution over the cross section of the bronze matrix, the presence of diffusion barriers and stabilizing materials. For a given volume fraction of niobium in the conductor, an increase in the number of fibers leads, respectively, to a decrease in their diameter. This leads to a noticeable increase in the Nb / Cu – Sn interaction surface, which greatly accelerates the growth process of the superconducting phase. Such an increase in the number of the superconducting phase with an increase in the number of fibers in the conductor provides an increase in the critical characteristics of the superconductor. In this regard, it is necessary to have a tool to control the volume fraction of the superconducting phase in the final product (composite superconductor).
When creating the program, the importance of conducting research on the materials from which superconducting cables are created was taken into account, since if the ratio of niobium to bronze is incorrect, an explosion of wires is possible, and, consequently, human sacrifice, monetary costs and loss of time. This program allows you to determine the quality of the wires on the basis of chemical physical analysis of the object.
Program block diagram
Description of the stages of the study.
Stage 1. Sample preparation: cutting of a composite superconductor on an EDM machine; pressing the sample into the plastic matrix; polishing the sample to a mirror state; etching a sample to isolate niobium fibers on a bronze matrix. Samples of pressed composite superconducting samples were obtained;
Stage 2 Image acquisition: metallographic imaging on a scanning electron microscope.
Stage 3 Image processing: creating a tool for determining the volume fraction of the superconducting phase in a metallographic image; a set of statistically significant data on a specific type of sample. Created mathematical models of various tools for image processing; a software development was created to estimate the volume fraction of the superconducting phase; the program was facilitated by combining several mathematical functions into one; An average value of the volume fraction of niobium fibers in a bronze matrix of 24.7 ± 0.1% was obtained. A low deviation rate indicates a high repeatability of the composite wire structure.
Electron microscopic images of composite superconductors
Methods of image processing in the program.
- Identification - an individual instance of an object belonging to a class is recognized.
- Binarization is the process of converting a color (or in grayscale) image into a two-color black and white.
- Segmentation is the process of dividing a digital image into several segments (multiple pixels, also called superpixels).
- Erosion is a complex process in which the structural element passes through all the pixels of the image. If at some position each unit pixel of the structural element coincides with the unit pixel of the binary image, then the logical addition of the central pixel of the structural element with the corresponding pixel of the output image is performed.
- Dilatation is a convolution of an image or a selected area of an image with some kernel. The kernel can have any shape and size. In this case, the only leading position in the kernel is allocated, which is combined with the current pixel in the calculation of convolution.
Formulas of the program
Binarization formula (Otsu method):
Formula of erosion:
Dilatation formula:
Dilation and erosion pattern
Segmentation formulas by color thresholds:
Determining the magnitude of the gradient of brightness for each pixel of the image:
Threshold Calculation:
Used equipment
- CHMER GX-320L CNC - machine for EDM cutting samples
- SimpliMet 1000 - hot pressing machine
- AutoMet 250 Buehler - grinding and polishing machine
- Axio Scope A1 Carl Zeiss - optical microscope for quality control of thin sections
- Hitachi TM-1000 - scanning electron microscope for metallographic images
Program interface
Source: https://habr.com/ru/post/350918/
All Articles