What will happen if you mix nuts, Arduino, OpenCV and Delphi. Part 2
In the first part, I tried to select nuts without OpenCV, and was wrong.
Programming in Delphi from the institute, starting with version 2, although being fairly close familiar with other PLs, I nevertheless began to look for titles specifically for Delphi. And found .
Having compiled the EdgeDetect example, and seeing the results, I realized that OpenCV tool is really powerful, simple and fast. Thanks to good people for the pascal header files to the C interface of this wonderful library, because they gave me the opportunity to write in the environment of my usual RAD. Having decided on PL, I began to develop software from scratch, this article describes my victories and misadventures, and please, do not judge the pain, this is only my second article on Habré.
The first rake was connected with a rather tangible memory leak: they were connected with the fact that after each cvFindContours you need to call cvClearMemStorage.
Soon realizing that at 30 FPS, which gave out my Logitech C270, I could not detect the nuts in free fall, I began to look for high-speed cameras. For the experiments, the PS3 Eye Camera was acquired, giving out transcendental 187 FPS at 320x240. As a result, another feature was found - the draw limit of 65 FPS under Win7. As it turned out, cvWaitKey is limiting - a solution was immediately found, namely: to call cvWaitKey not with each processed frame, but with a smaller frequency.
I will describe the algorithm itself.
For each sample from the base, an “album” of rotated samples was generated in 10-degree increments. This makes it possible to store much less samples in the database of standards and not to waste resources on rotation “on the fly”. I implement the primitive perspective correction on the fly using cvResize.
As a result of the sliding of nuts along the grooves, they quickly stain these same grooves with fat, which is very rich. This fact prevents more accurate finding of the contours of nuts. I tried both simple cvThreshold and cvThreshold with cvCanny on top - it worked poorly on a dirty background. Plus, the shadow, which was thrown by the nuts, hindered when spans at a small distance from the background. To solve this problem, I came up with my filter. Its essence is that it replaces the most "non-colored" pixels with white pixels.
For nuts sliding on a white background, there is a contour. A mask is made from the contour, which allows you to copy each nut with transparency into an array of PIplImage. Too small and very large contours are skipped.
The frame is divided into regions-> lines, in reality, these are separate gutters, along which nuts slide. At the end of each line there is an actuator, which is a nozzle that controls the flow of air under pressure.
In the application, each line is serviced by a separate thread (thread). Inside the thread, we find the nut closest to the nozzle, and determine its “similarity” with the base of the reference samples. Below is a code section that considers “similarity” via cvAbsDiff:
The value of the wcount variable is the coefficient of similarity between the nut and the standard in “parrots”. When this value is exceeded above the threshold, we transmit the line number through a com port in arduino. The controller opens the nozzle for a specified time, than “blows” the nut, in the normal state, the nozzle is closed. For asynchronous operation of executive devices, the following sketch was written.
Nozzles are an electromagnetic solenoid. We switch this load according to the following scheme. Each nozzle needs a separate key.
And this is the assembled device:
At the request of the customer, I can not publish images of the finished device. I hope the next video will provide an opportunity to present the final device.
I tried to describe the most difficult and interesting moments that I met as a result of working on this interesting project. Feel free to ask questions if, in your opinion, something is not detailed enough.
Thanks for attention.
UPD: Added slowmotion video, 75 FPS -> 1 FPS
Programming in Delphi from the institute, starting with version 2, although being fairly close familiar with other PLs, I nevertheless began to look for titles specifically for Delphi. And found .
Having compiled the EdgeDetect example, and seeing the results, I realized that OpenCV tool is really powerful, simple and fast. Thanks to good people for the pascal header files to the C interface of this wonderful library, because they gave me the opportunity to write in the environment of my usual RAD. Having decided on PL, I began to develop software from scratch, this article describes my victories and misadventures, and please, do not judge the pain, this is only my second article on Habré.
The first rake was connected with a rather tangible memory leak: they were connected with the fact that after each cvFindContours you need to call cvClearMemStorage.
Soon realizing that at 30 FPS, which gave out my Logitech C270, I could not detect the nuts in free fall, I began to look for high-speed cameras. For the experiments, the PS3 Eye Camera was acquired, giving out transcendental 187 FPS at 320x240. As a result, another feature was found - the draw limit of 65 FPS under Win7. As it turned out, cvWaitKey is limiting - a solution was immediately found, namely: to call cvWaitKey not with each processed frame, but with a smaller frequency.
Show
if gettickcount-rendertickcount >= 33 then begin // 1000 / 33 = ~30 FPS //... rendertickcount := gettickcount; cc := cvWaitKey(1); end; I will describe the algorithm itself.
For each sample from the base, an “album” of rotated samples was generated in 10-degree increments. This makes it possible to store much less samples in the database of standards and not to waste resources on rotation “on the fly”. I implement the primitive perspective correction on the fly using cvResize.
Show
procedure createAlbum(nsIndex:integer); var i : integer; rot_mat: pCvMat; scale: Double; center: TcvPoint2D32f; width, height : integer; begin with nsamples[nsIndex] do begin width := nutimgs[0].width; height := nutimgs[0].height; center.x := width div 2; center.y := height div 2; scale := 1; for i:= 1 to 35 do begin nutimgs[i].width := width; nutimgs[i].height := height; rot_mat := cvCreateMat(2, 3, CV_32FC1); cv2DRotationMatrix(center, i * 10, scale, rot_mat); cvWarpAffine(nutimgs[0], nutimgs[i], rot_mat, CV_INTER_LINEAR or CV_WARP_FILL_OUTLIERS, cvScalarAll(0)); cvReleaseMat(rot_mat); end; end; end; As a result of the sliding of nuts along the grooves, they quickly stain these same grooves with fat, which is very rich. This fact prevents more accurate finding of the contours of nuts. I tried both simple cvThreshold and cvThreshold with cvCanny on top - it worked poorly on a dirty background. Plus, the shadow, which was thrown by the nuts, hindered when spans at a small distance from the background. To solve this problem, I came up with my filter. Its essence is that it replaces the most "non-colored" pixels with white pixels.
Show
procedure removeBack(var img: PIplImage; k:integer); var x, y :integer; sat: byte; framesize :integer; begin cvcvtColor(img, hsv, CV_BGR2HSV); x := 1; framesize := img.width * img.height * 3; while x <= framesize do begin sat := hsv.imageData[x]; if sat < k then begin hsv.imageData[x-1] := 255; hsv.imageData[x+1] := 255; hsv.imageData[x] := 0; end; inc(x ,3); end; cvcvtColor(hsv, img, CV_HSV2BGR); end; For nuts sliding on a white background, there is a contour. A mask is made from the contour, which allows you to copy each nut with transparency into an array of PIplImage. Too small and very large contours are skipped.
Show
frame := cvQueryFrame(capture); cvCopy(frame, oframe); cvCvtColor(frame, gframe, CV_BGR2GRAY); cvThreshold(gframe, gframe, LowThreshVal, HighThreshVal, CV_THRESH_BINARY_INV); cvFindContours(gframe, storage, @contours, SizeOf(TCvContour), CV_RETR_EXTERNAL, CV_CHAIN_APPROX_NONE, cvPoint(0, 0)); b := contours; NutIndex := 0; while b <> nil do begin asize := cvContourArea(b, CV_WHOLE_SEQ); if ((asize > tbminObjSize) and (asize < tbmaxObjSize)) then begin _rect := cvBoundingRect(b); cvZero(mask); cvDrawContours(mask, b, CV_RGB(255, 0, 255), CV_RGB(255, 255, 0), -1, CV_FILLED, 1, cvPoint(0, 0)); snuts[nutIndex].snut.width := _rect.width; snuts[nutIndex].snut.height := _rect.height; cvSetImageROI(oframe, _rect); cvSetImageROI(mask, _rect); cvZero(snuts[nutIndex].snut); cvCopy(oframe, snuts[nutIndex].snut, mask); cvResetImageROI(oframe); cvResetImageROI(mask); snuts[NutIndex].rect := _rect; inc(NutIndex); end; b := b.h_next; end; The frame is divided into regions-> lines, in reality, these are separate gutters, along which nuts slide. At the end of each line there is an actuator, which is a nozzle that controls the flow of air under pressure.
In the application, each line is serviced by a separate thread (thread). Inside the thread, we find the nut closest to the nozzle, and determine its “similarity” with the base of the reference samples. Below is a code section that considers “similarity” via cvAbsDiff:
Show
cvAbsDiff(tnut, nsamples[tp1].nutimgs[angle], matchres); cvCvtColor(matchres, gmatchres, CV_BGR2GRAY); cvThreshold(gmatchres, gmatchres, tbminTreshM, 255, 0); wcount := cvCountNonZero(gmatchres); The value of the wcount variable is the coefficient of similarity between the nut and the standard in “parrots”. When this value is exceeded above the threshold, we transmit the line number through a com port in arduino. The controller opens the nozzle for a specified time, than “blows” the nut, in the normal state, the nozzle is closed. For asynchronous operation of executive devices, the following sketch was written.
Show
int timeout = 75; int comm; unsigned long timeStamps[8]; int ePins[] = {2, 3, 4, 5, 6, 7, 8, 9}; void setup() { for (int i=0; i <= 7; i++){ pinMode(ePins[i], OUTPUT); } Serial.begin(9600); while (!Serial) { ; // wait for serial port to connect. Needed for Leonardo only } } void loop() { if (Serial.available() > 0) { comm = Serial.read(); if (comm >= 0 && comm <= 7) { digitalWrite(ePins[comm], HIGH); timeStamps[comm] = millis(); } if (comm == 66) { Serial.write(103); // for device autodetection, 103 means version 1.03 } } for (int i=0; i <= 7; i++){ if (millis() - timeStamps[i] >= timeout) { digitalWrite(ePins[i], LOW); } } } Nozzles are an electromagnetic solenoid. We switch this load according to the following scheme. Each nozzle needs a separate key.
Show
And this is the assembled device:
Show
At the request of the customer, I can not publish images of the finished device. I hope the next video will provide an opportunity to present the final device.
I tried to describe the most difficult and interesting moments that I met as a result of working on this interesting project. Feel free to ask questions if, in your opinion, something is not detailed enough.
Thanks for attention.
UPD: Added slowmotion video, 75 FPS -> 1 FPS
')
Source: https://habr.com/ru/post/238783/
All Articles