Uncomplicated SCRUM and Image Segmentation for Highlighting Post-it Stickers

"The inspiration I’ve been looking for all morning
caught up in the most unfortunate moment.
And how to explain that I'm leaving for SCRUM?
- Come with me ?! "

Communication in a team is the dire need of big projects. It should not look like a scaffold or compulsory assembly of alcoholics anonymous. From the team you need to participate, you need a shine in the eyes, everyone should be torn from the desire to speak, as from the pathos of this proposal. Gradually, our team evolved into a SCRUM model, thanks in large part to simple and intuitive stickers. What kind of SCRUM without stickers? Almost everyone had stickers in childhood and, somewhere deep in the subconscious, memories came to us, when we, as a child, were taught to glue by the teacher and, if the sticker was glued evenly, as a reward, it didn’t take it. But even in our carefree childhood, we had to do things that seemed dull and incomprehensible to us — to remove toys, to scrub a wall from a pen, or to write under dictation. Having matured, we have a choice - we can shift the work to others. And who wants to write backlog (report) for everyone and then transfer the data to Jira? Using Jira directly in the rally process removes the participant from the discussion, therefore, after the adoption of the UN convention on the abolition of slavery, it remains to shift this task to the robots.

As a result, the idea was born to write a program for recognizing and tracking task cards on the SCRUM board.

Setting the minimum task is seen as:
')

• read the image of the SCRUM board;
• select stickers;
• save images of stickers;
• determine the area of ​​the board where the stickers are located;
• determine which task the sticker belongs to;
• create a file with information about the status of tasks.

Actually, the task looks quite elegant and simple at the same time, and we offered it this year as an educational task during our summer school on programming. This article proposes to consider the implementation of the first three requirements.

With the digitization of the image of the board, the rule is simple - the last person to get up from a chair photographs the board. In the future, it should be replaced by a robot.

Below is a picture of what a simplified SCRUM board might look like. Very simplified.


Fig.1. Sample SCRUM-boards.

Stickers on top for unselected tasks, below the developer tasks area (blue, green and red). The area of ​​each developer is divided into two parts - on the left are the tasks that are performed, on the right - completed.

Segmentation

Let's start with the banalities - loading the original image with a photo of the board is done using OpenCV tools:

int main(int argc, char** argv) { vector<cv::Mat> stickers; cv::CommandLineParser parser( argc, argv, keys ); String image_path = parser.get<String>( 0 ); if( image_path.empty() ) { help(); return -1; } cv::Mat image = cv::imread(image_path); 

To represent images in OpenCV, the class cv :: Mat is used. This is an interesting data structure and you can find some details about this class.

Next, you need the main function to highlight sticker images:

  recognizeStickers(stickers); 

In the first version, we simply save the found stickers into files named sticker1.jpg ... stickerN.jpg:

  saveStickers(stickers); } 

Let us consider in more detail the function of selecting images of stickers. The function prototype may be:

 void recognizeStickers(vector<cv::Mat> &stickers); 

The algorithm for solving the problem of selecting contrast objects on a uniform background can be implemented in various ways:

• Algorithm 1. Selecting objects (stickers) that have a specified color using the inRange function;
• Algorithm 2. Selection of bright objects (stickers) from the S-channel of an HSV image using the threshold function;

Algorithm 1

Selecting stickers with inRange can be as follows:

• determine the range of colors characteristic of the sticker (in the first implementation, we define the range explicitly, by constants);
• highlight points whose color differs from the background color using stepwise conversion;
• using the filter to combine the points of the intended sticker to obtain a solid image;
• highlight the boundaries of continuous areas;
• define the boundaries vertically and horizontally for the groups of points of the supposed stickers.

Below is a rough sketch illustrating the idea of ​​the algorithm:

 void recognizeStickersByRange(cv::Mat image,std::vector<cv::Mat> &stickers) { cv::Mat imageHsv; std::vector< std::vector<cv::Point> > contours; //   hsv,      cv::cvtColor(image, imageHsv, cv::COLOR_BGR2HSV); cv::Mat tmp_img(image.size(),CV_8U); //      cv::inRange(imageHsv, cv::Scalar(key_light-delta_light,key_sat-delta_sat,key_hue-delta_hue), cv::Scalar(key_light+delta_light,key_sat+delta_sat,key_hue+delta_hue), tmp_img); // ""      cv::dilate(tmp_img,tmp_img,cv::Mat(),cv::Point(-1,-1),3); cv::erode(tmp_img,tmp_img,cv::Mat(),cv::Point(-1,-1),1); //    cv::findContours(tmp_img,contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_NONE); for (uint i = 0; i<contours.size(); i++) { cv::Mat sticker; //      cv::Rect rect=cv::boundingRect(contours[i]); image(rect).copyTo(sticker); //     stickers.push_back(sticker); } } 

After a stepwise transformation and stretching of the area using a filter to combine the areas separated by noise (the cv :: dilate method), we get:


Fig.2. Image after binarization.

Send the binarized image to the input of the contour extraction algorithm v :: findConturs and for each contour we find the bounding rectangle with the help of cv :: boundingRect. For clarity, draw the bounding rectangles in green on the original image.

As a result of the selection of areas, we obtain successfully selected stickers. Below is the result of the algorithm on the test image.


Fig.3. Stickers are highlighted.

Knowing the parameters of the bounding rectangles, it is easy to cut and save images of stickers to disk as separate files:

 for (uint i = 0;i < stickers.size();i++) { cv::imwrite("sticker"+toString(i+1)+".jpg",stickers[i]); } 

As a result, sticker1.jpg ... stickerN.jpg files will be formed on the disk. An example of the contents of the sticker file is shown below:


Fig.4. An image of the highlighted sticker.

It should be noted that in the above example, we did not implement the algorithm for determining the color of the sticker, but set it with key_light, key_sat, key_hue constants in HSV-space, which is not good under normal conditions. And if suddenly the stickers are of a different color, the algorithm will need to be reconfigured. Boundary rectangles for developer areas (blue, green, red) are not highlighted. In principle, it is possible to set color constants, and for them to allocate a similar algorithm, which will automatically determine the boundaries of areas and determine the status of tasks.

Algorithm 2.

We use the function cv :: threshold, as shown in examples / 1 / and / 3 /. To begin with, we converted the input frame to HSV format using the cv :: cvtColor function, and the result was split using cv :: split. Result below:


Fig.5. H-channel image.

Fig.6. S-channel image.

Fig.7. V-channel image.

As can be seen from Figure 5 - Figure 7, the S-channel of the image is of the greatest interest for processing stickers on a white background, where the value of the color-saturated stickers will be maximum and the white background will be minimum. This is most clearly shown here . It can be assumed that using the cv :: threshold function with the correct value of the boundary, we obtain the desired binary image with selected stickers, from which stickers can be extracted using the cv :: findContours function, similar to algorithm 1.

  std::vector<cv::Mat> hsvPlanes; cv::split(inputHsvImage, hsvPlanes); cv::Mat image = hsvPlanes[1]; double thresh = 110; double maxValue = 255; threshold(image,image, thresh, maxValue, cv::THRESH_BINARY); 

In the above example, the value of the boundary at 110 leads to the desired result of binarization. As in the case of algorithm 1, we again encounter the need to choose the value of the boundary, which can be calculated by analyzing the image histogram.


Fig.8. Histogram for S channel image.

Since the stickers are light, the color of the sticker corresponds to the rightmost peak on the S-channel histogram. Defining its boundaries using the algorithm / 4 /, we obtain the desired boundary value for the stepwise transformation.

 int findMostRightExtremum(cv::Mat histNorm) { vector< float > data; for (int i=0; i<histNorm.rows; i++) data.push_back(histNorm.at<float>(i)); //   . Persistence1D p; p.RunPersistence(data); //     0,002. vector< TPairedExtrema > Extrema; p.GetPairedExtrema(Extrema, 0,002); sort(Extrema.begin(),Extrema.end(), [](const TPairedExtrema &a, const TPairedExtrema &b) -> bool { return (a.MaxIndex) > (b.MaxIndex); } ); //         return (Extrema[0].MinIndex)*(255/histNorm.rows); } 

As a result of binarization using the function cv :: threshold, we obtain:


Fig.9. Binarization with cv :: threshold for the calculated boundary.

As you can see, both stickers and colored boundary markers are highlighted, defining the boundaries of developer areas, which will allow you to highlight areas of the board for each of the developers.

This is the first article in a series of articles on the introduction of computer vision technology into the SCRUM process. The following tasks remained out of consideration:

• selection of board zones (“zone of unselected tasks”, areas “in progress” and “completed” for developers ”);
• determine the zone of the board in which the sticker is located;
• matching stickers after moving on the board;
• task text recognition;
• interaction with jira.

By the way, soon we will have a series of free webinars on programming in C ++ 11/14, with examples from the field of image processing and augmented reality.

Sources

1. Basic Thresholding Operations
2. Thresholding Operations using inRange
3. akaifi.imtqy.com/MultiObjectTrackingBasedhOnColor
4. Local Extremum Search Algorithm