Agro-robot with AI learned to carefully collect only ripe lettuce from the garden

Many types of plants and crops are still collected only by hand.

Engineers from the University of Cambridge (UK) have created a working prototype of a robotic crop picker that can independently recognize intact cabbages of lettuce, which are ready for cutting, as well as carefully process and assemble them.


In order to harvest a good crop in automatic mode, you need to solve several complex technological problems, such as:
')
- accurate and correct positioning at the gathering place, synchronization of actions with other collectors;

- analysis and selection of the right algorithms for use with different cultures;

- Recognition of ripened fruits (sheets, berries);

- utilization of damaged or not germinated elements;

- accurate capture and cutting of ready-to-harvest crops without damage;

- sorting and stacking the harvest;

- action logging and self-learning ability;

- checking the correctness of current actions and minimizing damage at the collection site.

In addition, it is necessary to perform all these tasks at high speed and in real, constantly changing conditions of the surrounding world (wind, rain, insects, animals, and so on).

A group of engineers at the University of Cambridge successfully solved some of these tasks and published their research in this work:

Link to a scientific article in the Journal of Field Robotics .

They have created a small and so far little autonomous prototype of an independent harvesting robot capable of analyzing and choosing the right elements for collection.

This robot-assembler is a manipulator on a wheeled platform with a width of one bed. The manipulator is equipped with a grip, a cutting knife and a video camera, an additional video camera is located at the top of its platform.

Since this is just a prototype, which for mass use will be significantly refined both elementally and functionally, the engineers simply installed all the robot components they need for conducting a field test on a passive wheeled platform, moved manually by the operator.

Their task was to teach the robot to understand the harvest process and automate its actions through machine learning.

After the robot is above the new heads of lettuce, the process of recognition and classification of heads of lettuce with video cameras and neural networks is activated.

Pictures from video cameras on the manipulator and platform were used for:

- localization of lettuce heads in front of the video camera - using the YOLO version 3 convolutional neural network, for which 1500 photos were used to train;

- classification of lettuce heads into four types (mature element, immature element, infected / damaged element, missing element) using the Darknet Object Classification Network convolutional neural network, for which 665 photos were used to teach.





Field tests of the robot-assembler showed that he is capable of:

- determine the heads of lettuce in front of you with an accuracy of 91%;

- correctly classify the heads of lettuce with an accuracy of 85%.

- the time of one work cycle (the transition from one head of lettuce to another) is 32 seconds, which is a lot, but this moment will be refined.

It turns out that the beds themselves are very overgrown and it is very difficult to locate and classify the germination sites of lettuce just like that:



Therefore, the salad is cleaned in manual mode quickly, but without success and special selection:



Yes, and to create a small platform with a robot cleaner is difficult, because the device must be mobile, fairly stable to stay on the surface on loose soil and not destroy the beds with its components.

Here are the size restrictions for the elements and parameters of the prototype of the robot-assembler:



What was the result of the field tests after its assembly:



By the way, the platform with the robot was called Vegebot .

The block diagram of the relationship components of the platform Vegebot:



Moreover, both video cameras in this solution are the simplest and cheapest USB webcams.

The software part of Vegebot (architecture and web interface), used by Robot Operating System (ROS):





Diagrams of cycles of work of Vegebot (cycles of training, collecting):



Preparation for field trials, study of beds and analysis of pictures of heads of lettuce from different heights of the camera:



But what a string-strap for a laptop! Engineering savvy in business!



Table with different types of photos for localization of lettuce heads in front of the video camera (for training the convolutional neural network YOLO version 3):



A table with different types of photographs for classifying the heads of lettuce into four types (mature element, immature element, infected element, absence of element) for training a Darknet Object Classification Network:





Intermediate options for mechanisms and elements Vegebot:



Double element for collecting heads of lettuce (grip and knife):



Rotary DC motor (left), linear drive with a knife mechanism (right):



Pneumatic cutter:



The final version of the mechanism of capture and harvesting with a belt drive and dual pneumatic drive system:



Characteristics and sizes of the mechanism of capture Vegebot:



The principle of operation of the Vegebot capture mechanism (T - force threshold, FR - ground reaction forces):



Positioning the capture mechanism Vegebot using two on-board video cameras:



Compensation of external lighting factors and image parameters from video cameras for optimal localization of the heads of lettuce:





Examples of localization (recognition) of the heads of lettuce on images from video cameras at different heights and from different angles:







The obtained field data on the localization (recognition) of the heads of lettuce using Vegebot (based on 60 attempts of Vegebot to recognize the head of lettuce):



Image parameters and matrix for classifying heads of lettuce:



The running time and trajectory of the Vegebot capture mechanism:



During the field trials, there was a problem with determining the number of unnecessary (protruding) leaves on the head of lettuce after harvest, which must be removed.

Here are some options here: ideally cut a head of lettuce without protruding leaves, head of lettuce with one protruding leaf, head of lettuce with two protruding sheets, head of lettuce with damaged sheets.



Depending on the number of extra leaves on the head of the lettuce and the need for their recognition and removal, the working cycle of Vegebot also changes:



Summary data on field trials of the prototype of the harvest robot Vegebot:



Of course, this prototype is still far from industrial agrarian use, but this is only the beginning of research.