Robotic arm control with Intel RealSense cameras

Roymotion 's Roy robotic arm was created as a result of a successfully funded project on the Kickstarter portal , launched in 2012. The description read: "The project is to create a moving character the size of a person using only mechanical parts produced by laser cutting and servo drives available for sale at hobby and modeling stores." In this experiment, Roy's hand management software was developed using the Intel® RealSense ™ SDK for Windows * . The management uses hand-tracking APIs that are available as part of the SDK.

The code for this project was developed in C # / XAML using Microsoft Visual Studio * Community 2015. The software can use the Intel RealSense F200 and SR300 cameras (coming soon). To see the robotic arm controlled by the software, in action, watch the following video on YouTube *.

About hand roy

The Roy Arm Assembly Kit is currently available on the Roemotion, Inc. website. in the form of an assembly kit. The kit includes:

• parts made by laser cutting;
• all necessary fasteners;
• 8 servos (the same as those used in managed models);
• 6 extension cables for servo drives.

As stated on the Roemotion website, this kit does not include electronics for control. This is due to the fact that initially the meaning of the project was to provide people with interesting mechanical systems for use with any controllers. In the experiment itself, a third-party servo controller is used to control the motors in the robotic arm (Fig. 1).
')

Figure 1. Roy's robotic arm

There are 6 servomotors installed in the hand: one for the index finger, middle finger, ring finger and little finger and two for the thumb. (Note. Two more servomotors are installed at the base of the arm to control the movement of the wrist, but in this experiment we do not control them.)

Hand Tracking APIs in the Intel® RealSense ™ SDK

As stated in the Intel RealSense SDK documentation , the hand tracking module provides three-dimensional hand tracking in real time and can track one or two hands, providing data on the exact location of all joints. In the framework of this experiment in real-time device management, of particular interest to us are the values ​​of the bend of fingers, obtained using calls to the QueryFingerData () method.

Control electronics

In this experiment, we used the Pololu Micro Maestro * 6-channel USB servo controller (fig. 2) to control six motors installed in the Roy hand. This device includes a rather complicated SDK for developing control applications designed for different platforms using different programming languages.


Figure 2. Servo Pololu Micro Maestro * Servo Controller

Servo controller setting

Before developing our own software for direct control of the robotic arm in this experiment, it was first necessary to understand the range of motion of each finger from the point of view of servo control parameters. In advanced robotic servomechanisms with integrated controllers, you can interrogate the position code before applying the driving moment. Unlike such systems, in Roy’s hand, it is necessary to supply electric power to the motors with great care in order to avoid too abrupt and fast movements, due to which fingers can bend down and damage gears.

Fortunately, the Pololu Micro Maestro SDK includes the Control Center application, with which the user can adjust the settings at the firmware level and save them to flash memory on the controller board. The empirically determined parameters for this application are shown in Fig. 3


Figure 3. Pololu Maestro Control Center application

After setting the parameters for the Min and Max positions, the servo controller firmware will not allow the servomotors to go beyond the allowed range of movement. This is very important for use in this case, because in a robotic arm the mobility of all joints is limited by mechanical stops. If the joint moves as far as it will go while the servomotor does not stop, this can lead to overheating and burnout of the electric motor and damage to the gear.

Another important parameter in this case is called On startup or error. Thanks to him, all the fingers by default (and in case of errors) occupy a straightened position, which prevents the thumb and index fingers from hooking if they were bent.

Two more parameters to pay attention to are the Speed ​​and Acceleration parameters that control speed and acceleration. Using these parameters, you can make the movement smoother at the level of the microprogram. This approach is preferable to high-level filtering algorithms, because of which the delays increase and the load on the main software application increases.

Note. In more advanced robotic servomechanisms with built-in controllers, a proportional-integral-differential algorithm (PID-algorithm) is often used. It allows you to write parameters to the firmware in flash memory to fine-tune the feedback at the lowest level possible (that is, as close as possible to the equipment) in order to ensure a smooth change of the operating modes of the motors without loading the high-level software.

Own program for management

In this experiment, we developed our own program (Fig. 4) that uses many of the hand tracking functions available in the SDK code samples.


Figure 4. Own program for management

The user interface has real-time fingertip tracking data, but in this particular experiment we used the following three parameters to control Roy's hand:

• Data alerts.
• Flexion data.
• Scale data.

Alert data

Alerts are the most important information for tracking in the application controlling the device (such as in this case the robotic arm) in real time. It is very important to understand (and control!) How the device will behave when the set values ​​are inaccessible or unreliable.
In this experiment, the application tracks the following alert data:

• Hand detected.
• The hand is calibrated.
• The hand is inside the borders.

The software program used prevents the control of the servomechanisms of the robotic arm in the event of any of these alerts. In order for the program to control the robotic arm, the user's hand must be properly calibrated and within the range of the camera.
As shown in the code snippet below, the application cycles through the number of triggered alerts and sets three logical variables: detectionStatusOk , calibrationStatusOk, and borderStatusOk (note that handOutput is an instance of PXCMHandData ).

for (int i = 0; i < handOutput.QueryFiredAlertsNumber(); i++) { PXCMHandData.AlertData alertData; if (handOutput.QueryFiredAlertData(i, out alertData) != pxcmStatus.PXCM_STATUS_NO_ERROR) { continue; } switch (alertData.label) { case PXCMHandData.AlertType.ALERT_HAND_DETECTED: detectionAlert = "Hand Detected"; detectionStatusOk = true; break; case PXCMHandData.AlertType.ALERT_HAND_NOT_DETECTED: detectionAlert = "Hand Not Detected"; detectionStatusOk = false; break; case PXCMHandData.AlertType.ALERT_HAND_CALIBRATED: calibrationAlert = "Hand Calibrated"; calibrationStatusOk = true; break; case PXCMHandData.AlertType.ALERT_HAND_NOT_CALIBRATED: calibrationAlert = "Hand Not Calibrated"; calibrationStatusOk = false; break; case PXCMHandData.AlertType.ALERT_HAND_INSIDE_BORDERS: bordersAlert = "Hand Inside Borders"; borderStatusOk = true; break; case PXCMHandData.AlertType.ALERT_HAND_OUT_OF_BORDERS: bordersAlert = "Hand Out Of Borders"; borderStatusOk = false; break; } } 

Before any attempt by the program to control the servos of the hand, a check is carried out: all three variables, that is, the detectionStatusOk , calibrationStatusOk and borderStatusOk , must be set to True. If at any time one of these three flags is set to False, all fingers are moved to the default Open position for security.

Flexion data

The program developed in the course of this experiment calls the QueryFingerData () method, which returns the value of the finger fold and the radius of the finger tip. The bend value can be from 0 (the finger is fully bent) to 100 (the finger is straightened).
The application receives data about the flexing of each finger in the frame acquisition and release cycle, as shown in the following code snippet (where handData is an instance of PXCMHandData.IHand ).

 PXCMHandData.FingerData fingerData; handData.QueryFingerData(PXCMHandData.FingerType.FINGER_THUMB, out fingerData); thumbFoldeness = fingerData.foldedness; lblThumbFold.Content = string.Format("Thumb Fold: {0}", thumbFoldeness); handData.QueryFingerData(PXCMHandData.FingerType.FINGER_INDEX, out fingerData); indexFoldeness = fingerData.foldedness; lblIndexFold.Content = string.Format("Index Fold: {0}", indexFoldeness); handData.QueryFingerData(PXCMHandData.FingerType.FINGER_MIDDLE, out fingerData); middleFoldeness = fingerData.foldedness; lblMiddleFold.Content = string.Format("Middle Fold: {0}", middleFoldeness); handData.QueryFingerData(PXCMHandData.FingerType.FINGER_RING, out fingerData); ringFoldeness = fingerData.foldedness; lblRingFold.Content = string.Format("Ring Fold: {0}", ringFoldeness); handData.QueryFingerData(PXCMHandData.FingerType.FINGER_PINKY, out fingerData); pinkyFoldeness = fingerData.foldedness; lblPinkyFold.Content = string.Format("Pinky Fold: {0}", pinkyFoldeness); 

Scale data

After receiving data on the bending of each of the user's fingers, the scale equations are processed in order to compare the obtained values ​​with the full-scale ranges of motion of each robotic finger. All full-scale values ​​(i.e., the control pulse duration in milliseconds, which is required to move a finger to either fully extended or fully bent) are defined as constants in the class servo.cs.

 // Index finger public const int INDEX_OPEN = 1808; public const int INDEX_CLOSED = 800; public const int INDEX_DEFAULT = 1750; . . . 

Individual constants are determined for each finger of a robotic hand, they are compared with the parameters of the Min and Max positions of the servo drive, which were recorded in the flash memory of the Micro Maestro controller board (Fig. 3). Similarly, the program defines a full scale value for a range of finger bend data.

 int fingerMin = 0; int fingerMax = 100; 

Since the bend range of the fingers is the same for all fingers (that is, from 0 to 100), it is enough to set the range only once, after which it can be used for data scaling operations for all fingers, as shown below.

 // Index finger int indexScaled = Convert.ToInt32((Servo.INDEX_OPEN - Servo.INDEX_CLOSED) * (index - fingerMin) / (fingerMax - fingerMin) + Servo.INDEX_CLOSED); lblIndexScaled.Content = string.Format("Index Scaled: {0}", indexScaled); Hand.MoveFinger(Servo.HandJoint.Index, Convert.ToUInt16(indexScaled)); . . . 

Conclusion

To implement this program experiment, it took only a few hours after we tested the servos and understood the principle of controlling them. The classic Windows 10 application was developed in C # / XAML. It uses many of the features available in the APIs and in the Intel RealSense SDK code samples.

For more information about the Intel RealSense SDK for Windows, see the Intel website .