Winning Kinect on Linux
This is the second article in a series on developing C ++ applications that work with Microsoft Kinect. This part will discuss how to make a device work in Linux and how it can be used in your applications.
The first article on Kinect development can be read here . I strongly recommend reading the first part, for without it the impressions of the second will be incomplete.
As already mentioned in the first article, in addition to the official Microsoft SDK, there are several other third-party libraries:
Alternative SDK, unlike the official, can be used for development under Linux and Mac OS.
The easiest to use is libfreenect (OpenNI, compiled and installed according to official documentation for me, for example, did not want to work in the latest Ubuntu, it is quite possible that the driver for Kinect requires an unstable version of the OpenNI platform, and an unstable is so unstable) . Everything that is said in the official documentation is true and yes, it works.
')
It is gratifying that for Ubuntu libfreenect is available as binary packages. This allows you to save a lot of time during installation - no need to collect and configure anything, just install the dependencies and the library itself. Examples work out of the box.
But the Indians are not looking for easy ways and therefore we will consider the process of building a library from source. What is good this way? That allows you to get the source code of the examples that come with the library, and figure out what's what, and also, if necessary, experiment with the source code of the examples or debug if you have questions about how this all works from the inside.
So the build process:
For Ubuntu x64, the last line will be
After that, you can drive on the build / bin folder - they illustrate quite well the library's capabilities.
Binary packages are also available for wxWidgets in Ubuntu (but only for the stable 2.8.x branch, which, however, is quite enough for work). Install the libwxgtk2.8-dev package and the issue is resolved.
Qt had a good motto: Write Once, Run Anywhere . It would be nice to organize the same, but with wxWidgets. But the situation is such that in addition to the source code, we still have project files (after the first part we got projects from Visual Studio, but there’s little use in Linux) and before writing the platform-specific part of the code for the new OS, it would be nice automate the creation and customization of project files. This will help us utility CMake .
What is CMake good for? First, Linux is all different. If some thirdparty library on your machine is on the same path, then it’s not a fact that all your colleagues or users will have it there. Even when using the pkg-config output, in the case of changing some of the build parameters, there will be a lot of places in projects written by hand where new parameters will have to be written by hand. By automating all this minor work through CMake, which creates project files already configured for a specific working machine, we can also save a lot of time.
With Windows, the story is the same - why prescribe all parameters with your hands if CMake itself can search for a fairly large number of different libraries, if they are installed in the system, and then generate projects with already configured parameters.
In addition to actually generating projects, in the script for CMake, you can specify conditional parameters for each platform, for example, include / exclude some files in a project for a particular OS, add or remove dependencies on thirdparty libraries specific for a particular OS, etc.
In any case, the topic of using CMake is worthy of a separate article. In our case, in order not to overload the text with listings of CMake scripts, I will provide links to them in the repository:
Let's summarize what we got from last time:
Well, remember everything, now you can move on to conquering libfreenect under Linux.
Before you start using the API to access Kinect devices, you must call the
After completing work with devices, it is necessary to deinitialize the library context using the
In our wrapper library, the initialization and deinitialization of the library context is handled by the
To get the number of available devices in libfreenect, use the
Unfortunately, libfreenect does not contain an API to get the unique name of a separate device, therefore, in order to somehow identify the devices in our application, we will have to come up with a workaround, for example, to produce some string value containing the device index, for example:
There are two ways to capture images from Kinect in the libfreenect library - using the synchronous or asynchronous API variant. In the case of the asynchronous variant, when a new frame arrives, the callback function will be called, in which the incoming buffer can be processed. The memory for the buffer can be allocated independently or the library itself can take care of creating an internal buffer and clearing the memory upon completion.
To start capturing images, you must call the function
To complete the image capture, call the
Although the API is “asynchronous”, it’s still impossible to do without creating a control flow manually. From time to time, you need to call the
In order to start receiving frames with a depth buffer, you need to call several functions that configure the device operation mode:
The freenect_resolution values ​​can be:
And freenect_depth_format are as follows:
As can be seen from the description, in libfreenect, in contrast to the official SDK from Microsoft, only 11 bits are used for the depth value, not 12. There is also no possibility of getting a player index.
In principle, the device configuration algorithm for obtaining a color image is similar to the algorithm when obtaining the depth buffer, only the functions
After all the above, let's look at the image capture initialization code:
Because the data format for the depth buffer is slightly different, and the function of converting the depth value to grayscale has also changed a bit:
As a result of all the actions described, a new grabber class was created, which works quite well in Linux. Unfortunately, there is no API in libfreenect to get the position of players, for this purpose you will have to use third-party (or your own) developments, for example, based on OpenCV. But the fact that there is enough for recognition, for example, movement.
I think we’ll finish this with theory. The next part will be practice.
I remind once again that all the source code of the sample and the wrapper library for working with Kinect can be downloaded here .
You can generate project files for Windows using the build / cm.bat script , for Linux, build / cmLinux.sh
Also, I will not refuse to help taming OpenNI for Linux and Mac OS.
The first article on Kinect development can be read here . I strongly recommend reading the first part, for without it the impressions of the second will be incomplete.
Introduction
As already mentioned in the first article, in addition to the official Microsoft SDK, there are several other third-party libraries:
Alternative SDK, unlike the official, can be used for development under Linux and Mac OS.
The easiest to use is libfreenect (OpenNI, compiled and installed according to official documentation for me, for example, did not want to work in the latest Ubuntu, it is quite possible that the driver for Kinect requires an unstable version of the OpenNI platform, and an unstable is so unstable) . Everything that is said in the official documentation is true and yes, it works.
')
Install libfreenect
It is gratifying that for Ubuntu libfreenect is available as binary packages. This allows you to save a lot of time during installation - no need to collect and configure anything, just install the dependencies and the library itself. Examples work out of the box.
But the Indians are not looking for easy ways and therefore we will consider the process of building a library from source. What is good this way? That allows you to get the source code of the examples that come with the library, and figure out what's what, and also, if necessary, experiment with the source code of the examples or debug if you have questions about how this all works from the inside.
So the build process:
sudo apt-get install git-core cmake libglut3-dev pkg-config build-essential libxmu-dev libxi-dev libusb-1.0-0-dev
git clone github.com/OpenKinect/libfreenect.git
cd libfreenect
mkdir build
cd build
cmake ..
make
sudo make install
sudo ldconfig /usr/local/lib/
sudo glview
For Ubuntu x64, the last line will be
/usr/local/lib64/ .After that, you can drive on the build / bin folder - they illustrate quite well the library's capabilities.
Installing wxWidgets on Linux
Binary packages are also available for wxWidgets in Ubuntu (but only for the stable 2.8.x branch, which, however, is quite enough for work). Install the libwxgtk2.8-dev package and the issue is resolved.
How not to do one job twice
Qt had a good motto: Write Once, Run Anywhere . It would be nice to organize the same, but with wxWidgets. But the situation is such that in addition to the source code, we still have project files (after the first part we got projects from Visual Studio, but there’s little use in Linux) and before writing the platform-specific part of the code for the new OS, it would be nice automate the creation and customization of project files. This will help us utility CMake .
What is CMake good for? First, Linux is all different. If some thirdparty library on your machine is on the same path, then it’s not a fact that all your colleagues or users will have it there. Even when using the pkg-config output, in the case of changing some of the build parameters, there will be a lot of places in projects written by hand where new parameters will have to be written by hand. By automating all this minor work through CMake, which creates project files already configured for a specific working machine, we can also save a lot of time.
With Windows, the story is the same - why prescribe all parameters with your hands if CMake itself can search for a fairly large number of different libraries, if they are installed in the system, and then generate projects with already configured parameters.
In addition to actually generating projects, in the script for CMake, you can specify conditional parameters for each platform, for example, include / exclude some files in a project for a particular OS, add or remove dependencies on thirdparty libraries specific for a particular OS, etc.
In any case, the topic of using CMake is worthy of a separate article. In our case, in order not to overload the text with listings of CMake scripts, I will provide links to them in the repository:
- The main script, which contains general settings and a list of projects in the solution.
- Script to create a static library wxKinectHelper
- Script for creating a test application project
Let's summarize what we got from last time:
- The solution is divided into two projects: the wxKinectHelper static library and the KinectTest test application.
- Project files (Visual Studio for Windows and Makefile / CodeBlocks for Linux) are generated using the CMake utility.
- In the wxKinectHelper project, there is a wxKinectHelper class that can return the number of Kinect devices in the system, give the device name by its index, and also create a grabber class
- The grabber class can capture images (depth, color) from the camera of the device, as well as get the position of players
- When you receive a new frame or information about the position of the players, an event is triggered to which you can subscribe, for example, the form class
- For drawing players, the wxKinectHelper class is used, which uses a separate implementation class for each platform, thereby implementing the bridge pattern.
- The test application consists of one form and can display a list of devices, a picture from the depth sensor, a color picture from the camera, and the position of the players
Well, remember everything, now you can move on to conquering libfreenect under Linux.
Coding
Library initialization and deinitialization
Before you start using the API to access Kinect devices, you must call the
freenect_init() method and pass the variable address as a parameter to it, which will contain the pointer to the initialized library context, in case of successful completion of the function.FREENECTAPI int freenect_init(
freenect_context **ctx,
freenect_usb_context *usb_ctx);After completing work with devices, it is necessary to deinitialize the library context using the
freenect_shutdown() function:FREENECTAPI int freenect_shutdown(freenect_context *ctx);In our wrapper library, the initialization and deinitialization of the library context is handled by the
wxKinectHelper class, for which, after switching to Linux, it was also necessary to implement the implementation classes separately for work on Windows and Linux.Getting a list of devices
To get the number of available devices in libfreenect, use the
freenect_num_devices() function:FREENECTAPI int freenect_num_devices(freenect_context *ctx);Unfortunately, libfreenect does not contain an API to get the unique name of a separate device, therefore, in order to somehow identify the devices in our application, we will have to come up with a workaround, for example, to produce some string value containing the device index, for example:
wxString KinectHelperImplFreenect::GetDeviceName(size_t index)
{
wxString name = wxT( "Unknown Kinect Sensor" );
if (m_Context)
{
name = wxString::Format(wxT( "Kinect %u" ), index);
}
return name;
}Image Capture with Kinect
There are two ways to capture images from Kinect in the libfreenect library - using the synchronous or asynchronous API variant. In the case of the asynchronous variant, when a new frame arrives, the callback function will be called, in which the incoming buffer can be processed. The memory for the buffer can be allocated independently or the library itself can take care of creating an internal buffer and clearing the memory upon completion.
To start capturing images, you must call the function
freenect_open_device() , which takes as parameters:- Pointer to library context
- Pointer to a variable that will contain the device context, in case of successful completion of the function
- device index
FREENECTAPI int freenect_open_device(
freenect_context *ctx,
freenect_device **dev, int index);To complete the image capture, call the
freenect_close_device() function.FREENECTAPI int freenect_close_device(freenect_device *dev);Although the API is “asynchronous”, it’s still impossible to do without creating a control flow manually. From time to time, you need to call the
freenect_process_events() function, which will handle the event queue from libusb. In a wxWidgets application, an event flow can be organized like this:wxThread::ExitCode KinectGrabberFreenect::Entry()
{
int status(0);
while (!GetThread()->TestDestroy() && status >= 0)
{
status = freenect_process_events(m_Context);
}
return NULL;
}Getting depth buffer
In order to start receiving frames with a depth buffer, you need to call several functions that configure the device operation mode:
- Using
freenect_set_depth_callback()you can set a function that will be called when a new frame arrives with a depth buffer. - Using
freenect_set_depth_buffer()you can set a memory area in which libfreenect will write depth data from the device (this is an optional step, but using this function can be convenient if some post-processing is required) - Using the
freenect_set_depth_mode()function, you need to set the device operation mode. The parameters of this function are passed to the freenect_frame_mode structure, which contains information about the desired image size and data format.
The freenect_resolution values ​​can be:
typedef enum {
FREENECT_RESOLUTION_LOW = 0, /**< QVGA - 320x240 */
FREENECT_RESOLUTION_MEDIUM = 1, /**< VGA - 640x480 */
FREENECT_RESOLUTION_HIGH = 2, /**< SXGA - 1280x1024 */
FREENECT_RESOLUTION_DUMMY = 2147483647, /**< Dummy value to force enum to be 32 bits wide */
} freenect_resolution;And freenect_depth_format are as follows:
typedef enum {
FREENECT_DEPTH_11BIT = 0, /**< 11 bit depth information in one uint16_t/pixel */
FREENECT_DEPTH_10BIT = 1, /**< 10 bit depth information in one uint16_t/pixel */
FREENECT_DEPTH_11BIT_PACKED = 2, /**< 11 bit packed depth information */
FREENECT_DEPTH_10BIT_PACKED = 3, /**< 10 bit packed depth information */
FREENECT_DEPTH_DUMMY = 2147483647, /**< Dummy value to force enum to be 32 bits wide */
} freenect_depth_format;As can be seen from the description, in libfreenect, in contrast to the official SDK from Microsoft, only 11 bits are used for the depth value, not 12. There is also no possibility of getting a player index.
Getting a color image
In principle, the device configuration algorithm for obtaining a color image is similar to the algorithm when obtaining the depth buffer, only the functions
freenect_set_video_callback() , freenect_set_video_buffer() and freenect_set_video_mode() are used, respectively.Some more general information about capturing images
After all the above, let's look at the image capture initialization code:
if (freenect_open_device(m_Context, &m_Device,
( int )m_DeviceIndex) < 0) break ;
freenect_set_depth_callback(m_Device, KinectGrabberFreenect::DepthCallback);
freenect_set_video_callback(m_Device, KinectGrabberFreenect::VideoCallback);
freenect_set_video_mode(m_Device, m_VideoMode);
freenect_set_depth_mode(m_Device, m_DepthMode);
m_VideoBufferLength = m_VideoMode.bytes;
m_DepthBufferLength = m_DepthFrameSize.GetWidth() * m_DepthFrameSize.GetHeight() * 3;
m_VideoBuffer = new unsigned char [m_VideoBufferLength];
m_DepthBufferIndex = 0;
m_GotDepth = false ;
m_DepthBuffers[0] = new unsigned char [
m_DepthFrameSize.GetWidth() *
m_DepthFrameSize.GetHeight() * 3];
m_DepthBuffers[1] = new unsigned char [
m_DepthFrameSize.GetWidth() *
m_DepthFrameSize.GetHeight() * 3];
freenect_set_video_buffer(m_Device, m_VideoBuffer);
DeviceHash[m_Device] = this ;
freenect_start_video(m_Device);
freenect_start_depth(m_Device);Because the data format for the depth buffer is slightly different, and the function of converting the depth value to grayscale has also changed a bit:
unsigned char KinectGrabberFreenect::GetDepthValue(unsigned short data)
{
// 0x7ff is 0000011111111111 in binary format - max value for 11bit depth;
return ( double )255 * ( double )data / ( double )0x7ff;
}As a result of all the actions described, a new grabber class was created, which works quite well in Linux. Unfortunately, there is no API in libfreenect to get the position of players, for this purpose you will have to use third-party (or your own) developments, for example, based on OpenCV. But the fact that there is enough for recognition, for example, movement.
In conclusion
I think we’ll finish this with theory. The next part will be practice.
I remind once again that all the source code of the sample and the wrapper library for working with Kinect can be downloaded here .
You can generate project files for Windows using the build / cm.bat script , for Linux, build / cmLinux.sh
Also, I will not refuse to help taming OpenNI for Linux and Mac OS.
Source: https://habr.com/ru/post/124009/
All Articles