Digital Termenvox based on Leap Motion (powered by .NET)
Since the advent of the Kinect for Windows sensor, I have had a quiet passion for programming such kind of devices - this is a magical feeling when a program written by you responds to human movements from a distance! Today I will tell you about the programming of another similar device - Leap Motion .
In essence, Leap Motion is a small Kinect that can recognize the position of the hands. According to the creators (well reflected in this video ), it can be used to control a computer in many different scenarios. In this post, we will look at how Leap Motion is programmed (on the .NET platform), and how you can create a simple musical instrument, the theremin, on the basis of it. Here is this:
')
From the point of view of the programmer, Leap Motion provides a set of convenient APIs ( documented here ) that allow you to:
Unlike Kinect, Leap Motion does not provide access to 3D data obtained from the camera.
You can program Leap Motion in a variety of languages ​​and platforms, including .NET. In this article, we will look at using the first opportunity — accessing a hand model.
For our experiments in Visual Studio, we will create an empty WPF project and add a colored circle to the main screen. Our first step is to learn how to move this circle around the screen with your finger. So, the markup of the main page (XAML) will look like this:
To use Leap Motion, you need to download the Leap Motion SDK and connect the link to LeapCSharp.NET4.0.dll to the project. However, this is not all - Leap SDK also requires the Leap.dll and LeapCSharp.dll libraries , which should be located in the same directory as the project executable file. Therefore, locate these files in the LeapSDK \ lib \ x86 directory (in the directory where the Leap SDK is installed), drag them to the Visual Studio project and set the properties “actions on compilation = content, copy to the output directory = copy if newer”.
All communication with the Leap controller is conducted through the Controller object. Create such an object in our main file MainWindows.xaml.cs:
Now Leap Motion is ready to provide us with images of hand position data. You can receive frames in two ways:
In our case, to create a controller poll cycle, we use DispatcherTimer, which will be triggered 30 times per second:
The dt_Tick method will focus the basic logic of motion tracking. We get the current frame (Frame), and if the fingers are visible in this frame, we take the first one of them and use its coordinates to change the position of the circle:
We have achieved that we can move the circle on the screen by moving your finger over the Leap Motion controller. Sample code at this stage can be viewed on GitHub .
For software generation of sound of different heights, we use the NAudio library. The easiest way to add NAudio support to a project is to use Nuget by typing package manager in the console:
The process of generating a variable pitch sound is described in detail in the article by Charles Petzold on MSDN . In short, we create a class that generates a sine wave of sound, and pass it to the input to the NAudio method, which reproduces this wave. In this case, there are a number of subtleties: in order for the sound to change more smoothly, it is necessary to smoothly change the frequency of the sound, and at the same time take into account the delay in audio buffering. All these subtleties are taken into account in the class PortamentoSineWaveOscillator.cs , which must be placed in the project.
To generate a sound, we create an object of type WaveOut and initialize it, passing an instance of the oscillator we created:
We carry out initialization in the page constructor:
Now we just need to change the Osc.Pitch property to change the pitch accordingly, and Osc.Amplitude to change the volume. Let the horizontal movement of the fingers are responsible for the volume, and the vertical - for the pitch. The coefficients are chosen empirically to achieve a comfortable sound:
A version of the project at this stage can be obtained here .
To make the project a little more beautiful, instead of the usual circle moving around the screen, you can add special effects based on a particle system (Particle Systems). I took this article as a basis on CodeProject , where the source code of the WPF control, which is a source of particles, is given. Replacing them with the circle added in the first step, we get the final project, the source code of which is available on GitHub .
If you want to continue the experiments, the scope for creativity is almost unlimited! In particular, you can:
If you are interested in this topic (write in the comments), then I will continue the series of articles, adding various additional interesting things related to Leap Motion, Kinect and the unexpected use of mobile devices. What would you like to hear? For example, this video shows what else you can do with Leap Motion in a couple of hours:
In essence, Leap Motion is a small Kinect that can recognize the position of the hands. According to the creators (well reflected in this video ), it can be used to control a computer in many different scenarios. In this post, we will look at how Leap Motion is programmed (on the .NET platform), and how you can create a simple musical instrument, the theremin, on the basis of it. Here is this:
')
Leap Motion for the programmer
From the point of view of the programmer, Leap Motion provides a set of convenient APIs ( documented here ) that allow you to:
- With a high frequency (more than 60-100 frames per second) get a model of space, which includes the coordinates and speeds of fingers, normal palms, position and speed of various “tools” (Leap Motion well recognizes pencils and pens)
- Recognize basic gestures: circle, swipe, press
- Emulate the touch surface, i.e. touching some virtual surface in front of the screen
- Recognize the basic movements of the hands and fingers (motions), translating them into transformations of rotation, movement or scaling.
Unlike Kinect, Leap Motion does not provide access to 3D data obtained from the camera.
You can program Leap Motion in a variety of languages ​​and platforms, including .NET. In this article, we will look at using the first opportunity — accessing a hand model.
We track the simplest movements
For our experiments in Visual Studio, we will create an empty WPF project and add a colored circle to the main screen. Our first step is to learn how to move this circle around the screen with your finger. So, the markup of the main page (XAML) will look like this:
WPF-markup page MainPage.xaml
<Window x:Class="LeapVoxSample.MainWindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" Title="MainWindow" Height="768" Width="1024" Background="Black"> <Canvas> <Ellipse x:Name="ptr" Fill="Yellow" Height="30" Width="30"/> </Canvas> </Window> To use Leap Motion, you need to download the Leap Motion SDK and connect the link to LeapCSharp.NET4.0.dll to the project. However, this is not all - Leap SDK also requires the Leap.dll and LeapCSharp.dll libraries , which should be located in the same directory as the project executable file. Therefore, locate these files in the LeapSDK \ lib \ x86 directory (in the directory where the Leap SDK is installed), drag them to the Visual Studio project and set the properties “actions on compilation = content, copy to the output directory = copy if newer”.
All communication with the Leap controller is conducted through the Controller object. Create such an object in our main file MainWindows.xaml.cs:
Controller Leap = new Controller();
Now Leap Motion is ready to provide us with images of hand position data. You can receive frames in two ways:
- Pass an instance of the Listener class to the Controller object, in which the OnFrame method will be overloaded. This method will be automatically invoked by the Leap Motion SDK at the highest possible frame rate.
- Create your own method that will periodically poll the Controller class and receive the current frame from it. This should be done, there is already a natural loop in the program, or if we want to execute the code in the UI thread of execution.
In our case, to create a controller poll cycle, we use DispatcherTimer, which will be triggered 30 times per second:
Controller polling cycle organization
DispatcherTimer dt = new DispatcherTimer() { Interval = TimeSpan.FromSeconds(1 / 30) }; public MainWindow() { InitializeComponent(); dt.Tick += dt_Tick; dt.Start(); }
The dt_Tick method will focus the basic logic of motion tracking. We get the current frame (Frame), and if the fingers are visible in this frame, we take the first one of them and use its coordinates to change the position of the circle:
void dt_Tick(object sender, EventArgs e) { var fr = Leap.Frame(); if (fr!=null && fr.Fingers.Count>0) { var f = fr.Fingers[0]; Canvas.SetLeft(ptr, 512 + f.TipPosition.x); Canvas.SetTop(ptr, 768-f.TipPosition.y); } }
We have achieved that we can move the circle on the screen by moving your finger over the Leap Motion controller. Sample code at this stage can be viewed on GitHub .
Software generation of sound of various heights
For software generation of sound of different heights, we use the NAudio library. The easiest way to add NAudio support to a project is to use Nuget by typing package manager in the console:
install-package NAudio
The process of generating a variable pitch sound is described in detail in the article by Charles Petzold on MSDN . In short, we create a class that generates a sine wave of sound, and pass it to the input to the NAudio method, which reproduces this wave. In this case, there are a number of subtleties: in order for the sound to change more smoothly, it is necessary to smoothly change the frequency of the sound, and at the same time take into account the delay in audio buffering. All these subtleties are taken into account in the class PortamentoSineWaveOscillator.cs , which must be placed in the project.
To generate a sound, we create an object of type WaveOut and initialize it, passing an instance of the oscillator we created:
WaveOut WaveGen = new WaveOut(); PortamentoSineWaveOscillator Osc = new PortamentoSineWaveOscillator(44100,120);
We carry out initialization in the page constructor:
WaveGen.Init(Osc); WaveGen.Play();
Now we just need to change the Osc.Pitch property to change the pitch accordingly, and Osc.Amplitude to change the volume. Let the horizontal movement of the fingers are responsible for the volume, and the vertical - for the pitch. The coefficients are chosen empirically to achieve a comfortable sound:
void dt_Tick(object sender, EventArgs e) { var fr = Leap.Frame(); if (fr!=null && fr.Fingers.Count>0) { var f = fr.Fingers[0]; ... // var p = Math.Abs(f.TipPosition.y / 2); var a = 255 - Math.Abs(f.TipPosition.x); if (p >= 0 && p <= 150) Osc.Pitch = p; if (a >= 0 && a <= 255) Osc.Amplitude = (short)a; } }
A version of the project at this stage can be obtained here .
Finishing touches
To make the project a little more beautiful, instead of the usual circle moving around the screen, you can add special effects based on a particle system (Particle Systems). I took this article as a basis on CodeProject , where the source code of the WPF control, which is a source of particles, is given. Replacing them with the circle added in the first step, we get the final project, the source code of which is available on GitHub .
Homework
If you want to continue the experiments, the scope for creativity is almost unlimited! In particular, you can:
- Add support for polyphony and control with multiple fingers
- Experiment with other methods of controlling the pitch, for example, to take as a basis not the position of the finger, but its speed
- Add support for gestures to extract special sounds (samples)
If you are interested in this topic (write in the comments), then I will continue the series of articles, adding various additional interesting things related to Leap Motion, Kinect and the unexpected use of mobile devices. What would you like to hear? For example, this video shows what else you can do with Leap Motion in a couple of hours:
Source: https://habr.com/ru/post/214907/
All Articles