We program voice control of a kopter with use of Node.js and ARDrone

In this tutorial, we will analyze the creation of a drone program with voice control on Node.js and Web speech API. Kopter - Parrot ARDrone 2.0.

We remind: for all readers of "Habr" - a discount of 10,000 rubles when writing to any Skillbox course on the promotional code "Habr".

Skillbox recommends: Practical course "Mobile Developer PRO" .

Introduction

Drones are amazing. I really like to play with my copter, recording photos and videos or just having fun. But unmanned vehicles (UAVs) are not only used for entertainment. They work in cinema, study glaciers, are used by the military and representatives of the agricultural sphere.
')
In this tutorial, we will analyze the creation of a program that allows you to manage the drone. using voice commands. Yes, the copter will do what you tell him. At the end of the article - the finished program and video management UAV.

Iron

We need the following:

• Parrot ARDrone 2.0;
• Ethernet cable;
• good microphone

Development and management will be conducted on workstations with Windows / Mac / Ubuntu. I personally worked with Mac and Ubuntu 18.04.

Software

Download the latest version of Node.js from the official site .

Also need the latest version of Google Chrome .

We understand with a kopter

Let's try to understand how Parrot ARDrone works. This copter has four motors.



The opposing motors work in one direction. One pair rotates clockwise, the other - against. The drone moves by changing the angle of inclination relative to the surface of the earth, changing the speed of rotation of the motors and a few more maneuverable movements.



As we see in the diagram above, a change in various parameters leads to a change in the direction of movement of the copter. For example, reducing or increasing the speed of rotation of the left and right rotors create a roll. This allows the drone to fly forward or backward.

Changing the speed and direction of movement of the motors, we set the angles of inclination, allowing the copter to make movement in other directions. Actually, it is not necessary to study aerodynamics for the current project, it’s just worth understanding the basic principles.

How does Parrot ARDrone work

Drone is a Wi-Fi access point. In order to receive and send commands to the copter, you need to connect to this point. There are many different applications that allow you to control copters. It looks like this:



Once the drone is connected, open the terminal and telnet 192.168.1.1 - this is the IP of the driver. For Linux, you can use Linux Busybox .

Application architecture

Our code will be divided into the following modules:

• user interface with voice API for voice detection;
• command filtering and benchmarking;
• sending commands to the drone;
• live video feed.

The API works subject to internet connection. To provide it, we add an Ethernet connection.

It's time to create an application!

Code

First, create a new folder and switch to it using the terminal.

Then we create a Node project using the commands below.

First, install the required dependencies.

npm install

We will support the following commands:

• takeoff;
• landing;
• up - the drone rises to half a meter and freezes;
• down - goes down by half a meter and freezes;
• to the left - goes to the left by half a meter;
• to the right - goes to the right by half a meter;
• turn - rotates clockwise 90 degrees;
• forward - goes forward half a meter;
• back - goes back half a meter;
• Stop.

Here is the code that allows you to receive commands, filter them and control the drone.

const express = require('express'); const bodyparser = require('body-parser'); var arDrone = require('ar-drone'); const router = express.Router(); const app = express(); const commands = ['takeoff', 'land','up','down','goleft','goright','turn','goforward','gobackward','stop']; var drone = arDrone.createClient(); // disable emergency drone.disableEmergency(); // express app.use(bodyparser.json()); app.use(express.static(__dirname + '/public')); router.get('/',(req,res) => { res.sendFile('index.html'); }); router.post('/command',(req,res) => { console.log('command recieved ', req.body); console.log('existing commands', commands); let command = req.body.command.replace(/ /g,''); if(commands.indexOf(command) !== -1) { switch(command.toUpperCase()) { case "TAKEOFF": console.log('taking off the drone'); drone.takeoff(); break; case "LAND": console.log('landing the drone'); drone.land(); break; case "UP": console.log('taking the drone up half meter'); drone.up(0.2); setTimeout(() => { drone.stop(); clearTimeout(); },2000); break; case "DOWN": console.log('taking the drone down half meter'); drone.down(0.2); setTimeout(() => { drone.stop(); clearTimeout(); },2000); break; case "GOLEFT": console.log('taking the drone left 1 meter'); drone.left(0.1); setTimeout(() => { drone.stop(); clearTimeout(); },1000); break; case "GORIGHT": console.log('taking the drone right 1 meter'); drone.right(0.1); setTimeout(() => { drone.stop(); clearTimeout(); },1000); break; case "TURN": console.log('turning the drone'); drone.clockwise(0.4); setTimeout(() => { drone.stop(); clearTimeout(); },2000); break; case "GOFORWARD": console.log('moving the drone forward by 1 meter'); drone.front(0.1); setTimeout(() => { drone.stop(); clearTimeout(); },2000); break; case "GOBACKWARD": console.log('moving the drone backward 1 meter'); drone.back(0.1); setTimeout(() => { drone.stop(); clearTimeout(); },2000); break; case "STOP": drone.stop(); break; default: break; } } res.send('OK'); }); app.use('/',router); app.listen(process.env.port || 3000); 

But the HTML and JavaScript code that listens to the user and sends a command to the Node server.

 <!DOCTYPE html> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <title>Voice Controlled Notes App</title> <meta name="description" content=""> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/shoelace-css/1.0.0-beta16/shoelace.css"> <link rel="stylesheet" href="styles.css"> </head> <body> <div class="container"> <h1>Voice Controlled Drone</h1> <p class="page-description">A tiny app that allows you to control AR drone using voice</p> <h3 class="no-browser-support">Sorry, Your Browser Doesn't Support the Web Speech API. Try Opening This Demo In Google Chrome.</h3> <div class="app"> <h3>Give the command</h3> <div class="input-single"> <textarea id="note-textarea" placeholder="Create a new note by typing or using voice recognition." rows="6"></textarea> </div> <button id="start-record-btn" title="Start Recording">Start Recognition</button> <button id="pause-record-btn" title="Pause Recording">Pause Recognition</button> <p id="recording-instructions">Press the <strong>Start Recognition</strong> button and allow access.</p> </div> </div> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script> <script src="script.js"></script> </body> </html> 

And more JavaScript-code in order to work with voice commands, sending them to the server Node.

 try { var SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; var recognition = new SpeechRecognition(); } catch(e) { console.error(e); $('.no-browser-support').show(); $('.app').hide(); } // other code, please refer GitHub source recognition.onresult = function(event) { // event is a SpeechRecognitionEvent object. // It holds all the lines we have captured so far. // We only need the current one. var current = event.resultIndex; // Get a transcript of what was said. var transcript = event.results[current][0].transcript; // send it to the backend $.ajax({ type: 'POST', url: '/command/', data: JSON.stringify({command: transcript}), success: function(data) { console.log(data) }, contentType: "application/json", dataType: 'json' }); }; 

Run the application

The program can be run as follows (it is important to make sure that the driver is connected to Wi-Fi, and the Ethernet cable is connected to the computer).

Open localhost: 3000 in the browser and click Start Recognition.



We try to manage the drone and rejoice.

Broadcast video from drone

In the project, create a new file and copy this code there:

 const http = require("http"); const drone = require("dronestream"); const server = http.createServer(function(req, res) { require("fs").createReadStream(__dirname + "/public/video.html").pipe(res); }); drone.listen(server); server.listen(4000); 

But the HTML code, we place it inside the public folder.

 <!doctype html> <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> <title>Stream as module</title> <script src="/dronestream/nodecopter-client.js" type="text/javascript" charset="utf-8"></script> </head> <body> <h1 id="heading">Drone video stream</h1> <div id="droneStream" style="width: 640px; height: 360px"> </div> <script type="text/javascript" charset="utf-8"> new NodecopterStream(document.getElementById("droneStream")); </script> </body> </html> 

Launch and connect to localhost: 8080 to view video from the front camera.

Useful tips

• Manage this drone indoors.
• Always put the protective cover on the drone before takeoff.
• Check if the battery is charged.
• If the drone behaves strangely, hold it below and turn it over. This action will put the copter into emergency mode, and the rotors will immediately stop.

Ready code and demonstration

LIVE DEMO

DOWNLOAD

Happened!

Write the code and then watch the car start to listen, it will give you pleasure! Now we understand how to teach the drone to listen to voice commands. In fact, the possibilities are much greater: face recognition of the user, autonomous flights, gesture recognition and much more.

What can you offer to improve the program?

Skillbox recommends:

• Applied online course "Python Data Analyst" .
  
• Online course "Profession frontend-developer" .
• Practical annual course "PHP developer from 0 to PRO" .