Geekly Articles each Day
📜 ⬆️ ⬇️

A mini-game with head position tracking or how I met headtrackr.js


11.02.2013 Habrayuzer omfg published an article with which I began my acquaintance with headtrackr.js .
In this topic, I will tell you how to get get the coordinates and tilt angle of the user's head in front of the monitor using browser tools with getUserMedia support, how to take into account defects of an image received from a webcam and filter them, and how to use this technology in my projects using only html + JavaScript .

Applications for this can come up with a huge amount. For simplicity, in this topic we will make a mini-game in which the snake will crawl from top to bottom and change direction depending on the position of the player's head.
The most impatient: the result is here .


Meet Headtrack.js


As the authors write on the project page , headtrack.js is a library for recognizing the face and head in real time, tracking the position of the head and its position relative to the screen, using a webcam and the standard webRTC / getUserMedia.
')
Let's try to make a small Hello World:

1) Create an html file.
Content:
<!doctype html> <html lang="en"> <head> <title></title> <meta http-equiv="X-UA-Compatible" content="IE=Edge"/> <meta charset="utf-8"> <style> body { background-color: #f0f0f0; margin-left: 10%; margin-right: 10%; margin-top: 5%; width: 40%; overflow: hidden; font-family: "Helvetica", Arial, Serif; position: relative; } </style> <script type="text/javascript" src="js/jquery.js"></script> </head> <body> <script src="js/headtrackr.js"></script> <canvas id="compare" width="320" height="240" style="display:none;"></canvas> <video id="vid" autoplay loop width="320" height="240"></video> <canvas id="overlay" width="320" height="240"></canvas> <canvas id="debug" width="320" height="240"></canvas> <p id='gUMMessage'></p> <p>  : <span id='headtrackerMessage'></span></p> <br> <p><input type="button" onclick="htracker.stop();htracker.start();" value=""></input> <br/><br/> <input type="checkbox" onclick="showProbabilityCanvas()" value=""></input> </p> <button id='stop_ang'></button> <div id='tab_p' style='height:100px; overflow:scroll;'> <table id='angles' border=1 cellspacing=0> </table> </div> <div id='slider_wrap'> <div id='slider'></div> </div> <script> //   video  canvas var videoInput = document.getElementById('vid'); var canvasInput = document.getElementById('compare'); var canvasOverlay = document.getElementById('overlay') var debugOverlay = document.getElementById('debug'); var overlayContext = canvasOverlay.getContext('2d'); canvasOverlay.style.position = "absolute"; canvasOverlay.style.top = '0px'; canvasOverlay.style.zIndex = '100001'; canvasOverlay.style.display = 'block'; debugOverlay.style.position = "absolute"; debugOverlay.style.top = '0px'; debugOverlay.style.zIndex = '100002'; debugOverlay.style.display = 'none'; //  ,   statusMessages = { "whitebalance" : "    ", "detecting" : " ", "hints" : "-  ,  ", "redetecting" : " , ..", "lost" : " ", "found" : "  " }; supportMessages = { "no getUserMedia" : "   getUserMedia", "no camera" : "  ." }; document.addEventListener("headtrackrStatus", function(event) { if (event.status in supportMessages) { var messagep = document.getElementById('gUMMessage'); messagep.innerHTML = supportMessages[event.status]; } else if (event.status in statusMessages) { var messagep = document.getElementById('headtrackerMessage'); messagep.innerHTML = statusMessages[event.status]; } }, true); //   var htracker = new headtrackr.Tracker({altVideo : {ogv : "", mp4 : ""}, calcAngles : true, ui : false, headPosition : false, debug : debugOverlay}); htracker.init(videoInput, canvasInput); htracker.start(); //    «»  document.addEventListener("facetrackingEvent", function( event ) { // clear canvas overlayContext.clearRect(0,0,320,240); // once we have stable tracking, draw rectangle if (event.detection == "CS") { overlayContext.translate(event.x, event.y) overlayContext.rotate(event.angle-(Math.PI/2)); overlayContext.strokeStyle = "#CC0000"; overlayContext.strokeRect((-(event.width/2)) >> 0, (-(event.height/2)) >> 0, event.width, event.height); overlayContext.rotate((Math.PI/2)-event.angle); overlayContext.translate(-event.x, -event.y); document.getElementById('ang').innerHTML=Number(event.angle *(180/ Math.PI)-90); } }); // \    () function showProbabilityCanvas() { var debugCanvas = document.getElementById('debug'); if (debugCanvas.style.display == 'none') { debugCanvas.style.display = 'block'; } else { debugCanvas.style.display = 'none'; } } </script> </body> </html>



Download the library and connect to our project:
 <script src="js/headtrackr.js"></script>


If we now open our example in the browser, we will see the following image:


And in debag mode:


As we see, our application works, the face is successfully determined.
This article discusses the problem of removing the angle of rotation of the head around the normal to the screen plane, so pay attention to this parameter.

Add the following code to facetrackingEvent:
 document.getElementById('ang').innerHTML=Number(event.angle *(180/ Math.PI)-90);


And in the Html document itself this:

 : <span id='ang'></span>


Now our page can display the deviation of the head from the vertical in degrees.
However, look at the graph of the angle values ​​obtained in 20 seconds:


Such a distressing picture speaks of both the noise in the image taken from the camera and the errors in the calculations of the library itself.
Do not panic, now we will deal with it.
The first thing I did was try to drive the data through the moving average filter , but the picture did not make me happy:


A slight improvement is present, but this is not at all what I expected.
Remembering the diploma (in which I put a whole bunch of filters on the current sensor), I decided to try the Kalman filter , which DA Kurnosov discovered for me. (teacher at my university), (also later, in many ways to understand the principle of its work from the point of view of the program code, the article justserega on Habré helped me, which helped me a lot by answering a lot of stupid and not so much questions in a personal ):


Already much better. However, this filter is taken with arbitrary settings.
We select the covariance and adjust the measurement error and obtain:


Just great. Here it is, but then I moved my head left and right:


This picture suits us.
Here is the code of the filter itself:
 var Q = 2; var R = 85; var F = 1; var H = 1; var X0; var P0; var State = 0; var Covariance = 0.1; function SetState(state_s,covariance_s){ State = state_s; Covariance = covariance_s; } function Correct(data) { X0 = F*State; P0 = F*Covariance*F + Q; var K = H*P0/(H*P0+R); State = X0 + K*(data - H*X0); Covariance = (1 - K*H)*P0; } SetState(0,0.1);



In the final archive, it is in a separate kalman.js file.

Test


To test the resulting system, I made a slider that moves left or right, depending on the inclination of the head:


Inspired by the results, I decided to sketch something more “visually understandable” from the point of view of displaying the smoothness of changes in coordinates:

Code painting on canvas "snake"
 var angles = [0]; var canvas = document.getElementById("canvas"); var rc=document.getElementById("canvas").getContext('2d'); rc.clearRect(0, 0, canvas.width, canvas.height); setInterval(function(){redraw(angles);},20); function redraw(angles){ rc.clearRect(0, 0, canvas.width, canvas.height); rc.beginPath(); for (var i=0;i<=angles.length-1;i++){ rc.lineTo(angles[i]+150,i+0); rc.moveTo(angles[i]+150,i+0); } rc.arc(angles[angles.length-1]+153, 200, 6, 0 , 2 * Math.PI, false); rc.stroke(); rc.moveTo(angles[angles.length-1]+150,200); rc.fillStyle = 'green'; rc.fill(); }



The angles array accumulates and stores the last 200 values ​​of the angle; when new values ​​are received, a left shift is made:
  angles[angles.length] = (angle*1.5); if (angles.length > 200){ angles.shift(); }


Result:


If the snake in the example starts to twitch convulsively - try to move away from the camera and reload the page.
Archive with a working example can be downloaded here.
Run the file 1.html
Attention, the example may not work if you run it from the local computer, so you can see it live here .
Excel file with removed values ​​for a pure signal and passed through filters and diagrams for all of this here: http://goo.gl/FWMBE

Next, I think either to develop the theme in the direction of pseudo-3D , or to refine the example from the article to something more serious (a menu that is controlled by a glance? Moving around the map by tilting the head? Etc.)

Thank you for your attention, have a nice day.

UPD: Free hosting is covered, the page on github: http://paulsmith220.github.com/htrack/

Source: https://habr.com/ru/post/169249/

More articles:


All Articles