CameraTablet - how to make a graphics tablet using a webcam
Hello, my name is Dmitry. Here I want to talk about why the mouse does not suit me, and how I try to replace it. I develop CAE programs for engineers (calculating statics and dynamics of mechanical systems), I work as a project manager for a third of the working time, and for the rest of the time I am a system architect, developer and tester in my own and in other projects. I always have ten to twenty windows open, between which I have to constantly jump:
As a result, I have to use a bunch of different programs every day. Each program has its own one and only graphical interface. And each interface has no shortcuts at all, or they are their own, unique and inimitable, which I cannot remember.
As a result, all my working day is a constant throwing between the mouse and the keyboard:
')
At one point, my right hand said: "Sorry, master, but something is too big for me." The brush was ill for almost two months, at first weakly, then more and more. In the end, I bought a vertical mouse, learned more shortcuts in Visual Studio and switched to Total Commander.
Now the hand does not hurt, but the problem with the mouse remains. I don’t know about you, but I don’t like this constant movement from left to right to grab the mouse and right to left to go back to the keyboard. Mouse in computers in 1981, more than thirty years have passed, and since then almost nothing has changed.
I don’t like any of the standard alternatives to the mouse: PointStick (pipka in the center of the keyboard for laptops), touchpad, graphics tablet - everyone has either poor accuracy or you still have to pull your hand off the keyboard.
There are programs and hardware for tracking eyes, like the Camera Mouse or the Tobii Eye Trackers. Honestly, I didn’t want to spend a hundred bucks on the Tobii Eye Tracker, and the Camera Mouse didn’t recognize the position of the eyes very well (I suspect my glasses are to blame).
In the end, I decided to write software myself, the benefit of the webcam is always there. The idea is simple: turn part of the keyboard into a graphics tablet. On the shortcut, the camera begins to track the position of the index finger and move the mouse to the appropriate place on the screen, another pair of shortcuts - emulation of the left and right mouse buttons. Here is a video of how it works.
I decided to write the program in Python, ostensibly it is possible to develop prototypes very quickly in this language. The funny thing is that in the end CameraTablet turned out to be very small. It took most of the time to understand how opencv recognizes images. Plus, I had to tinker a bit to find a normal mouse emulation script. As for the rest, thanks to the mass of Python libraries, we managed to quickly do everything we wanted. What I needed in the end:
The only more or less mathematical problem I encountered was how to calculate the position of the cursor on the screen if the position of the finger is known. The finger moves inside a non-equilateral quadrilateral, and the cursor moves inside a rectangular screen. So the problem is reduced to the projection of an arbitrary quadrilateral on a rectangle. It turned out, though not the first attempt, but pretty good.
At the moment the program is stable. Accuracy is enough to switch between windows and select menu items.
Fich, which at the moment there, but who really want to do:
And then came the most difficult moment, when the most delicious 20% of the work has already been done, but you don’t want to cut 80% alone. Then I decided that it was time to put everything on GitHub , since I had long wanted to do something for open source. I would be very happy if someone wants to take part in the project (testing, programming, documentation - anything you like). And, of course, it would be interesting to listen to thoughts on the interfaces of human-machine interfaces in general and alternatives to mice in particular.
- Visual Studio is actually a software, usually three or four different solutions.
- SourceSafe - versioning.
- Outlook
- Skype
- Total Commander - files, files.
- Internet Explorer - software planning, project progress monitoring, doxygen documentation, etc.
- Word, Excel - a list of tasks, meeting minutes, project requirements, quick scheduling, etc.
- PowerPoint - presentations for management.
- Notepad ++ - view intermediate results, configs, etc.
- CAE soft is our test and basic version.
- Other firms CAE soft - for tests
As a result, I have to use a bunch of different programs every day. Each program has its own one and only graphical interface. And each interface has no shortcuts at all, or they are their own, unique and inimitable, which I cannot remember.
As a result, all my working day is a constant throwing between the mouse and the keyboard:
')
- Mouse: open the document, scroll through,
- Keyboard: write three or four lines.
- Mouse: click on the button, move to another program, click on the button
- Keyboard: in the appeared window write 2-3 values.
- Mouse: open another window. Press the button. Select a menu item. Then the item in the sub-menu. Then in the sub-sub-menu. Cry! New window!
At one point, my right hand said: "Sorry, master, but something is too big for me." The brush was ill for almost two months, at first weakly, then more and more. In the end, I bought a vertical mouse, learned more shortcuts in Visual Studio and switched to Total Commander.
Now the hand does not hurt, but the problem with the mouse remains. I don’t know about you, but I don’t like this constant movement from left to right to grab the mouse and right to left to go back to the keyboard. Mouse in computers in 1981, more than thirty years have passed, and since then almost nothing has changed.
I don’t like any of the standard alternatives to the mouse: PointStick (pipka in the center of the keyboard for laptops), touchpad, graphics tablet - everyone has either poor accuracy or you still have to pull your hand off the keyboard.
There are programs and hardware for tracking eyes, like the Camera Mouse or the Tobii Eye Trackers. Honestly, I didn’t want to spend a hundred bucks on the Tobii Eye Tracker, and the Camera Mouse didn’t recognize the position of the eyes very well (I suspect my glasses are to blame).
CameraTablet program
In the end, I decided to write software myself, the benefit of the webcam is always there. The idea is simple: turn part of the keyboard into a graphics tablet. On the shortcut, the camera begins to track the position of the index finger and move the mouse to the appropriate place on the screen, another pair of shortcuts - emulation of the left and right mouse buttons. Here is a video of how it works.
How CameraTablet is made
I decided to write the program in Python, ostensibly it is possible to develop prototypes very quickly in this language. The funny thing is that in the end CameraTablet turned out to be very small. It took most of the time to understand how opencv recognizes images. Plus, I had to tinker a bit to find a normal mouse emulation script. As for the rest, thanks to the mass of Python libraries, we managed to quickly do everything we wanted. What I needed in the end:
- opencv
- finger recognition on video
- save photo file with finger
- camera calibration
- keyboard - handling hot keys
- TkInter - five buttons for GUI.
The only more or less mathematical problem I encountered was how to calculate the position of the cursor on the screen if the position of the finger is known. The finger moves inside a non-equilateral quadrilateral, and the cursor moves inside a rectangular screen. So the problem is reduced to the projection of an arbitrary quadrilateral on a rectangle. It turned out, though not the first attempt, but pretty good.
What's next
At the moment the program is stable. Accuracy is enough to switch between windows and select menu items.
Fich, which at the moment there, but who really want to do:
- Multi-monitor support
- MouseDown und MouseUp emulation (necessary for text selection)
- Linux support
And then came the most difficult moment, when the most delicious 20% of the work has already been done, but you don’t want to cut 80% alone. Then I decided that it was time to put everything on GitHub , since I had long wanted to do something for open source. I would be very happy if someone wants to take part in the project (testing, programming, documentation - anything you like). And, of course, it would be interesting to listen to thoughts on the interfaces of human-machine interfaces in general and alternatives to mice in particular.
Source: https://habr.com/ru/post/331288/
All Articles