Touchpad and keyboard from the handy device on Windows Mobile
Do you have a communicator with WM and the need to control your PC with it?
Maybe you want to spend some tricky presentation that requires a mouse in your pocket?
Ok, you need a cat (there are pictures).
')
If you are only interested in the final product, but not technical details, I recommend not to go to the FAQ section without stopping.
Task: Turn the WM-communicator into a full-fledged touchpad and keyboard, and at the same time gain the ability to control any win-system with its help.
In general, a managed system may not be win, but more on that below.
As a transport, we will use UDP / IP over Wi-Fi, or (for connoisseurs) Bluetooth.
What is required for this:
Written under the .NET Compact Framework 3.5 on C # , tested on HTC HD2 with WM 6.5 .
The client application does not use any vendor-specific API, so it should work on all devices with WM 5 and 6.
The principle of capturing input from the keyboard and touchscreen is extremely simple:
A little about the terms in which the principle of touchpad emulation is expressed - they are used in the further description:
ConfigApply and ConfigStore are irrelevant; All MouseHandler work comes down to handling its events, whose purpose as a whole is clear from the names.
The device is trivial:
Developments:
Class whose task is to generate control messages for sending to the server:
We get the requested action through a method of the type KeyboardCommand (), MouseMove (), etc; form the message, send it to the socket via the OnPacketSend event.
The user is presented with a unique opportunity to edit their configs in the root XML format.
How the application implements work with them, see for yourself if anyone is interested. I do not want to talk about it.
Screen:
The purpose of the main configs fields, which is not obvious from the name:
When drawing the interface, I was inspired by the popular picture of Kazimir Malevich:
Since the GUI is considered to be a self-documenting user interface, I still left button labels (screen on the left), which disappear after the first touch of the touchpad (screen on the right).
Yes, I have not said: this black field on the screen is the touchpad; the purpose of the buttons should be obvious from their name. Using the keyboard, too, should not cause questions.
The portrait orientation of the window is used to save screen space for the touchpad; at the same time the use of the touchpad itself is assumed in the landscape (but it is customizable).
The first attempt to create a more or less cross-platform server using python and SendKeys failed because it turned out that SendKeys does not know Unicode. How to work from a python with a mouse, I did not even look.
The second attempt (successful) was using WinAPI from a console application in C, where the work of writing commands to the mouse and keyboard stream is performed using the SendInput function.
A simple console application that listens to a UDP socket and periodically calls SendInput () is not worth considering. I note only that it practically does not consume system resources (especially when the client is disabled), so that it can be kept running all the time.
As far as portability is concerned, it is not difficult to write a platform-specific application for any other platform (the audience would need it), since a simple to implement command is used to deliver commands to the managed system
Data is transmitted only from the client to the server, no confirmation or return commands are provided.
Any command is sent in a fixed-size frame consisting of the following fields:
Q: Ok, I want to use Device2Keyboard. What do I need to do for this?
A: Download the client and server application on the links at the end of the post, and run them. The default settings assume that the communicator and the server must be on the same subnet, since the client has the default server address 255.255.255.255:12358 (broadcast IP address). Everything will work out of the box, no additional configuration is required.
Q: I do not want to use wireless interfaces. How to be?
A: Connect the communicator to a managed PC with ActiveSync installed, leaving the default client settings. Everything will work.
Q: I fixed a click by executing Drag'n'Drop, but the connection to the server was interrupted, and the click remained clamped. What to do?
A: Wait 3 seconds. If the server does not receive a command to set the state of mouse clicks from the client during this time, it will release all held buttons.
Q: When the mouse cursor is moved for a long time, my network will fall down due to the terrible traffic, right?
A: The default frequency of updating the cursor position is 10 ms (100 times per second); packet size (3 + 1 + 1 + 4) == 9 bytes, + overhead UDP / IP frame 28 bytes, we get 37 bytes in 1 message, or 3.6 kbit / s. If the mouse does not move, its position is not updated. The answer is negative.
Application for WM and desktop, installation do not require: dl.dropbox.com/u/3815390/dev2kbd/bin/bin.7z
Sources: dl.dropbox.com/u/3815390/dev2kbd/src/src.7z
The application and its sources are distributed under the WTFPL license.
Photo from the title, source: migdal-or.livejournal.com/55703.html
Maybe you want to spend some tricky presentation that requires a mouse in your pocket?
Ok, you need a cat (there are pictures).
')
If you are only interested in the final product, but not technical details, I recommend not to go to the FAQ section without stopping.
Concept
Task: Turn the WM-communicator into a full-fledged touchpad and keyboard, and at the same time gain the ability to control any win-system with its help.
In general, a managed system may not be win, but more on that below.
As a transport, we will use UDP / IP over Wi-Fi, or (for connoisseurs) Bluetooth.
What is required for this:
- capture input from the default keyboard of the device;
- using the touchscreen communicator to emulate:
- the usual touchpad, with scrolls and clicks;
- mouse wheel;
- mouse buttons (with the ability to move the cursor while holding the click for drag'n'drop);
- an application on a managed system (server) capable of writing to the keyboard and mouse stream the data received from the device;
- protocol of the client application exchange with the server on the managed system.
WM application
Written under the .NET Compact Framework 3.5 on C # , tested on HTC HD2 with WM 6.5 .
The client application does not use any vendor-specific API, so it should work on all devices with WM 5 and 6.
The principle of capturing input from the keyboard and touchscreen is extremely simple:
- Create some kind of control, and stretch it to fit the window (Dock = Fill);
- We put a focus on him;
- Subscribe to KeyDown, KeyUp, KeyPress events to intercept keystrokes;
- MouseDown, MouseUp, MouseMove for touchpad emulation.
Touchpad
A little about the terms in which the principle of touchpad emulation is expressed - they are used in the further description:
- Static Move - if the user has pulled his finger to the border of the touchpad and left it there, the controlled cursor will slowly crawl in the direction of that border. Convenient for dragging the cursor with a clicked click;
- Scroll - the user ran his finger over a particular region at the edge of the touchpad, which is interpreted as a scrolling wheel;
- Scroll Pad is the “special region” mentioned above at the edge of the touchpad. If the finger starts moving on it, then scrolling is activated, otherwise it works like a normal touchpad.
- Inertial Scrolling - the user activated scrolling, and removed his finger, without stopping his movement. Scrolling will be performed further at the same speed without user intervention, until he is able to touch the touchpad again.
- Click and Drag - clicked, removed the finger, and immediately pressed again - now you can steer the cursor with the click pressed.
ConfigApply and ConfigStore are irrelevant; All MouseHandler work comes down to handling its events, whose purpose as a whole is clear from the names.
Keyboard
The device is trivial:
Developments:
- OnKeyCommand - received a control character, one of these . In general, they are standardized.
- OnKeyUnicode - received a printable character. K.O. notices that in unicode.
Customer
Class whose task is to generate control messages for sending to the server:
We get the requested action through a method of the type KeyboardCommand (), MouseMove (), etc; form the message, send it to the socket via the OnPacketSend event.
Configs
The user is presented with a unique opportunity to edit their configs in the root XML format.
How the application implements work with them, see for yourself if anyone is interested. I do not want to talk about it.
Screen:
The purpose of the main configs fields, which is not obvious from the name:
- buttonRightClickHold - show the right-click lock button. Those. when you press it, the right click sticks until you press again. By default, the button is not shown.
- buttonSettings - show the button for calling the concise editor of configs, whose screenshot is available above. By default, this button is for some reason turned on.
- clickAndDragDelay - the time during which you need to re-click on the touchpad to get Click and Drag (description above).
- locationUpdateInterval - the cursor position update interval, in milliseconds.
- staticSpeed ​​- Static Move speed, pixels per packet.
- staticBoundsRelative - field boundaries where Static Move is activated; set relative to the total size of the window.
- scrollInMousePixels - sensitivity Scroll Pad; the smaller the number, the higher the sensitivity.
Interface
When drawing the interface, I was inspired by the popular picture of Kazimir Malevich:
Since the GUI is considered to be a self-documenting user interface, I still left button labels (screen on the left), which disappear after the first touch of the touchpad (screen on the right).
Yes, I have not said: this black field on the screen is the touchpad; the purpose of the buttons should be obvious from their name. Using the keyboard, too, should not cause questions.
The portrait orientation of the window is used to save screen space for the touchpad; at the same time the use of the touchpad itself is assumed in the landscape (but it is customizable).
Server
The first attempt to create a more or less cross-platform server using python and SendKeys failed because it turned out that SendKeys does not know Unicode. How to work from a python with a mouse, I did not even look.
The second attempt (successful) was using WinAPI from a console application in C, where the work of writing commands to the mouse and keyboard stream is performed using the SendInput function.
A simple console application that listens to a UDP socket and periodically calls SendInput () is not worth considering. I note only that it practically does not consume system resources (especially when the client is disabled), so that it can be kept running all the time.
As far as portability is concerned, it is not difficult to write a platform-specific application for any other platform (the audience would need it), since a simple to implement command is used to deliver commands to the managed system
Protocol
Data is transmitted only from the client to the server, no confirmation or return commands are provided.
Any command is sent in a fixed-size frame consisting of the following fields:
- header - 3 bytes, the string "d2k"
- packet sequence number (to avoid packet repetition during broadcasting, which is used by default) - 1 byte
- command type (control character, unicode character, move mouse, etc) - 1 byte
- these commands (cursor coordinates, character code, etc) - 4 bytes, little endian
FAQ
Q: Ok, I want to use Device2Keyboard. What do I need to do for this?
A: Download the client and server application on the links at the end of the post, and run them. The default settings assume that the communicator and the server must be on the same subnet, since the client has the default server address 255.255.255.255:12358 (broadcast IP address). Everything will work out of the box, no additional configuration is required.
Q: I do not want to use wireless interfaces. How to be?
A: Connect the communicator to a managed PC with ActiveSync installed, leaving the default client settings. Everything will work.
Q: I fixed a click by executing Drag'n'Drop, but the connection to the server was interrupted, and the click remained clamped. What to do?
A: Wait 3 seconds. If the server does not receive a command to set the state of mouse clicks from the client during this time, it will release all held buttons.
Q: When the mouse cursor is moved for a long time, my network will fall down due to the terrible traffic, right?
A: The default frequency of updating the cursor position is 10 ms (100 times per second); packet size (3 + 1 + 1 + 4) == 9 bytes, + overhead UDP / IP frame 28 bytes, we get 37 bytes in 1 message, or 3.6 kbit / s. If the mouse does not move, its position is not updated. The answer is negative.
Conclusion
Application for WM and desktop, installation do not require: dl.dropbox.com/u/3815390/dev2kbd/bin/bin.7z
Sources: dl.dropbox.com/u/3815390/dev2kbd/src/src.7z
The application and its sources are distributed under the WTFPL license.
Photo from the title, source: migdal-or.livejournal.com/55703.html
Source: https://habr.com/ru/post/120247/
All Articles