How did I connect Wii-nunchak to HoloLens, or why virtual objects need real buttons

Our entire team was looking forward to delivering the long-awaited HoloLens, it had to be ordered through friends overseas and kept fists, so that customs would not become interested in WiFi certified for Europe. Finally, in early September, the coveted oval case came to us. We open, charge, put on the head - this is the gate to the virtual world. The truth is asking for a pass at the gate - you need to log in with a password from Microsoft. Yes, by the way, it is long crypt-resistant, with special characters and in different registers. While our pioneer of virtual worlds, tweaking the air with an air-tap gesture, stuffed something like Opa on a keyboard floating in space, opa # 00FF00 ograd @ !, the rest had time to think about how we are going to control this 3D.

Problem

On desktops, and with some reservations on tablets and mobile phones, we have two types of input devices. Mouse - you can specify any of a million pixels. Fast, but inaccurate. Keyboard - press one or several buttons from a much smaller range (<100), but it works very reliably and even blindly. Special buttons like Escape are pressed almost without errors, you will not confuse.

In HoloLens, the mouse analogue is the point of the cursor hanging in front of the nose. The air-tap pinch is not as convenient as clicking a mouse button, but the attached one-button Clicker controller helps. It is easy to select the desired projected button and give the command “go”, but with the cancellation of the action there are serious problems - there is a catastrophic lack of Escape. For example, in Galaxy Explorer it is easy to show on the planet and call up detailed information, but attempts to return to the galaxy map or simply change the scale strongly resemble dancing with a tambourine, especially if you look at a person wearing glasses from the side.
')
Microsoft recommends giving teams by voice, through Cortana, but have you tried to do it at least five of us in the same room? Hi cubicles and Open Space. When using a working conveyor, as in our case, external noise interferes. In addition, looking at the screen, the user with the mouse immediately sees the available buttons and hyperlinks, and on the physical keyboard you can even touch the possible choices. But here in front of me is a microphone, how, looking at him, understand which commands are supported? It is not clear even what language to speak with him, not to mention the fact that our team is international and native language and the pronunciation of each is different.

Idea

So the idea was born to fasten a nunchuck from Wii to HoloLens. Two high-quality, hardware buttons with anti-debugging, joystick, plus, in perspective, the ability to use the accelerometer. The device is compact, mobile, durable and cheap. Moreover, the Wii remote is recognized via Bluetooth as a HID input device, what could be simpler?

The first disappointment was waiting for me here - to connect, the controller from Nintendo asks for a PIN, which is the device’s address that is written down the other way around. For example, my controller has the address 00: 1E: A9: 5D: A6: BC and you need to somehow enter a string of 5 characters with the codes BC, A6,5D, A9,1E, and at the end the final 00. Maybe someone and I was lucky with the controller's address, but I had all the available addresses in the range of non-printable characters.

Nunchuk gives out data on the I2C bus and from the microcontroller it is easy to interrogate the state of the buttons and accelerometer. I decided to transfer data to HoloLens via Bluefruit EZ-Key - this is a ready-made HID keyboard controller, capable of sending an arbitrary combination of up to 6 keys pressed simultaneously.

It turned out this architecture:

Arduino, as without her

Logic from the microcontroller requires a minimum - read your data from I2C, and send the corresponding USB HID code from the table . A ready-made library for the Arduino was found for reading, and he took it, specifically the choice fell on a miniature Arduino Pro Mini at 8MHz with 3.3 V power supply. handle and the microcontroller for a couple of kopecks.

We put together a prototype on a breadboard, run on HoloLens Internet Explorer as the simplest replacement for Notepad and ... hurray! Earned! When you click the buttons in the address bar of the browser letters appear!

First com clod

When debugging the code for Arduino, he stepped on a rake - suddenly Bluefruit stopped sending codes, although Bluetooth was fine and the device was still defined by HoloLens as a keyboard. The Arduino Mini is programmed with an FTDI cable or another Arduino, and the next time the prototype was reworked, Bluefruit wasn’t powered. Arduino, on the other hand, got the current from the computer through the programmer cable, and the program that started up started to honestly transmit the codes of the buttons. 3.3 volts attached to the input of the de-energized board burned the input RX-pin of the serial interface on the side of the Bluetooth module. And after all, I once read an article on Easyelectronics why you can't do this !

Fortunately, after replacing the Bluefruit with a new one, it went further. The connection scheme is very simple - no active elements, you only need to connect the pins to each other.



For power left nunchak cable and soldered USB-A plug. The Arduino stabilizer converts 5 volts from USB to 3.3, which feed the nunchuk and Bluefruit EZ-Key logic. At the end of the cable brought the Pair button from Bluetooth. Unusually, according to the instructions, this button should be connected to the power bus, and not to the ground, like all other buttons on the Bluefruit.

Programming and assembly

The program executed on the Arduino is very simple - 20 times per second to interrogate the pressed buttons, add their USB HID codes to the buffer (several presses are possible at the same time) and send them via a serial interface to the Bluetooth module. The only feature - it is important to correctly specify the type of Arduino in the IDE. My model works at 8 MHz, and not at the standard for Arduino 16, so at first the serial interface worked at half speed and gave the data at 4800 baud instead of the expected 9200.



All source codes and other files are laid out under the MIT-license on GitHub .

Arduino Mini and Bluefroot have similar sizes, and for compactness, I soldered them together in such a sandwich:


For isolation between the boards, a U-shaped plastic insert is made, which also protects the screen of the Bluetooth module from being squeezed. Cutting the cable, soldered to the nunchuk logic.


In general, it turned out quite a compact design, even fit inside the body. It was necessary to cut only some stiffeners.


The whole construction assembly. At the end of the cable is a power plug and a Bluetooth connection button.

Unity

From the point of view of HoloLens, my controller is a simple keyboard whose support has been built since the beginning of time. No external dependencies or third-party libraries are needed.

The directions of the joystick turn into ASDW, the “Z” button into Enter, and the “C” into Escape.

if (UnityEngine.Input.GetKey(KeyCode.A)) { // Joystick left ... } if (UnityEngine.Input.GetKey(KeyCode.D)) { // Joystick right ... } 

This code is very convenient to debug by running the HoloLens emulator on the desktop, you can simply press the buttons on the keyboard and, from the point of view of the application, there is no difference.

The result is this:



Video shot directly from points through the built-in function Mixed Reality Capture. Sources of the demo are also on Github.

What's next?

Power Bank is easy to put in your pocket and powered by USB does not greatly reduce mobility, but you want a fully autonomous device. I think to fasten the lithium-ion jar 18650 and the charging controller from Micro-USB. Arduino can be supplied with any voltage from 3.3 to 8 volts, so additional stabilization is not needed. Alas, the battery does not fit in the case, you will have to print your own on a 3D printer.


In nunchuk there is also an accelerometer, which I have not yet engaged. In principle, it can be used to simulate a mouse, but we have not yet figured out how to use it sensibly. HoloLens is planned to be used not sitting at a table in a quiet office, but on the go and in a noisy warehouse. The user will not be able to hold his hand horizontally, tilting the controller with precise movements, but clearly recognizable button presses are the thing.

Maybe someone of genius will invent a system of commands and gestures suitable for glasses, but for now the control of the keyboard and mouse is something that users are well aware of and developers are well able to program.

At the support forum, people demand new gestures and circuses , let's see what else Microsoft is inventing.