Using a synthesizer as a computer keyboard

Recently a thought occurred to me: is it possible, having connected a synthesizer to a computer, to type text on it? I tried to implement it, and I did it. My program reads the keystrokes of the synthesizer and emulates the keystrokes of a conventional keyboard. In this article I will tell how to implement it. We will write under Linux on C ++ using Qt.

Reading data from a synthesizer

So, there is a laptop with Linux and a synthesizer Yamaha DGX-200. We connect the synthesizer through the USB connector to the laptop and see that the device is recognized:
')


From the device comes a constant stream of questions, among which other characters appear when you press the keys of the synthesizer. By the way, an interesting fact: if you write this output to a file, and then read from the file and write back to / dev / midi2, then the synthesizer through its columns reproduces those notes that were pressed during recording, but without pauses.

The next task is the analysis of this stream. After a long search in google, I decided to use the portmidi library. The documentation for it is rather scant, I haven’t found any working examples at all. Well, now will be one more example. Get the list of devices:

int count = Pm_CountDevices(); for(int i = 0; i < count; i++) { const PmDeviceInfo* info = Pm_GetDeviceInfo(i); qDebug() << i << ": " << info->name << " input: " << info->input << " output: " << info->output; } 

I got this result:

 0: Midi Through Port-0 input: 0 output: 1 1: Midi Through Port-0 input: 1 output: 0 2: YAMAHA Portable G MIDI 1 input: 0 output: 1 3: YAMAHA Portable G MIDI 1 input: 1 output: 0 

For further work with the device, we need to know only the id, which is listed at the beginning of the line. Device 3 is suitable for us - input (input = 1) stream from our synthesizer. Open the desired stream:

 PortMidiStream* stream = 0; PmError e = Pm_OpenInput(&stream, good_id, 0, 100, 0, 0); if (e != pmNoError) { qWarning() << "Can't open input, error: " << e << endl; return 2; } 

After that, we periodically read the data. I used a Qt-slot with a periodic timer call, but the usual while (true) and sleep is also suitable.

 PmEvent event; // ,       int c = Pm_Read(stream, &event, 1); //      if (c > 0 && Pm_MessageStatus(event.message) == 144) { unsigned int note = Pm_MessageData1(event.message), volume = Pm_MessageData2(event.message); //   note  volume } 

To clarify what these magic numbers are, I will tell you how the MIDI commands are arranged.

MIDI commands

Each message (also known as a MIDI command) consists of three integers, which are called status, data1, data2 in portmidi. A table with possible statuses can be found here . We are only interested in the status of 144 - the change in the state of the note on the first channel. In data1, the note number is transmitted, and in data2, its volume. For example, when you press the “before” key of the first octave on the synthesizer, the command 144 60 95 comes, and when you release - 144 60 0 .

The volume varies from 0 to 127 and depends on the strength of the impact on the key. Theoretically, you can display capital letters instead of lowercase letters when the user hits the keyboard hard. Well, the note number is just a sequence number, the correspondence of the notes to the numbers can be seen in this picture:

The designation of notes and chords

I decided to designate the note as “3C” or “3C #”, where 3 is the number of the octave (and the octave starts with “A”, it's easier), C is the note designation (“before”), and if necessary, a sharp is added. Here is how it is implemented:

 class Note { public: Note(int midi_number); QString to_string() const; int tone, octave; }; Note::Note(int midi_number) { int n = midi_number - 21; octave = n / 12; tone = (n - octave * 12); } QString Note::to_string() const { return QObject::tr("%1%2").arg(octave).arg( tone == 0? "A": tone == 1? "A#": tone == 2? "B": tone == 3? "C": tone == 4? "C#": tone == 5? "D": tone == 6? "D#": tone == 7? "E": tone == 8? "F": tone == 9? "F#": tone == 10? "G": tone == 11? "G#": "??" ); } 

If the user presses a chord (several notes simultaneously), then several messages come almost immediately. We can programmatically monitor this situation and distinguish single clicks from chords. In my program you can put different chords in correspondence with different letters. To get the chord designation, we combine the plus designations of the keys included in it: “3C # + 3E + 3G #”. When the user presses a note or chord, the program looks for a string in the layout that matches this designation, and emulates pressing the corresponding key. When the key on the synthesizer is released, the release of the key is emulated. Modifiers (Shift, Ctrl, etc.) are no different from other keys. All combinations work as expected.

Keystroke Emulation

Hooray, we can determine when notes are pressed and released. Now we learn to emulate keystrokes. We will use the solution I found on Stack Overflow .

 #include <X11/Xlib.h> #include <X11/keysym.h> #include <X11/extensions/XTest.h> Display* display = XOpenDisplay(0); void emulate_key(QString key, bool pressed) { KeySym sym = XStringToKeysym(key.toAscii()); if (sym == NoSymbol) { qWarning() << "Failed to emulate key: " << key; return; } XTestFakeKeyEvent(display, XKeysymToKeycode(display, sym), pressed, 0); XFlush(display); } 

I added the use of the XStringToKeysym function to emulate a key by its name, which we will take from the configuration file. A list of valid keys can be found in the header file /usr/include/X11/keysymdef.h.

The layout will be stored in the layout.ini file of the following form:

 ; letters X = 3C#+3G# Y = 3D#+3G# Z = 3E+3G# ; navigation Space = 2E Return = 1A+2A BackSpace = 4C# Delete = 4D# Left = 4A Right = 4C 

Layout

There remains the last task - to come up with a layout. A simple option - each letter on the key - turned out to be inconvenient. The keys are enough, but end-to-end, besides, you often have to carry hands because of the large dimensions of the synthesizer. Fortunately, we can use keyboard shortcuts, and it is much more convenient to press them on a synthesizer than on a computer keyboard.

Let's try to fit on 26 keys (from A to G #) 26 Latin letters. The choice of letters for the seven white keys is obvious - these are letters from A to G, which are the generally accepted notation for the corresponding notes. Then I wrote out the remaining letters in decreasing order of frequency and tried to make a word from the letters that are closer to the top of the list. I got the word HINTS, and I gave these five letters to the black keys. For the rest of the keys, I assigned the major and minor third in the same octave in alphabetical order. There are still a lot of options for placing other letters (for example, Russian).

The rest of the keys, too, found a place on the keyboard. I got this layout:



A music score editor, MuseScore, was used.

The video attached to the post shows a set of the Hello world program and its compilation using a synthesizer instead of a keyboard.

The program code is laid out on Github .