Musical rehearsals: now and through the Internet

The article briefly describes several similar programs, and also describes the experience of developing its own software in this area, since available programs did not have all the necessary features at the same time.

Foreword

In early 2012, on the New Year holidays, I came across a curious movie:





The video shows that people are singing live, as written in the comments, located in different studios. It became interesting what kind of technology. The only clue was the word NETDUETTO. It turned out that there is such software produced by one of the laboratories of the Japanese company Yamaha. But the trouble is, the program is installed only on the Japanese locale. After searching the network, an unofficial distribution was found that installs on any locale. But now the executable file itself did not start. Included with the distribution was a patch, but I did not like it, and I decided to make my patch. Having found the locale definition code through IDA and patched it with the substitution of nops in the HEX editor, the program started, but the interface was not displayed. For it was made under the Japanese locale. Having tinker in Restorator, I taught the program an English-language interface (which, by the way, was in the .exe file itself, was simply inactive) and it was possible to start conducting experiments ( link ).

')

Netduetto

The program works fine, up to 4 participants of online music rehearsal are supported. Yes, that's bad luck. The maximum audio quality is 44.1 kHz / mono. Mono to me, as a keyboard player (and at keyboard instruments, stereo effects are sometimes harshly necessary), I definitely didn’t like it and the search for analogues began.

Ninjam

The only adequate counterpart was Ninjam. But he used the computer of one of the participants of the online rehearsal as a server, so the sound from one participant to another goes through this server, rather than indirectly. Taking into account the specifics, when milliseconds are important for a well-coordinated game, such an intermediate link is unacceptable (Netduetto works directly, each participant in the online rehearsal exchanges audio data with each). And, even without testing this tool, I began to write my program with support for both stereo mode and working without an intermediate server.

Steinberg VST Connect SE

But first I will mention another serious program with similar functionality. It differs from the rest in two parts. One is for the sound engineer, and is launched from under the DAW (Digital Audio Workstation - the general name of professional programs for working with sound), and the other for the performer is a separate executable file. Another limitation is that unlike other programs, this connects only two people and is positioned not as a rehearsal, but as a remote recording without the need to visit the studio. But at the moment it can only be tested by owners of Cubase 7, to whom, unfortunately, I can not count it.

Music Over The World Tool

So, my development . Using long-term observations of the mTorrent torrent client, as well as experience with UDP, this program was written. Next, I want to talk about some things that seemed interesting to me in the process of developing a program.



The program is written in pure Winapi. This is due to the fact that I needed to make an executable file that would run without any installation of myself this time. And also without installing any third-party software or libraries (such as, for example, .NET) these are two. As a result, the file size of just over 100 kB is fully functional and works even on Windows XP SP2.



Finding your own ip-address from within the program in the absence of its servers on the Internet (which are, for example, Yamaha and which are used in Netduetto to store information about the so-called “rooms” with musicians) was not obvious. After all, the Internet can be either direct or behind a router (roughly speaking, of course). And it is necessary to determine the external ip-address, and not the address of the end LAN, returned by the regular functions of determining the interface address. Therefore, a site was found that responds with a small page with your external ip-address. Since there are a huge number of such sites on the Internet, the one that was chosen was the minimum size and page design of which has not changed for several years.



Buffering, or struggle with excess signal delays. Here you took a guitar, connected to the microphone input of a computer sound card, took a chord, and then what? And then the signal is digitized by the ADC of the sound card, and n samples (where n is a multiple of 2; usually 128, 256 or 512) are transmitted to the ASIO driver . The program (for example, both Netduetto and Music Over The World Tool) has access to this driver, which signals that n data samples are updated in this buffer (ASIO driver input buffer). Total periodically in the program are data on n samples of the audio signal. Also, periodically, the Netduetto and Music Over The World Tool programs (I am comparing them now, because there is one important difference in terms of buffering) over the network receive data from other online rehearsal participants in the form of network packets containing m audio samples from each remote member. So, there are two events in the program - new n audio samples were received from the sound card input, and new m audio signal samples were received from the remote participant over the network. N samples: are transmitted to the output ASIO-buffer (so that you yourself hear what you are playing), and are also sent to the rest of the participants in the online rehearsal. M counts: here we are waiting for the main difference between the program Netduetto Music Over The World Tool. In Netduetto, data from a remote participant get into an intermediate buffer, which has a dynamically adjustable size, depending on the ping between the participants, the size of the input and output buffers of the ASIO driver, and the settings for compressing the audio streams of the participants. Thus, in Netduetto, the scheme for receiving audio samples from remote participants looks like this:



(m samples taken over the network) -> (intermediate buffer) -> (output buffer of the ASIO driver) -> (sound card DAC)



Whereas in the program Music Over The World Tool this scheme looks like this:



(m samples taken over the network) -> (output buffer of the ASIO driver) -> (sound card DAC)



And what does the lack of buffer mean? This means that the other person will hear your chord a few milliseconds earlier. And what is a few milliseconds? Many or few? And the context of a specific task is quite noticeable. Without this buffer in Music Over The World Tool, you can easily play at a 120-140 tempo. Of course, there are no miracles, and you have to pay for everything - you need to install the same size of the input and output buffers of the ASIO drivers on all the participants' computers. Also, with this approach, only raw uncompressed data can be transferred. But it's worth it.



Further, since Music Over The World Tool works on the basis of p2p technologies, we had to tinker while the mechanism was debugged whereby the UDP packets with the readings of the other participants were sent to the ip-address of the disconnected peer. On the service channel (a separate UDP port), a list of ip-addresses with which it is currently connected is transmitted to each participant in the online rehearsal once a second. And if one of the participants disconnected (closed the program), then a hail from the UDP packets of the main channel of audio signal samples (a separate UDP port for each participant) continued to fall on his ip-address. To avoid this, we had to introduce control of data exchange on the main channel. As well as storing a table of all ip-addresses that have ever been connected and not sending packets with audio samples to the addresses from which data has not been received for several seconds.



Another important difference from other programs is integrated VST support, and even VSTi (which is connected indirectly, via VST FLStudio). Those. From this program, you can hang up a handler, and, for example, your played chord will sound like on an overloaded electric guitar. You can also connect a MIDI keyboard and play VSTi-instruments. And all without the need to think of routing audio signals or run third-party programs (except for the need for a VST version of FLStudio to support VSTi).



Speaking of audio routing. Netduetto comes with a great driver that can replace VAC that has gotten blunt (in VAC, switching the input to the input occurs at the top level of abstraction through a separate Windows application, which takes more resources and less efficiently and elegantly than in the Yamaha driver, where the output switching to input occurs inside the driver kernel). In addition to its main purpose (in Netduetto, the entire output of the ASIO driver is duplicated to it, which is useful, for example, when broadcasting this audio to the Internet) you can connect the DAW output to the input of at least Netduetto or Music Over The World Tool.

In addition, you can install more than one copy of these virtual audio drivers.



If someone has read this far, but at the same time he hardly imagines what it is and why. These are the implementations of technologies that make distances invisible. You can sit at home in front of a computer and play music with friends who are many kilometers away as if they were in the same room with you. With CD-audio quality.



PS: History. Long times ago ... Several years ago, our compatriots had already created a similar program. But then there was neither ASIO, nor the modern Internet. Initially, the program was called Omnimusic , then it became the Musigy project. At the moment, no trace of that project was found on the Internet.