"Volume" music: how VR-technologies transmit sound
Photo david pacey / CC
3D and VR technologies cover more and more areas. The most popular are computer games and the film industry. In one of the last articles, we talked about the possibilities of using VR technologies in the film industry, and also talked about 3D sound and its prospects.
')
In this article, you will learn about the features of sound and music playback in VR, what problems audio professionals face and how they can be solved.
"Really what you realize"
There are two ways to transfer information using VR technology. In the first case, developers are trying to recreate a picture as close as possible to reality. In the second - to generate a completely different world and atmosphere, to create an illusion in which everything that happens is logically connected.
With the second method, everything is simple: if the situation, the characters and their actions are not real, then the laws of this world are different from reality. It is much more difficult to recreate what is real in itself. The difficulty lies in the fact that in some games or video plots there are strange moments that “pull out” the human consciousness from this illusion.
Game developers and multipliers are familiar with the hypothesis, which is called the effect of the “sinister valley” (uncanny valley). The essence of the hypothesis is that a character who looks or acts almost like a person, but still differs slightly from him, causes us to fear or dislike. Something similar happens with the sound in VR.
Unlike visual perception, to determine which object is in front of us, by ear, is much more difficult. It is possible to know its approximate dimensions and relative distance to it only if the object makes sounds. Due to the fact that we have a rich “auditory” experience of perceiving the world, in virtual reality the brain compares new information with what is happening to us in reality.
In this case, the human voice is the most “familiar” factor for us: therefore, its presence in the sound design of the game makes our “auditory immersion” into virtual reality much easier. But the voice, like the human-like character, is subject to the effect of an ominous valley. For example, too high sound compression will cause a “distrust” in our consciousness.
The video below helps you understand what our brains are capable of. All information about the movement of objects the brain receives only from the sound signal. This and the following video is recommended to listen in headphones and with eyes closed.
The sound in this video was created using the method of binaural recording: it is carried out through two microphones, and the final piece must be heard in the headphones.
In addition to the special arrangement of microphones, either a mannequin imitating a real head, or the so-called Jacklin disc (see below) is involved in the recording. Do not confuse the concept of "binaural" and "stereo". Regular stereo recording does not take into account the distance between the ears and the reflection of sound from the head and auricles - factors affecting sound propagation.
Jacklin Disc / CC
In modern virtual reality helmets, these conditions, unfortunately, are not taken into account. Moreover, companies such as Project Morpheus and HTC Vive are supplying VR helmets without headphones, inviting customers to use their technology. The Oculus Rift comes with branded headphones, but even here it is supposed to be replaced with a “higher quality” model.
Headset with Surround Sound 7.1 technology is very popular with gamers. On the website of the Kotaku gamers community a vote was taken for the best gaming headphones. The most popular models were the Astro A50 and Logitech G930 wireless headphones that support this technology. In other words, the community itself has determined that “high quality” implies a surround sound effect.
So, binaural sound is different from stereo. Stereo (2.0), in turn, differs from surround sound (7.1): If special sound recording methods are required for binaural and stereo sound, in order to achieve a surround sound effect, you need to play the sound through a large number of sound sources. It is due to the number of speakers that the immersive effect is created.
However, for VR, the first thing that matters is not immersion, but “realistic” sound. If a binaural recording is subjected to processing via 7.1 headsets, then the final recording is unlikely to meet the listener's expectations. This means that the standard methods cannot solve this problem.
The main thing is realism
Photo pixabay / CC
To achieve realistic sound, you need to take into account the position of the body in space, as well as monitor the interaction with the transfer function of the head (HRTF).
In VR, the user has the ability to move and make head movements in any direction. Accordingly, if the head, or rather the ears, change their position, the incoming audio signal must also change. Binaural recording can enhance the effect of immersion in virtual reality, emphasizing the depth of events.
In addition to the sounds we are used to hearing - the so-called diegetic sounds, there are also non-diegetic sounds: we do not hear them in everyday life, but they can be part of a computer game or movie. An example is background music or narrator speech.
In a computer game, we are not interested in where the voice and music come from. Another thing is virtual reality. In order to enter non-diegetic sounds into the virtual world, it is necessary to use connecting objects. For example, it would be logical to look if the narrator speaks on the radio or into the telephone receiver.
As for music, there are other questions. Of course, it can be tied to the radio, so that it is clear where the sound comes from. But this is not enough: the music emanating from it will sound uneven.
One of the possible solutions is the use of dynamic compression along with low-frequency amplification, causing a “proximity effect”. In the video below, Alexander J Turner explores this phenomenon in detail. It is closely connected with the microphone device: the closer to him the speaker, the brighter the lower frequencies at the output and the more intense the sound.
In games, compositions recorded in studios or other acoustic spaces are often used. These spaces convey the "liveliness" of the sound due to the fact that the sound of the instruments is mixed with the sound reflections in the room. So, at a concert at the conservatory, there is no feeling that the music is pressing or interfering with personal space. On the contrary, we feel that she is at a distance from us.
However, VR does not use “too live” recordings: the listener will not understand where the sound comes from. An alternative solution to this problem would be recording music in a neutral environment with subsequent processing so as not to disturb the organic sound. Also, it may be worth refusing stereo sound and reduce everything to a more centered sound. You can compress it a bit, and then apply the equalizer settings to it. Then the music will sound as if it plays not “somewhere out there,” but “in our head”.
The effect of realism can also be achieved through an unobtrusive emphasis on the surrounding situation. For example, in the next video, both visual and sound tension is felt. It is worth noting that the effect of the diegetic sounds in this recording is stronger than from non-diegetic sounds, and more emphasis is placed on the setting than on the sound.
Another example of the same approach. Just a few sounds create an inhaling and tense atmosphere.
Thus, the sound, adjusted too clean and neat, may seem artificial. At the same time, even if it is “dirty”, but it creates a realistic picture - with the help of additional sound effects or a dynamic environment - the impression of listening will be much stronger.
In other words, do not strive to make the sound perfect. It is necessary to make it as realistic as possible so that the listener can feel the emotions that the creator of this or that work conceived.
We are constantly versed in new products and just study interesting sound materials. Most recently, we began to prepare thematic digests , a couple of examples and other materials on the subject - on the links below:
Source: https://habr.com/ru/post/397147/
All Articles