Sound effects in virtual worlds

This article is the third in a series about the qualitative development of VR projects. We have previously studied the principles of optimization in the article “Rendering and Optimization in VR Development” . Links to other materials can be found in the review article .

Until now, sound effects have played a secondary role in software development and were hardly noticeable. But with VR development, they become an integral part of the project.

The correct soundtrack of a VR project greatly enhances user immersion and heightens emotions.
')
A good VR project takes us to virtual worlds and allows us to experience this modeled reality on an emotional level. However, there are certain expectations, and if our projects cannot meet them, immersion and believability will disappear.

One aspect of these expectations is the sound effects that arise in real life. When a glass is put on a wooden table, it is pressed on the door handle or it is carried out with a hand over the chair - in all these situations there are sounds that we undoubtedly expect.

In classic games or applications, we take not so much participation, so the lack of sound doesn’t bother us so much. But in VR it's not like that. Therefore, a new problem appears, but also a chance to transfer impressions to a new level.

Here we need the fact that we know the exact position, and the turn of the head, and, for example, in the case of Vive, also the position of the hands relative to the head. About what cool effects can be embedded in your project, based on this data, we will look at the end of the article.

First, we will go over the hardware and software and find out why, even today, when simulating sound, they resort to deception.

Technology

The technology of modeling and recognition of sound waves has been around for a long time. Many still remember the 5.1 "surround sound system", which has been popular for some time.

For realistic sound perception in headphones , we have only two sound receivers (left and right). But the most important is the source of the sound. In the case of VR simulations, these are sound effects.

Sound waves are not a simple phenomenon, as they are reflected in space and change due to various materials and other influences.

In the real world, as long as the sound wave does not find its way from the source to our ears, it changes significantly and receives, depending on the circumstances - for example, when we sit in the bathroom or stand at a train station - certain characteristics.

Real life creates certain expectations for the sound, our virtual world must meet these expectations.

This approach is very clearly presented in the next video. In spite of the fact that you perceive video in ordinary headphones, you can transfer directly to the scene of events due to high-quality and realistic sound recording. You can even catch the location of sound sources, for example, a car signal.


Because of this complexity, a plausible sound simulation on a computer is worth a fortune, the power required to calculate several sound sources far exceeds the average computer.

Therefore, when developing software, they resort to deception and, at the right moment, play previously recorded audio tracks , for example, when a circle is broken.

Then during the game, these records will be adjusted depending on the environment. For example, in a large room, a gradual attenuation of the sound with multiple reflections (reverberation) will be added. In most cases, this substitution is sufficient.

Sensors give us very useful information about the exact position of the head in space. Thanks to this, we can attach sound effects to the corresponding source in the virtual room, and the software will calculate the direction and distortion.

This substitution is often called Location Based Sound. It allows you to experience a very realistic background noise in VR. In practice, high-quality simulations are achieved thanks to the VR SDK from Oculus or Valve.

All we have to do is place the sound effects in the virtual room.

Touching on this topic, it is worth mentioning the report on Oculus Connect 2, in 2015:

Increase the immersion

After we put sound effects on the stage when interacting with objects, we can make an even deeper immersion. Depending on the project, add trivia or do extensive work with sound.

You can also use the position and rotation of the controller for certain effects. Developers from Hover Junkers used this opportunity, during the rapid movement of a virtual weapon (controller) you can hear "swooosh" or metal rattling.

This cool idea is shown in detail in the video:


The platform under which the project is being developed dictates various restrictions . When developing for Cardboard VR, the performance and memory options are more scarce than Vive VR.

The cool benefits of 3D sound and body positioning can be used to attract attention. Thus, it is possible to highlight important points of the script or interactive, so that the user does not miss them and continues to navigate normally in the virtual world.

Dive deeper into the subject

In this article, we only slightly touched on this extensive and diverse topic.

For further study of the material, I strongly recommend the Nicolas Tsingos report from VISION SUMMIT 2016:


Also very interesting article Engadget

And in this article, Todd Baker and Manesh Mistry - developers from Land's End and Monument Valley - provide subject information about sound design in virtual reality. It is worth reading!

In the next article, “VR-Design: User Interface,” we will deal with the interface in VR (human-machine interaction).