Opportunities for using 360 ° video technology to present cultural events
As part of the work at the Center for Design and Multimedia at ITMO University, we wrote an interesting article on the application of virtual reality technology to preserve cultural heritage. They described many of their own "cases" - museums, concerts, cultural events, and also told about a unique project of an interactive tour of the Korela fortress (Leningrad region), where they combined for the first time 3D-reconstruction and interactive "video 360".
In the English version, the materials will be presented in May at the international conference “ArtsIT, Interactivity & Game Creation” (Esbjerg, Denmark). Here is the Russian, slightly adapted original.
')
Virtual reality and its application
Currently, virtual reality technologies and their applications are rapidly developing. Great opportunities are opened by the use of such technologies in culture and art , as they provide the effect of the presence of the viewer in the center of cultural events. One of the most promising and currently being actively developed approaches to creating content for virtual reality systems is the 360 ° video technology, which allows you to create panoramic videos with varying degrees of interactivity, where the viewer controls the viewing angle of the video at will. Such a video can be viewed both in a virtual reality helmet (for example, in the Oculus Rift , Samsung Gear VR , HTC Vive , etc.), and with the help of a special application on the smartphone, while the picture changes in accordance with the user's head turns. You can view 360 ° video on the display of a personal computer. In the latter case, the user controls the view using the mouse or keyboard.
Watching video in 360 ° format in the Samsung Gear VR headset
Many large companies (Facebook, Nokia, Samsung, Google, etc.) are currently developing cameras for shooting video in 360 ° format, virtual reality headsets for various smartphones and desktops, as well as various recording devices that provide surround sound (binaural). a) sound and allowing to realize the whole complex of “multimedia 360 °” technologies.
The use of 360 ° video technology in education is especially important. Interactive video 360 ° can provide interaction with the end user, so this method can be widely used to train specialists in various fields where the development of practical experience requires a presence at the center of the event. The 360 ° video technology allows you to analyze the teamwork of staff while performing various tasks, which leads to an increase in the quality of the educational process.
A significant role is played by a 360 ° panoramic video to preserve and ensure access to cultural heritage. This technology provides a unique opportunity to see with the immersion effect not only closed archives of museums, but also reconstructed historical monuments that were destroyed by time or circumstances (three-dimensional virtual reconstruction). In addition, virtual reality is a unique opportunity for people with disabilities to travel and walk around the world.
Speaking of entertainment in the aspect of technology video 360 ° - viewers can visit the largest cultural festivals, find themselves on stage with their favorite musicians or watch a theatrical performance from the royal box.
The ITMO University Design and Multimedia Center is implementing a project for the development and application of 360 ° video technologies in the arts and culture Video360Production.com . Within the framework of this project, it was possible to achieve certain results necessary for obtaining high-quality content, starting with the development of original video recording devices 360 ° and sound and ending with the development of software for solving a number of tasks in this area.
360 ° multimedia technology
At the moment, special cameras are being developed with the help of which it would be possible to shoot video in 360 ° video format. However, with the help of special tools and software, you can use existing cameras to shoot 360 ° video.
A vivid example of this approach is a number of projects offering special rigs for GoPro HERO 4 cameras. The number of cameras in the riga is from 6 to 14 (stereo versions), depending on the tasks. After shooting, video streams are stitched into the general video sphere with the help of separately supplied programs.
In addition, several large IT companies and electronics manufacturers announced their own fully automated solutions for shooting content for virtual reality - Google Jump + GoPro Odyssey , Samsung Beyond , Nokia Ozo .
GoPro Odyssey, Nokia OZO and Samsung Project Beyond
We have described our personal impressions of the Nokia OZO camera here .
The 360 ° video is a new and progressively developing direction that does not yet have standards of direction, which provides a wide scope for creativity and the development of new ideas and approaches. The effect of full immersion is achieved by a camera with multiple lenses, which allows you to shoot everything that happens around. However, this entails many difficulties, because many laws of direction do not apply when shooting in 360 ° video format. The main cardinal difference in viewing 360 ° video in virtual reality headsets is the fact that the viewer has the opportunity to watch content around himself, which means that the director will have to give the viewer the ability to dynamically select the viewing angle, rather than focus on a specific part of the screen.
The solution to this problem is the detailed reasoning of the scenario from the point of view of the paradigm shift - the departure to 3D video space. Each shooting requires both standard preparation (setting up the light, the most advantageous points for the camera, scenario and location of the scene), and non-standard approaches (special and often unique for each camera mount, direction taking into account the features of video recording 360 °, correct camera orientation, surround sound recording). When recording concerts, to achieve greater realism, the sound from the camera or from the stereo microphone is often combined with the sound from the director's console.
Representation of cultural events using 360 ° multimedia technology
The video360production.com collection of video footage for virtual reality headsets includes more than fifty different videos: concerts and music festivals, excursions to historical places, excursions in museums, theatrical performances, various cultural events and technological processes.
Group "Bravo", "Love does not burn" in the format of panoramic video
Grouping "Leningrad" in the format of panoramic video
Technology of laying asphalt "Ecohiter", panoramic promotional video
When recording concerts, it is necessary to transmit in high quality both the image and the sound. Moreover, it is necessary to think about the location of the camera, since when recording to a 360 ° camera, such concepts as “front” and “background” become irrelevant. In addition, sometimes the scene was simultaneously up to twenty people, which complicated the installation of the camera. The footage allowed the musicians to look at themselves from the outside, to evaluate the location of the musicians during the concert, as well as to see the public reaction to the performance. The spectators, in turn, with the help of this survey were able to examine in detail the play of each musician and immerse themselves in the atmosphere created by the orchestra or group.
Concert of the group "Picnic" in the format of panoramic video
During the shooting of the famous Russian group "Picnic", the following problems arose - the concert was held in a dark room of the concert hall under the bright light of spotlights. These conditions are complex for ordinary cameras, since even at maximum resolution, the transmission of realistic color depth is quite a challenge.
Video recordings of concerts in the 360 ° format have a huge advantage over standard video recordings, as the viewer feels the effect of being present in the stage space and has full autonomy to decide for himself where to look at the moment — whether to watch the guitarist or look at the soloist, or maybe see concert "eyes" of musicians, turning his back to the stage, and watch the lively reaction of the public.
To achieve the effect of complete immersion, not only a high-quality picture is necessary, but also a “live” sound. For musicians and frequent visitors to concerts, it is not a secret that the “technical” sound on the stage is far from what is heard in the hall. Sound engineers of the project combined sound from a stereo microphone with a studio recording. As a result, the sound remains of high quality, but it retains its “liveliness”: while watching the recording you can hear how the audience reacts and sings.
In addition to musical groups, the performers of the project video360production.com cooperates with the largest St. Petersburg theaters, where a number of performances and musical performances were shot. A unique opportunity to provide shooting historical sites in cities and excursions in museums. In the Museum of Anthropology and Ethnography. Peter the Great (Kunstkamera) in a 360 ° video format, before the reconstruction began, a large academic ( Gottorp ) globe was filmed - a diplomatic gift to Tsar Peter I from the Duke of Golstinsky Karl Friedrich during the Northern War.
Shooting a large academic (Gottorp) globe in 360 ° video format
The implementation of a joint project of the ITMO University and the State Hermitage Museum “Points of Presence” has begun, within which it is planned to present events in various areas of the museum using panoramic video filming technologies.
Shooting in the State Hermitage the video clip “Peacock Clock” in 360 ° video format
The virtual walk along the rivers and canals of St. Petersburg is also of heightened interest among the visitors of the YouTube channel created by the authors of the project.
Particular attention was paid to filming a 360 ° video of various events in the Solovetsky Monastery. The Solovki monastery played an exceptional role in the history and culture of Russia. Originated in the XV century. as a place of desertification of spiritual followers of the "father of Russian monasticism" Sergius of Radonezh, in a short time, he became the spiritual, cultural, political and economic capital of the North of Russia. By the works of the Solovetsky monks, who gave the Russian Church a whole galaxy of holy ascetics, the Christian faith spread to the endless expanses of the Russian Circumpolar.
The activity of the Solovetsky Monastery decisively contributed to the influx of the Russian population into the White Sea region. The marine monastery had a decisive impact on the development of the economy and the marine part of Pomorye. This was a prerequisite for the development of the Arctic Ocean by Russian seafarers and the transformation of Russia into a great Arctic state.
The geographical remoteness of the Solovetsky Monastery makes it difficult for many to take a personal part in the events taking place there. Shooting such events in the 360 ° video technology and providing access to them via the Internet allows, when using virtual reality gadgets, to feel like a participant in such events.
As a result of the work done, more than 10 scenes were filmed on the Solovetsky Archipelago: Excursion to the Ascension Church on Sekirnaya Hill, Service in the temple of the Solovetsky Monastery, Christmas Liturgy, All-night Vigil, Children's Christmas Festival in the Solovetsky Monastery, Monastic caroling, various kinds of monastery and Great Solovetsky Island.
Service in the temple of the Solovki monastery in 360 ° video format
Excursion to the Church of the Ascension on Sekirnaya Hill (Solovetsky Islands) in 360 ° video format
Interactive video format 360 °
Despite the fact that viewing 360 ° surround video in various virtual reality headsets provides high-quality virtual immersion in video content, the next step is the possibility of introducing various interactive elements into the video format of 360 ° video.
Such elements may be:
• Active tags inside the virtual space for movement along various trajectories previously captured in 360 ° video technology;
• Introduction to 360 ° video of various additional content (images, video, hyperlinks, etc.) - the function “picture in picture”;
• Transition from video image in 360 ° video format to simulated 3D space of reconstructed reality.
On the basis of the Unity3D system, an interactive tour in the 360 ° video format by the Korela Fortress Museum was implemented at the ITMO University Center for Design and Multimedia. This panoramic video tour with interactive controls is designed for viewing in the Oculus Rift virtual reality headsets.
Interactive interaction allows you to select the route: the user, at certain points - forks, is given the opportunity to choose the desired continuation of the tour, or go back. Hovering on an element is done by turning the head, which is tracked using the Oculus Rift. When you hold the sight on the selected element for several seconds, the element is activated and the next 360 ° video segment is launched. To increase the visibility of the interactive interaction, when you hover the sight, the animation of filling the circular loading bar starts.
Map of interactive travels in the Korela Fortress Museum
At the aisles “forward” there is a professional guide, telling about the objects on the passable part of the path, the guide is no longer present on the aisles back so as not to duplicate the same information. If desired, the user can skip the video section by pressing a key on the keyboard or by using an interactive element.
The second form of interactive interaction is the ability to switch from a 360 ° video to a virtual three-dimensional reconstruction of the fortress. The reconstruction of the Korela fortress was carried out as part of the project “The Multimedia Information System“ Ancient Fortresses of the North-West ””. At certain points of the video tour, an element appears, activating which the user moves to a 3D reconstruction, with the possibility of free movement in the virtual reconstruction space using the Unity3D graphic engine. In the places of virtual reconstruction, coinciding with the real recorded static points-marks, the reconstruction contains luminous portals. Entering such a portal, the user returns to the appropriate point of the video tour.
Conclusion
360 ° multimedia technology is a new promising technology direction that allows you to create high-quality interactive content for virtual reality headsets in all areas of human activity (culture, art, education, science, tourism, advertising, industrial production, public administration, etc.) .
From the point of view of the entertainment industry and, especially, the creation of feature films in the 360 ° format, a number of new approaches and methodologies of creative and technological processes are expected to develop, which will allow the viewer to dive into the new virtual and interactive levels.
Authors of the article: Nikolay Borisov, Artem Smolin, Denis Stolyarov, Pavel Shcherbakov, Vasily Trushin
In the English version, the materials will be presented in May at the international conference “ArtsIT, Interactivity & Game Creation” (Esbjerg, Denmark). Here is the Russian, slightly adapted original.
')
Virtual reality and its application
Currently, virtual reality technologies and their applications are rapidly developing. Great opportunities are opened by the use of such technologies in culture and art , as they provide the effect of the presence of the viewer in the center of cultural events. One of the most promising and currently being actively developed approaches to creating content for virtual reality systems is the 360 ° video technology, which allows you to create panoramic videos with varying degrees of interactivity, where the viewer controls the viewing angle of the video at will. Such a video can be viewed both in a virtual reality helmet (for example, in the Oculus Rift , Samsung Gear VR , HTC Vive , etc.), and with the help of a special application on the smartphone, while the picture changes in accordance with the user's head turns. You can view 360 ° video on the display of a personal computer. In the latter case, the user controls the view using the mouse or keyboard.
Watching video in 360 ° format in the Samsung Gear VR headset
Many large companies (Facebook, Nokia, Samsung, Google, etc.) are currently developing cameras for shooting video in 360 ° format, virtual reality headsets for various smartphones and desktops, as well as various recording devices that provide surround sound (binaural). a) sound and allowing to realize the whole complex of “multimedia 360 °” technologies.
The use of 360 ° video technology in education is especially important. Interactive video 360 ° can provide interaction with the end user, so this method can be widely used to train specialists in various fields where the development of practical experience requires a presence at the center of the event. The 360 ° video technology allows you to analyze the teamwork of staff while performing various tasks, which leads to an increase in the quality of the educational process.
A significant role is played by a 360 ° panoramic video to preserve and ensure access to cultural heritage. This technology provides a unique opportunity to see with the immersion effect not only closed archives of museums, but also reconstructed historical monuments that were destroyed by time or circumstances (three-dimensional virtual reconstruction). In addition, virtual reality is a unique opportunity for people with disabilities to travel and walk around the world.
Speaking of entertainment in the aspect of technology video 360 ° - viewers can visit the largest cultural festivals, find themselves on stage with their favorite musicians or watch a theatrical performance from the royal box.
The ITMO University Design and Multimedia Center is implementing a project for the development and application of 360 ° video technologies in the arts and culture Video360Production.com . Within the framework of this project, it was possible to achieve certain results necessary for obtaining high-quality content, starting with the development of original video recording devices 360 ° and sound and ending with the development of software for solving a number of tasks in this area.
360 ° multimedia technology
At the moment, special cameras are being developed with the help of which it would be possible to shoot video in 360 ° video format. However, with the help of special tools and software, you can use existing cameras to shoot 360 ° video.
A vivid example of this approach is a number of projects offering special rigs for GoPro HERO 4 cameras. The number of cameras in the riga is from 6 to 14 (stereo versions), depending on the tasks. After shooting, video streams are stitched into the general video sphere with the help of separately supplied programs.
In addition, several large IT companies and electronics manufacturers announced their own fully automated solutions for shooting content for virtual reality - Google Jump + GoPro Odyssey , Samsung Beyond , Nokia Ozo .
GoPro Odyssey, Nokia OZO and Samsung Project Beyond
We have described our personal impressions of the Nokia OZO camera here .
The 360 ° video is a new and progressively developing direction that does not yet have standards of direction, which provides a wide scope for creativity and the development of new ideas and approaches. The effect of full immersion is achieved by a camera with multiple lenses, which allows you to shoot everything that happens around. However, this entails many difficulties, because many laws of direction do not apply when shooting in 360 ° video format. The main cardinal difference in viewing 360 ° video in virtual reality headsets is the fact that the viewer has the opportunity to watch content around himself, which means that the director will have to give the viewer the ability to dynamically select the viewing angle, rather than focus on a specific part of the screen.
The solution to this problem is the detailed reasoning of the scenario from the point of view of the paradigm shift - the departure to 3D video space. Each shooting requires both standard preparation (setting up the light, the most advantageous points for the camera, scenario and location of the scene), and non-standard approaches (special and often unique for each camera mount, direction taking into account the features of video recording 360 °, correct camera orientation, surround sound recording). When recording concerts, to achieve greater realism, the sound from the camera or from the stereo microphone is often combined with the sound from the director's console.
Representation of cultural events using 360 ° multimedia technology
The video360production.com collection of video footage for virtual reality headsets includes more than fifty different videos: concerts and music festivals, excursions to historical places, excursions in museums, theatrical performances, various cultural events and technological processes.
Group "Bravo", "Love does not burn" in the format of panoramic video
Grouping "Leningrad" in the format of panoramic video
Technology of laying asphalt "Ecohiter", panoramic promotional video
When recording concerts, it is necessary to transmit in high quality both the image and the sound. Moreover, it is necessary to think about the location of the camera, since when recording to a 360 ° camera, such concepts as “front” and “background” become irrelevant. In addition, sometimes the scene was simultaneously up to twenty people, which complicated the installation of the camera. The footage allowed the musicians to look at themselves from the outside, to evaluate the location of the musicians during the concert, as well as to see the public reaction to the performance. The spectators, in turn, with the help of this survey were able to examine in detail the play of each musician and immerse themselves in the atmosphere created by the orchestra or group.
Concert of the group "Picnic" in the format of panoramic video
During the shooting of the famous Russian group "Picnic", the following problems arose - the concert was held in a dark room of the concert hall under the bright light of spotlights. These conditions are complex for ordinary cameras, since even at maximum resolution, the transmission of realistic color depth is quite a challenge.
Video recordings of concerts in the 360 ° format have a huge advantage over standard video recordings, as the viewer feels the effect of being present in the stage space and has full autonomy to decide for himself where to look at the moment — whether to watch the guitarist or look at the soloist, or maybe see concert "eyes" of musicians, turning his back to the stage, and watch the lively reaction of the public.
To achieve the effect of complete immersion, not only a high-quality picture is necessary, but also a “live” sound. For musicians and frequent visitors to concerts, it is not a secret that the “technical” sound on the stage is far from what is heard in the hall. Sound engineers of the project combined sound from a stereo microphone with a studio recording. As a result, the sound remains of high quality, but it retains its “liveliness”: while watching the recording you can hear how the audience reacts and sings.
In addition to musical groups, the performers of the project video360production.com cooperates with the largest St. Petersburg theaters, where a number of performances and musical performances were shot. A unique opportunity to provide shooting historical sites in cities and excursions in museums. In the Museum of Anthropology and Ethnography. Peter the Great (Kunstkamera) in a 360 ° video format, before the reconstruction began, a large academic ( Gottorp ) globe was filmed - a diplomatic gift to Tsar Peter I from the Duke of Golstinsky Karl Friedrich during the Northern War.
Shooting a large academic (Gottorp) globe in 360 ° video format
The implementation of a joint project of the ITMO University and the State Hermitage Museum “Points of Presence” has begun, within which it is planned to present events in various areas of the museum using panoramic video filming technologies.
Shooting in the State Hermitage the video clip “Peacock Clock” in 360 ° video format
The virtual walk along the rivers and canals of St. Petersburg is also of heightened interest among the visitors of the YouTube channel created by the authors of the project.
Particular attention was paid to filming a 360 ° video of various events in the Solovetsky Monastery. The Solovki monastery played an exceptional role in the history and culture of Russia. Originated in the XV century. as a place of desertification of spiritual followers of the "father of Russian monasticism" Sergius of Radonezh, in a short time, he became the spiritual, cultural, political and economic capital of the North of Russia. By the works of the Solovetsky monks, who gave the Russian Church a whole galaxy of holy ascetics, the Christian faith spread to the endless expanses of the Russian Circumpolar.
The activity of the Solovetsky Monastery decisively contributed to the influx of the Russian population into the White Sea region. The marine monastery had a decisive impact on the development of the economy and the marine part of Pomorye. This was a prerequisite for the development of the Arctic Ocean by Russian seafarers and the transformation of Russia into a great Arctic state.
The geographical remoteness of the Solovetsky Monastery makes it difficult for many to take a personal part in the events taking place there. Shooting such events in the 360 ° video technology and providing access to them via the Internet allows, when using virtual reality gadgets, to feel like a participant in such events.
As a result of the work done, more than 10 scenes were filmed on the Solovetsky Archipelago: Excursion to the Ascension Church on Sekirnaya Hill, Service in the temple of the Solovetsky Monastery, Christmas Liturgy, All-night Vigil, Children's Christmas Festival in the Solovetsky Monastery, Monastic caroling, various kinds of monastery and Great Solovetsky Island.
Service in the temple of the Solovki monastery in 360 ° video format
Excursion to the Church of the Ascension on Sekirnaya Hill (Solovetsky Islands) in 360 ° video format
Interactive video format 360 °
Despite the fact that viewing 360 ° surround video in various virtual reality headsets provides high-quality virtual immersion in video content, the next step is the possibility of introducing various interactive elements into the video format of 360 ° video.
Such elements may be:
• Active tags inside the virtual space for movement along various trajectories previously captured in 360 ° video technology;
• Introduction to 360 ° video of various additional content (images, video, hyperlinks, etc.) - the function “picture in picture”;
• Transition from video image in 360 ° video format to simulated 3D space of reconstructed reality.
On the basis of the Unity3D system, an interactive tour in the 360 ° video format by the Korela Fortress Museum was implemented at the ITMO University Center for Design and Multimedia. This panoramic video tour with interactive controls is designed for viewing in the Oculus Rift virtual reality headsets.
Interactive interaction allows you to select the route: the user, at certain points - forks, is given the opportunity to choose the desired continuation of the tour, or go back. Hovering on an element is done by turning the head, which is tracked using the Oculus Rift. When you hold the sight on the selected element for several seconds, the element is activated and the next 360 ° video segment is launched. To increase the visibility of the interactive interaction, when you hover the sight, the animation of filling the circular loading bar starts.
Map of interactive travels in the Korela Fortress Museum
At the aisles “forward” there is a professional guide, telling about the objects on the passable part of the path, the guide is no longer present on the aisles back so as not to duplicate the same information. If desired, the user can skip the video section by pressing a key on the keyboard or by using an interactive element.
The second form of interactive interaction is the ability to switch from a 360 ° video to a virtual three-dimensional reconstruction of the fortress. The reconstruction of the Korela fortress was carried out as part of the project “The Multimedia Information System“ Ancient Fortresses of the North-West ””. At certain points of the video tour, an element appears, activating which the user moves to a 3D reconstruction, with the possibility of free movement in the virtual reconstruction space using the Unity3D graphic engine. In the places of virtual reconstruction, coinciding with the real recorded static points-marks, the reconstruction contains luminous portals. Entering such a portal, the user returns to the appropriate point of the video tour.
Conclusion
360 ° multimedia technology is a new promising technology direction that allows you to create high-quality interactive content for virtual reality headsets in all areas of human activity (culture, art, education, science, tourism, advertising, industrial production, public administration, etc.) .
From the point of view of the entertainment industry and, especially, the creation of feature films in the 360 ° format, a number of new approaches and methodologies of creative and technological processes are expected to develop, which will allow the viewer to dive into the new virtual and interactive levels.
Authors of the article: Nikolay Borisov, Artem Smolin, Denis Stolyarov, Pavel Shcherbakov, Vasily Trushin
Source: https://habr.com/ru/post/392565/
All Articles