Behind the scenes of virtual reality from Nike

Hello! While he continues to slowly but surely prepare the material about hromakey, he decided to bring to your attention a translation of an interview about how the high-tech Nike Hypervenom commercial was created. This video will allow you to stay in the shoes of a famous football player and score the decisive goal during a real match. But the most important thing is not just shooting with a view from the eyes - here you can twist your head! Actually, the video itself:



And the story about how this video was created under the cut.

This is a translation of an English-language article :
')
Legend of Brazilian football Neymar has become a star in the video attraction for Nike Hypervenom Phantom boots. In the video, we are right out of the eyes of a football player as the striker runs to the goal and scores a goal - this stunning project was made possible by the IM360 (platform for working with panoramic video), as well as the interaction of Digital Domain and Immersive Media. Directing - Adam Berg, agency - Wieden + Kennedy.

- How did you start by starting such a large project?

Chris Fieldhouse (visual effects producer): We had a lot of tasks to solve. Will it be a view from the eyes or from a third person? How can this be done? Director Adam Berg never worked with a panoramic shot. So we all dived into the project and began to think how to implement it. It was a truly collective project. We had to carry out this idea from the creative concept to the implementation, thinking of how to do everything from the technical side.

- What were the difficulties in working with what the observer sees?

Janelle Kroshaw (Visual Effects Producer, Studio Studios ): The first step, of course, was to create a superior visualization. Previously, we created the “Evolution of Verse” movie, which was completely computerized, and also the video for Björk, which, on the contrary, consisted entirely of real shots - that is, we already used these tools from both sides. First, we rendered a 360-degree transvisualization. Adam had various ideas related to choreography and Neymar's attack, so that the revival helped to get closer to the final result.

Fieldhouse: We had to decide whether it would be hromakey or, nevertheless, a real stadium. Will we shoot it in 2D, and then convert it into a panorama, or will we immediately do everything in a panorama? As a result, the decision became a kind of fusion of all options. We had a football coordinator who helped us plan and stage all moves - Nike wanted everything to be as realistic as it can only be in a real football game.



Kroshaw: With the help of our employees and bowling pins, we estimated the approximate choreography of the players' movements, then we rendered it in the form of a panoramic video, after which Adam was able to see everything in glasses of virtual reality. It was a great way to plan all our follow up actions. Gradually, the previsualization team grew and ended up with the fact that the staff helped each other to hone football moves and feints, then transferring them to the previsalization.

Fieldhouse: In the office, we saw a lot of funny moves - people came out, put helmets on their heads from the camera and ran, kicking a soccer ball and wondering how all the animation might look like. We viewed the whole scene from absolutely all angles.

Kroshaw: Despite the fact that we roughly understood what construction we would need for filming, we still faced a lot of questions. We tried many different options, went to the football field and conducted quite funny tests. The main thing that had to be understood was how to sew the shooting from all the cameras into a single panorama. Remove all the six cameras mounted on the mount "unicorn" (cameras are mounted on a kind of monopod protruding from the head forward along the line of sight) or it will be something like a helmet with cameras around the head. We had to find a solution that would give us the best resolution and ability to “stitch” video clips.



- And how did you solve the problem?

Fieldhouse: We needed to make a video of the highest quality possible - we didn’t just want to take a bag of GoPro and take it all down. We spent many, many, many months planning and testing the best combination of cameras and mounts. We stopped at a hybrid 2D shooting on the Alexa camera and gluing together the shooting made by an array of GoPro and Codex cameras. We used a camera suspended on a cable over the field. We also used several Red cameras located at the edges of the stage.

Kroshaw: In the end, we used 15 cameras: Alexa, Codex, three Red Epic and nine GoPro.
Fieldhouse: We used a 360-degree array of cameras mounted on trails; there was also a 360-degree array of cameras attached to Neymar’s head. It was very difficult shooting - only three days of direct shooting, plus two days of preparation.

- Did you make quick frame gluing on the spot to see if you are moving in the right direction?

Kroshaw: Yes, of course. IM360 is just great! We looked through the jogging Neymar, taken from the helmet. There was a huge number of duplicates. And the guys from IM360, even if not instantly, but in a few minutes we sewed everything into a single panorama, then we put on virtual reality glasses and immediately found ourselves in the place of our hero, who scores a goal. Right on the spot, we received feedback from Adam and the guys from the agency, and then shot the next take. Another difficult task was the repetition of all the movements of Neymar by our suspended structure.

Fieldhouse: This has become a serious technical challenge for our team. That is, first we shot the panorama from Neymar's helmet, then to ordinary 2D cameras, and then we had to glue all this together quickly and see if everything goes according to plan ... Wow, the problem!

The IM360 platform includes VR Toolkit - this is an instant preview, that is, you can look at what is happening directly during the shooting. We really wanted to use this tool, but unfortunately Neymar could not run around the field with oblique wires behind him, so circumstances did not allow us to do everything in real time. But still, within a few minutes, we easily got a rough panoramic shot.



Kroshaw: In my 17 years in the visual effects industry, this was the project with the most tracking and compositing. Tracking tools turned out to be truly invaluable. When we finished, we had countless duplicates from cameras attached to the cables, because shooting them looked like they were shot by an amateur - managing the entire structure was extremely difficult, so to get a good shot we had to do the same thing over and over .

As soon as post-production began, our first task was to make a rough assembly of the entire movie, so the first thing we needed to do was tracking the main camera (calculating the camera's trajectory in three-dimensional space using contrast points in the video) —it was pretty simple. The main trick was to glue the filming from the main camera with the filming from the helmet - after all, in the final video we needed Neymar's body, and we could only take it from the filming with GoPro cameras. We could not use the material from these cameras for the whole environment, as the quality of the shooting was insufficient, plus the image jumped too much and shook. I will say more, when viewing this material just really rock. Well, if you look only at your feet, then in general this is perceived normally, but now if you start to turn your head around, then immediately seasickness.

Using the image from the camera on the tracks, we managed to achieve such a smooth image. What you can see now is a hybrid, and each of these players was “cut out” from the image into a separate element, from which we then collected all this. It was a huge number of frames, because each player had 90 seconds of “on-air” time with 60 frames in each.

And when we combined all the parts into a single whole, the time came to add graphic effects. In parallel with the work of gluing video, there was work on the creation of the stadium. The fans were filmed separately. We got 50 different templates, which we, by analogy with the work on the films “Living Steel and“ Rock for Ages ”, had to impose on separate planes and, using Nuke software, multiply 2.5D throughout the stadium. This was the original plan, but we shot the panorama, and all the distortions peculiar to it miraculously turned our 2.5D plan into dust. So we had to use Houdini and do the same thing in 3D.

The video contains a huge amount of detail, starting with the player himself and beyond: the players, the coach, the judges, the crowds - all this is a real shooting without graphics. All the surroundings, the airship, all the effects and the grass are computer graphics. The ball itself is either real or computer - in some places it was too oiled, and we had to replace it with a computer model. It was the same with Neymar's body - on some frames the shooting with GoPro didn’t look very good, so sometimes the computer graphics skip there.



- How did you insure yourself that the viewer did not look in the wrong direction? Would you like to limit the range of possible movements and smooth them out for this?

Kroshaw: Yes, we thought about it, because, it is unlikely, a footballer can run forward with his head turned 180 degrees back. But despite this, in the video you can turn your head at all 360 degrees - all the same while running forward, and the viewer himself wants to look ahead, so the restriction is perhaps not needed.

Fieldhouse: Smooth camera movements were part of our tests. We tried several times to make the video exactly the way the viewer would see it, whether the camera was simply attached to the player, but it looked like you were not playing football, but just springing on someone's head. Therefore, we used a camera that could give us a smooth trajectory of movement, and then simply combined the shooting and got a smooth movement. It was quite funny, because we constantly thought that it was necessary to get rid of jerks as much as possible, but then, almost at the end of the process, a client came to us and asked to add more shaking. So, to summarize, we made an extremely smooth image, and then added artificial jerks of a player running across the field.



When you watch a movie from Neymar's eyes, you automatically try to focus on the ball. But since we had a panoramic video, we wanted to be sure that if someone did not look at the ball, something interesting would happen from all sides.

Kroshaw: In the original version, the people in the stands were rather dark, and in the 360-degree version they became almost completely incurable. Therefore, everything ended with the fact that we added luminous spotlights, so that the viewer had something to look at.

Kroshaw: Fast and sharp movements of Neymar's arms and legs are mainly real shots.

Fieldhouse: Actually, this was one of the main reasons for our concern, as we understood that in the real world we couldn’t attach six Alexa cameras to a football player’s helmet, so we deliberately put this quality limitation on hold. GoPro specifications were far from ideal, that is, we understood that motion blur, shutter speed and everything else - this is what we will have to seriously work with later. Despite the fact that most of the material is actually a real shot, we created the graphic double of Neymar, which we used in just a few points in order to improve the quality of the final material. Mostly this is true for boots - in the end, after all, customers wanted to see them when they came to us to order a video .