Interactive video and what it eats
Six months ago, over a cup of coffee, the programmer, the director, and the game designer had a friendly conversation on the subject of cinema becoming boring. “Yes, the special effects are getting steeper, the amount of D becomes more, but this is now a little who can be surprised. People cling plot. Now imagine that someday we will watch movies that you can control? ”
“I would kill John Connor.”
“I wouldn’t have thrown away my elder wand and become a bad wizard.”
“And I would use the flywheel of time on the spot of Potter to save his parents, and not klyuvkryla.
And, as they say, it started ...
')
Last week we asked Habrazhiteli whether it was interesting to learn about the technical aspects of creating a game. Now we are ready to present the fruits of our work and tell us what difficulties we faced, how we struggled with them and what results we got.
Links to demo and production versions are at the bottom of the article.
Objective : to design an interactive video (in other words, to pull the Heavy Rain and Mass Effect experience on the video)
Immediately answer your question: “Why on the video? Why not set the game environment and complete freedom of action? "
We have come to the conclusion that there are already such games, and nothing gives a greater presence effect than high-quality video, and you will not get stuck in textures. We decided to run on the web and not be confronted with the problem of releasing the product on DVD and the other delights of gamedev. Moreover, the quality of the Internet connection for the current day in most cases allows you to watch full hd videos without prior download.
To create the MVP, we decided not to fence several plot lines and took the path of a single storyline divided into specific checkpoints with dead-end branches.
Work with management
Since all of this should be public and work in the browser, our choice fell on JavaScript (“only JavaScript, only hardcore!” © our programmer).
The QTE-model of management was chosen, similar to the management of the (stressful) scenes in Heavy Rain: at a certain point in time, buttons / series of buttons appear that you have to press. I managed to press - well, I did not have time - I lost some of my health / self-esteem. If the level of health falls to 0, a transition to a dead-end path will occur at the nearest checkpoint.
It was decided to borrow elements of the game interface from desktop and console games. As a result, we have: the avatar of the main character, his emotions, thoughts, pulse and health.
Before us it became necessary to display a lot of custom things.
Using html in our case implies a large load on the DOM , each update of which caused restructuring and redrawing of all objects, and, sooner or later, we would have rested against the computer resources.
It was decided to change the output tool and use Canvas , which allows us to do custom things and redraw them when we need to. At the moment we are faced with the fact that there are many static elements on one canvas, which are rarely updated, and elements that are updated very often. And when you redraw a frequent element, heavy elements can be redrawn. In the future, we think either to raise 2 canvases (for frequently changing and rarely changing), or rarely changing to make html (the load on the DOM will be minimal).
Controls used by us:
1) scrolling the mouse - formally responsible for running the main character on the video. If you do not turn the mouse wheel for 2.5 seconds, the health / self-esteem will start to fall very quickly.
2) The keystroke is responsible for some actions of the main character on the video (the control model resembles a typical shooter: WASD - directions, E - action, SPACE - jump, etc.)
• Incorrectly pressed / not pressed key - takes away the level of health. A certain time is given for pressing the key.
• Enter a series of keys
• Enter individual phrases and sentences
About keystrokes it is worth mentioning that the reaction of all people is different.
Solution : we collect statistics on the delay time at which the keys are pressed (from the moment the key appeared to the moment it was pressed). Every time we need to show the next key, we show it with an average advance for the user. As a result, the user presses a key almost at the moment when it is necessary to perform an action. The very first key in this approach does not solve anything and serves to initialize anticipation (almost as in sapper: the first field can never be a mine). But you need to press this key so that further control is comfortable. For this feature had to enter the weight of the keys. And in the first series of keys in the first video, the maximum weight (incorrectly pressing / not pressing the key immediately resets the health level to zero. In our case, it is easier to send the user to a dead end and to the replay than to allow him to play with uncomfortable, late control) Naturally, the control model does not depend on the language keyboard layout: an event is waiting for a key, not a letter.
In the future, we plan to expand the control with a mouse on a given trajectory, as well as add the ability to interact with several objects (for example, the frame goes into slowmotion and in a few seconds you need to select an object from those presented in the frame).
The entire control model is superimposed on the video by triggering a specific event at a particular point in time.
Work with video
When working with browsers, we encountered some limitations, for example, in Google Chrome, you can simultaneously preload 10 video files (we didn’t fit into this restriction even with our first MVP). Due to the fact that Safari is the same Web Kit, the behavior in it is very similar. For Firefox, the situation was similar. I had to look for another solution.
Option 1 . There is a pool of subject and dead-end video tracks. While the user interacts with the current storyline, in the background, the next storyline and deadlock are preloaded. As soon as he reaches the checkpoint (a fraction of a second before the end of the current video), we look at how cool the user has played and show him either the ending of the current track, or let it go further along the plot. All that is no longer needed is deleted.
Total : 1 file is playing, two in the preload. Previous deleted.
Pros : Easily sync footage (done at the video production level, when the last frame of the current track is the beginning of the next storyline and stub branch)
Minuses : Easy frieze at the place of transition from one track to another.
Option 2 . One long storyline and a dead end pool. A checkpoint is defined by a certain timing, on which the transition to a dead-end branch is made. In the background, the next deadlock is preloaded.
Total : 1 is lost, 1 is preloaded. Removed only previous dead ends.
Pros : There is no frieze when the user moves along the storyline.
Minuses : It is almost impossible to synchronize the frame of transition to a dead end, because the event of a change of position in the video file comes at some intervals that we can not influence in any way. As a result, the transition to a dead end does not begin with the desired frame, and even with the frieze.
Option 3 : All video tracks in one file, synchronized by transition time (in one large file, all videos are played in parallel at one time). Using WebGL, we will be able to display a portion of this file that is equal in size to one particular video file inside. It sounds scary, seemingly awkward, but still want to try in the near future.
Pros : In theory, the free model: the transition from one part from video to another is one frame of drawing and at 60 fps it is 1/60 of a second.
Minuses : The size of the resulting file is not clear yet and how browsers will react to it.
The question arises how to work with sound, most likely you will have to conjure over its synchronization. In the above options, the sound is sewn to the video tracks.
The main disadvantage of all three approaches : if the user has made critical errors at the beginning of the current storyline, then he will continue moving along the plot to the nearest checkpoint with zero health.
Solution : we shorten the plot branches as much as possible so that the “retribution” for the committed or imperfect action is immediate. At the moment, some tracks have a timing of about 5 seconds. The optimal timing of a single video track after testing on the focus group of about 1,000 people was 15–20 sec.
Yes, looking at it, you will say: “ Here and does not smell like managing a video series, the maximum that it will pull is a pseudo-control ”.
It really is. No matter how hard we try, we can’t talk about complete freedom of control over video. For cinema, in our opinion, absolute freedom of control is not required. What is the difference that you give the main character for breakfast, buckwheat or oatmeal? Put sneakers or sneakers on him. By management, we mean key actions that can affect the course of the plot: there should be a lot of them and they should be very dynamic. Every minute chases, fights, shootouts, the choice of dialogue phrases - in one word Hardcore , only with a joystick in hand. Managing melodrama is not for this approach.
As a result, I chose Option 1 for working with video.
From the point of view of production from the post production, the most difficult part was to hide the “teleportation” of the hero from location to location and some computer graphics. The number of assembly joints in the 5-minute video was more than 20. Separate attention was paid to the sound: every sound was recorded on the site. The number of parallel audio tracks is more than 100.
Some server history
Initially, we tried to run on the web under the hardware we had, and watched the slideshow. Taking into account the expected load, the question arose of starting a content server that is designed to deliver large amounts of video with a good channel. Plus, the cheapest traffic is from the CDN , and the video sequence is loaded to the user from the physical server nearest to it. As a result, all the statics was created under the CDN, the site, scripts and statistics work under a different server.
Production
It was a test of willpower, fortitude, limits of physical and psychological capabilities of a person. At the shooting was assigned a week. At this time, included: writing a script, location scouting, search for extras, shooting themselves. On average, the shooting day lasted 18-20 hours.
By the way, the pedometer of the main character did not digest such a volume of data and died. Runtastic producer, who ran 5 times less than the main character, showed 91 km of the distance covered in a week.
Some scenes were simple, some demanded colossal preparations and were shot in pieces on different days. Some of the scenes were written blindly without playback . For example, there was only 5 minutes to go to the stage with Kaspersky: 6 takes in a row with the hope that there would be a shot and then select the best one from the available ones. The weather contributed 5 kopecks: suitable flight conditions were needed for filming the intro. As a result, they waited for them on the 6th shooting day. There was only 20 minutes on the stage until the fire was set. As a result, he sat down and we took off with our hands. Also problems with copters were waiting for the last scene. According to the scenario, at one point in time, in a confined space surrounded by tons of metal structures, five copters were to fly. The sensors went crazy, the copters threw from side to side. And on the shooting of this scene, we also had 20 minutes in view of the tests (the largest hexacopter blew one battery per takeoff, and in between times, with its air flow, scattered smaller copters). Benefit at the helm were piloting aces (students of Innopolis University, who are engaged in robotics)
Testers go to paradise out of turn
This video haunts them in nightmares. They know what a man of the crowd was wearing, sitting on an orange pear-chair on the left of the last but one row on a frame with a timing of 1:55 min (do not look for: blue T-shirt and blue pants). They know how many doors were open, how many steps the main character took, how his pulse changed, and so on. The number of replays exceeded 3000 times in 2 weeks, which is approximately 250 hours of viewing and gameplay. The moments of the appearance of the keys were set with an accuracy of 3 decimal places. Shifting the timing of one event for a split second could break the next series of buttons while waiting. Therefore, a lot of attention was paid to testing the control model. In short, testers can pass this game without losing a single unit of self-esteem ... With one hand ... Blindfolded ...
Innopolis University
Special thanks to the University of Innopolis , which supported our development and helped bring them to life. Before that, we had 2 products that were not posted on the network, lived only in localhost and waited in the wings. After their presentation at the university of the future, the idea of ​​a new product was born. In the end, having crossed game development, video production, programming and cinema, we helped the IT university team show selection as exciting as no one in the world had done before.
In the near future
We plan to refactor and rewrite to TypeScript , since it has support for objects and classes, which is absent in JavaScript and in the latter implemented through crutches, plus we will get high in terms of performance code.
Plan launch under mobile browsers and touchscreen control options.
Now we are preparing a pilot with a timing of about 40 minutes, several storylines and various “buns”. Plans for the distant future potestit camera, which writes the video in 360 and adapt the product for vr points.
Here are the promised links
Demo version (for which we were almost killed in reality. Valiant police thought that a real abduction was taking place. Responding to 4 armed clowns in business suits on a tinted transporter at 4 am was lightning fast. Fortunately, after 20 minutes of discussion, Kazan police officers understood reacted to startups from IT Park). On the basis of this tie, we are developing the current full-length project .
Be IT Hero is a production version developed for Innopolis University . Where, as in the city of the future, try to launch similar technologies?
Thanks for attention!
“I would kill John Connor.”
“I wouldn’t have thrown away my elder wand and become a bad wizard.”
“And I would use the flywheel of time on the spot of Potter to save his parents, and not klyuvkryla.
And, as they say, it started ...
')
Last week we asked Habrazhiteli whether it was interesting to learn about the technical aspects of creating a game. Now we are ready to present the fruits of our work and tell us what difficulties we faced, how we struggled with them and what results we got.
Links to demo and production versions are at the bottom of the article.
Objective : to design an interactive video (in other words, to pull the Heavy Rain and Mass Effect experience on the video)
Immediately answer your question: “Why on the video? Why not set the game environment and complete freedom of action? "
We have come to the conclusion that there are already such games, and nothing gives a greater presence effect than high-quality video, and you will not get stuck in textures. We decided to run on the web and not be confronted with the problem of releasing the product on DVD and the other delights of gamedev. Moreover, the quality of the Internet connection for the current day in most cases allows you to watch full hd videos without prior download.
To create the MVP, we decided not to fence several plot lines and took the path of a single storyline divided into specific checkpoints with dead-end branches.
Work with management
Since all of this should be public and work in the browser, our choice fell on JavaScript (“only JavaScript, only hardcore!” © our programmer).
The QTE-model of management was chosen, similar to the management of the (stressful) scenes in Heavy Rain: at a certain point in time, buttons / series of buttons appear that you have to press. I managed to press - well, I did not have time - I lost some of my health / self-esteem. If the level of health falls to 0, a transition to a dead-end path will occur at the nearest checkpoint.
It was decided to borrow elements of the game interface from desktop and console games. As a result, we have: the avatar of the main character, his emotions, thoughts, pulse and health.
Before us it became necessary to display a lot of custom things.
Using html in our case implies a large load on the DOM , each update of which caused restructuring and redrawing of all objects, and, sooner or later, we would have rested against the computer resources.
It was decided to change the output tool and use Canvas , which allows us to do custom things and redraw them when we need to. At the moment we are faced with the fact that there are many static elements on one canvas, which are rarely updated, and elements that are updated very often. And when you redraw a frequent element, heavy elements can be redrawn. In the future, we think either to raise 2 canvases (for frequently changing and rarely changing), or rarely changing to make html (the load on the DOM will be minimal).
Controls used by us:
1) scrolling the mouse - formally responsible for running the main character on the video. If you do not turn the mouse wheel for 2.5 seconds, the health / self-esteem will start to fall very quickly.
2) The keystroke is responsible for some actions of the main character on the video (the control model resembles a typical shooter: WASD - directions, E - action, SPACE - jump, etc.)
• Incorrectly pressed / not pressed key - takes away the level of health. A certain time is given for pressing the key.
• Enter a series of keys
• Enter individual phrases and sentences
About keystrokes it is worth mentioning that the reaction of all people is different.
Solution : we collect statistics on the delay time at which the keys are pressed (from the moment the key appeared to the moment it was pressed). Every time we need to show the next key, we show it with an average advance for the user. As a result, the user presses a key almost at the moment when it is necessary to perform an action. The very first key in this approach does not solve anything and serves to initialize anticipation (almost as in sapper: the first field can never be a mine). But you need to press this key so that further control is comfortable. For this feature had to enter the weight of the keys. And in the first series of keys in the first video, the maximum weight (incorrectly pressing / not pressing the key immediately resets the health level to zero. In our case, it is easier to send the user to a dead end and to the replay than to allow him to play with uncomfortable, late control) Naturally, the control model does not depend on the language keyboard layout: an event is waiting for a key, not a letter.
In the future, we plan to expand the control with a mouse on a given trajectory, as well as add the ability to interact with several objects (for example, the frame goes into slowmotion and in a few seconds you need to select an object from those presented in the frame).
The entire control model is superimposed on the video by triggering a specific event at a particular point in time.
Work with video
When working with browsers, we encountered some limitations, for example, in Google Chrome, you can simultaneously preload 10 video files (we didn’t fit into this restriction even with our first MVP). Due to the fact that Safari is the same Web Kit, the behavior in it is very similar. For Firefox, the situation was similar. I had to look for another solution.
Option 1 . There is a pool of subject and dead-end video tracks. While the user interacts with the current storyline, in the background, the next storyline and deadlock are preloaded. As soon as he reaches the checkpoint (a fraction of a second before the end of the current video), we look at how cool the user has played and show him either the ending of the current track, or let it go further along the plot. All that is no longer needed is deleted.
Total : 1 file is playing, two in the preload. Previous deleted.
Pros : Easily sync footage (done at the video production level, when the last frame of the current track is the beginning of the next storyline and stub branch)
Minuses : Easy frieze at the place of transition from one track to another.
Option 2 . One long storyline and a dead end pool. A checkpoint is defined by a certain timing, on which the transition to a dead-end branch is made. In the background, the next deadlock is preloaded.
Total : 1 is lost, 1 is preloaded. Removed only previous dead ends.
Pros : There is no frieze when the user moves along the storyline.
Minuses : It is almost impossible to synchronize the frame of transition to a dead end, because the event of a change of position in the video file comes at some intervals that we can not influence in any way. As a result, the transition to a dead end does not begin with the desired frame, and even with the frieze.
Option 3 : All video tracks in one file, synchronized by transition time (in one large file, all videos are played in parallel at one time). Using WebGL, we will be able to display a portion of this file that is equal in size to one particular video file inside. It sounds scary, seemingly awkward, but still want to try in the near future.
Pros : In theory, the free model: the transition from one part from video to another is one frame of drawing and at 60 fps it is 1/60 of a second.
Minuses : The size of the resulting file is not clear yet and how browsers will react to it.
The question arises how to work with sound, most likely you will have to conjure over its synchronization. In the above options, the sound is sewn to the video tracks.
The main disadvantage of all three approaches : if the user has made critical errors at the beginning of the current storyline, then he will continue moving along the plot to the nearest checkpoint with zero health.
Solution : we shorten the plot branches as much as possible so that the “retribution” for the committed or imperfect action is immediate. At the moment, some tracks have a timing of about 5 seconds. The optimal timing of a single video track after testing on the focus group of about 1,000 people was 15–20 sec.
Yes, looking at it, you will say: “ Here and does not smell like managing a video series, the maximum that it will pull is a pseudo-control ”.
It really is. No matter how hard we try, we can’t talk about complete freedom of control over video. For cinema, in our opinion, absolute freedom of control is not required. What is the difference that you give the main character for breakfast, buckwheat or oatmeal? Put sneakers or sneakers on him. By management, we mean key actions that can affect the course of the plot: there should be a lot of them and they should be very dynamic. Every minute chases, fights, shootouts, the choice of dialogue phrases - in one word Hardcore , only with a joystick in hand. Managing melodrama is not for this approach.
As a result, I chose Option 1 for working with video.
From the point of view of production from the post production, the most difficult part was to hide the “teleportation” of the hero from location to location and some computer graphics. The number of assembly joints in the 5-minute video was more than 20. Separate attention was paid to the sound: every sound was recorded on the site. The number of parallel audio tracks is more than 100.
Some server history
Initially, we tried to run on the web under the hardware we had, and watched the slideshow. Taking into account the expected load, the question arose of starting a content server that is designed to deliver large amounts of video with a good channel. Plus, the cheapest traffic is from the CDN , and the video sequence is loaded to the user from the physical server nearest to it. As a result, all the statics was created under the CDN, the site, scripts and statistics work under a different server.
Production
It was a test of willpower, fortitude, limits of physical and psychological capabilities of a person. At the shooting was assigned a week. At this time, included: writing a script, location scouting, search for extras, shooting themselves. On average, the shooting day lasted 18-20 hours.
By the way, the pedometer of the main character did not digest such a volume of data and died. Runtastic producer, who ran 5 times less than the main character, showed 91 km of the distance covered in a week.
Some scenes were simple, some demanded colossal preparations and were shot in pieces on different days. Some of the scenes were written blindly without playback . For example, there was only 5 minutes to go to the stage with Kaspersky: 6 takes in a row with the hope that there would be a shot and then select the best one from the available ones. The weather contributed 5 kopecks: suitable flight conditions were needed for filming the intro. As a result, they waited for them on the 6th shooting day. There was only 20 minutes on the stage until the fire was set. As a result, he sat down and we took off with our hands. Also problems with copters were waiting for the last scene. According to the scenario, at one point in time, in a confined space surrounded by tons of metal structures, five copters were to fly. The sensors went crazy, the copters threw from side to side. And on the shooting of this scene, we also had 20 minutes in view of the tests (the largest hexacopter blew one battery per takeoff, and in between times, with its air flow, scattered smaller copters). Benefit at the helm were piloting aces (students of Innopolis University, who are engaged in robotics)
Testers go to paradise out of turn
This video haunts them in nightmares. They know what a man of the crowd was wearing, sitting on an orange pear-chair on the left of the last but one row on a frame with a timing of 1:55 min (do not look for: blue T-shirt and blue pants). They know how many doors were open, how many steps the main character took, how his pulse changed, and so on. The number of replays exceeded 3000 times in 2 weeks, which is approximately 250 hours of viewing and gameplay. The moments of the appearance of the keys were set with an accuracy of 3 decimal places. Shifting the timing of one event for a split second could break the next series of buttons while waiting. Therefore, a lot of attention was paid to testing the control model. In short, testers can pass this game without losing a single unit of self-esteem ... With one hand ... Blindfolded ...
Innopolis University
Special thanks to the University of Innopolis , which supported our development and helped bring them to life. Before that, we had 2 products that were not posted on the network, lived only in localhost and waited in the wings. After their presentation at the university of the future, the idea of ​​a new product was born. In the end, having crossed game development, video production, programming and cinema, we helped the IT university team show selection as exciting as no one in the world had done before.
In the near future
We plan to refactor and rewrite to TypeScript , since it has support for objects and classes, which is absent in JavaScript and in the latter implemented through crutches, plus we will get high in terms of performance code.
Plan launch under mobile browsers and touchscreen control options.
Now we are preparing a pilot with a timing of about 40 minutes, several storylines and various “buns”. Plans for the distant future potestit camera, which writes the video in 360 and adapt the product for vr points.
Here are the promised links
Demo version (for which we were almost killed in reality. Valiant police thought that a real abduction was taking place. Responding to 4 armed clowns in business suits on a tinted transporter at 4 am was lightning fast. Fortunately, after 20 minutes of discussion, Kazan police officers understood reacted to startups from IT Park). On the basis of this tie, we are developing the current full-length project .
Be IT Hero is a production version developed for Innopolis University . Where, as in the city of the future, try to launch similar technologies?
Thanks for attention!
Source: https://habr.com/ru/post/280372/
All Articles