Photo-realistic graphics in a mobile game or the world's first "video" game (mobile)

A video game, according to Wikipedia, is a game using images generated by electronic equipment.

In our case, the "video" game is a video game based on real video.

If you find this title too yellow and there is a “video” -game in our understanding, made earlier - write about it in the comments and I will correct the title. At that time, we did not find and did not remember similar games.

Update : here I meant mobile games, but thanks to the commentators, the heading has been corrected. Also add that on YouTube at one time video quests were popular. To real videos, by the built-in functionality of YouTube, transitions to other videos were attached, which gave a certain interactivity. First available example: Hostage .

An article can no longer be counted as an advertisement, since Since the development and publication of the game, so much time has passed that my developer account has expired and the game is no longer available in the Apple AppStore.

In the article I will talk about the process of preparing the game and about the features of working with video in IOS and Android on Adobe AIR.

It all started on a bright sunny May St. Petersburg morning. Although no, this does not happen. It all started very much in the second half of the usual May day, when the sky was covered with clouds and we left to shoot a video.

I introduce myself, we are two developers: Andrei is a server programmer, far from game development, who, in fact, planned to make a game, and Fyodor is a professional game programmer who helped with video shooting and development tips.

Although, no, it all started earlier. Somehow the idea to make a game based on real video shooting came to mind. At the same time, the video had to be interactively tied to the game. The video was supposed to be beautiful. The game is both interesting, entertaining and simple. So the idea of ​​PiterJump was born and then everything started to happen.

Preparing to shoot

A script was written which, according to the full list of streets of St. Petersburg (street names returned to normal: Ave., Avenue, Street - Street, etc.) made inquiries (like St. Petersburg "Nevsky Prospect") on Google, received a number of results and thus the rating of street fame was compiled.

It was planned to use the shooting of these streets for the game, assuming that since the street is more famous, then it is more interesting. The idea was untenable. The list has been preserved for history, I cite its beginning (sorted by descending "fame").

As you can see, especially the residents of the city, the streets are not the most significant. For example, Nevsky Prospect, perhaps the most famous central avenue of St. Petersburg, is on the 273th place in the rating with 3.8 million results.

I had to go on the simplest way - to use the shooting of the city center.

Made the route (map of the central part of St. Petersburg on 8 A4 sheets)

We drove through it once with a record of tracking in the navigator

The length of the route is 30 kilometers.

Video preparation

So, we have chosen a bright day, but also so that there is no bright sun, for uniform illumination.

Charge the batteries, stocked up with cassettes.

They took the camera (Canon XH A1), a tripod, installed, fixed.

Special sign

During filming

The plans were to make about 30 minutes of beautiful video, it is easy to cut it into pieces and make levels for the game. The reality was cruel.

The video turned out so much that I had to buy an additional cassette on the way. We received a total of 1 hour and 40 minutes of video.

Beautiful views from the window were not so beautiful in the video, as their eyes saw.

The video turned out just incredibly many uninteresting moments that needed to be cut out and get in the end about 40 minutes of gameplay.

Errors novice operators

Too fast travel speed when shooting. I tried to go at a speed of about 30-40 km. / H. It turned out that it was too fast and gave several video flaws.

First, when the sidewalk is close to the roadway, the relative speed of pedestrians is too fast, which results in a lack of reaction speed to jump over them. During the game, I had to take this moment into account and do an auto-jump.

Secondly, the video turned out with a small, but noticeable, effect of a stroboscope. Probably, this could be corrected by the camera settings, but on the camera screen it seemed that the video looks good, and we are not professional operators. Later on, I tried to fix this for a long time in a video editor, applying various filters. As a result, I stopped at two filters at the same time: a slight horizontal blur (so that motion blurred) and a stabilization filter, which slightly improved the video.

Wrong point selection for shooting. The center of St. Petersburg consists of cars.

More precisely from the machines-machines-machines. All streets, streets, alleys are made by cars. Recently, the city is a little civilized in this direction, paid parking is introduced. But at that time, that was, that was. Yes, the camera was set quite high, but not enough. As a result, the video in some places turned out almost a continuous series of cars, instead of the beauty of St. Petersburg, houses and people. Ideally, it was necessary to shoot at the same height as the cars that shoot Yandex / Google panoramas.

Writing a game

At first, on the advice, I was thinking of writing a game on Unity (then still Unity3D). But Unity had a fatal flaw. After numerous experiments, it turned out that on mobile devices it is not supported to overlay video as texture on the object. There were plugins that supported it. But one was paid, the second one reproduced a video broken into pictures and, with a long duration, the “video” gave an unfit frame rate for the game.

Looked at Xcode. It immediately became clear: to write a game on Xcode, you need to be a super hero or have ready-made ideas that can be used in all games. In fact, you first need to write the graphics engine of the game, then just the game itself. I was not ready for such a feat (besides, an alien language syntax from solid square brackets ...).

I don’t remember how to use Adobe AIR. I knew Flash quite well (I didn’t see such a convenient graphic / animation editor before or after), ActionScript 2.0 / 3.0, but only rumors were heard about AIR - there is such a technology for mobile.

Did you know that Adobe, despite the decline of flash, still regularly releases new versions of AIR, there is a community (and far from 1.5 people) that various plug-ins for AIR write, there are integrations for basic mobile technologies like in-app purchases under Apple / Google, all the necessary integration for Google / Apple gaming platforms?

Armed with Flash Develop + FLEX + AIR SDK, we started new tests. The video was surprisingly easy - it was enough to set the rendering settings using the GPU. Later, the rendering had to be changed to "Direct", since The GPU did not support the overlay on the video images with the alpha channel.

To display the video in AS 3.0, the video loading technique in the Stage is used. Documentation and examples are not always obvious and did not work immediately, but in general, everything is quite simple.

Set the event handler that StageVideo is available.

main.main_stage.addEventListener(StageVideoAvailabilityEvent.STAGE_VIDEO_AVAILABILITY, init_video); 

After checking the availability of StageVideo, the init_video callback is called.

 public function init_video(e:StageVideoAvailabilityEvent):void 

In which a streaming video object is created from a network / file

 var nc:NetConnection = new NetConnection(); nc.connect(null); ns = new NetStream(nc); ns.client = this; 

When StageVideo is available (on IOS, Android part), create an object, bind the streaming object

 stageVideoAvail = (e.availability == StageVideoAvailability.AVAILABLE); if (stageVideoAvail) { sv = main.main_stage.stageVideos[0]; sv.addEventListener(StageVideoEvent.RENDER_STATE, onRender); sv.attachNetStream(ns); } 

When unavailable (on the part of Android, on the emulator on the PC) create a regular video object

 video = new Video(Data.video_width, Data.video_height); video.y = int(Data.height / 2 - Data.video_height / 2); addChildAt(video, 0); video.attachNetStream(ns); 

Next, the stream starts to play.

 var fl:File = File.applicationStorageDirectory.resolvePath(quality + "/level_" + level + ".mp4"); ns.play(fl.url); 

At the same time, you need to set the ViewPort, without which the video is not displayed in StageVideo mode

 public function onRender(e:StageVideoEvent):void { sv.viewPort = new Rectangle(0, Data.height / 2 - Data.video_height / 2, Data.video_width, Data.video_height); 

When working with video in AIR, you need to consider that you need to compile the flash drive or, in the “Software rendering” mode, for testing on a PC in emulator mode. Or in the "GPU rendering" mode - it works on IOS, but does not support the overlay of images with the alpha channel (the video just disappears), and it seems to work fine on Android. Or in the mode "Direct rendering" - works on IOS, but probably gives more power consumption.

During the development of the prototype gameplay, refactoring, alpha version - an article appeared on Habré at that time, in which there was a mention of a certain Adobe Gaming SDK.

This package provides the developer with many useful developments for mobile games. Interface, access to the native IOS / Android API, your 2D and 3D engines.

At that time, the alpha version of the game was already ready, and again there was no point in refactoring the game for using the ready-made components of the 2D engine, so I took only plug-ins of the native IOS functions of accessing the game center and the product store. Plug-ins in AIR are implemented as some ready-made packaged modules ANE (AIR Native Extensions) - which are plain zip archives containing xml files (describing the namespace and access interfaces) and packed compiled swf-files.

After the implementation of the main video display functionality, without which it was impossible to make a game, everything else was not particularly difficult: draw simple graphics, arrange in flash, make animation with built-in tools, write various classes of the game’s business logic and send for approval in Apple AppStore .

Results

At the time of the writing of the game, I thought that the stream of downloads would soon begin, I would have to add servers (the video was loaded into the game on the fly, as it took about 3 GB in total in different quality), the money would go (purchases were built into the game 4 levels - free to go to the rest - 1 dollar), development of further MoscowJump, ParisJump, New-YorkJump will begin, people will be able to shoot themselves on the phone and add new levels, others will play them, give marks, share comments and play turn into some kind of social en a log of recent games Pokemon.

In fact, I, of course, did not think so. I saw all the flaws of the game. To start the game now (even with some kind of new game approach) - money is required, promotion. The game itself must be of better quality, interesting features must be built into the game, the graphics are a little better, the video is more qualitative.

In the meantime, as a result, I will say that Adobe AIR + Flash is, albeit forgotten now, but a good technology, with the help of which you can easily make mobile games even to non-professionals.

For the story, I published a full gameplay , a full video of all levels of the game on youtube, maybe someone recognizes himself on video :) and transferred the source code from a private repository to github .