This is not a toy for you: how the Playkey cloud gaming service works

Hi, Geektimes! This is the first post from Playkey - the first and largest Russian cloud gaming service. To play on a cloud gaming service, you need a high-speed Internet connection and almost any hardware, both on Mac OS and Windows.

Outside it looks like a fairy tale: a decade old PC, and GTA V flies online without brakes. But, if you look inside, then launching a top toy in the “cloud” and transmitting it in real time to the user, instantly reacting to his actions is not easy as hell.

For our intro publication, we decided to talk with Playkey technical director Alexey Lykov about how Playkey technology works in principle, how we solve the “last mile” problem and fight with high ping. Go!

(oh well, who are we kidding, Alexey is just looking at the monitor for the frame! In general, meet Alexey Lykov)

How it works

For a user, the operation of Playkey looks simple: you choose a game in the catalog, click to play - and you get a live picture in the client window.

From our side it's all a bit more complicated. It's like “riding a bike that burns, and you burn, and everything burns, and you're in hell.”

At the core of the Playkey service is a game server. This is a virtual unit that serves one user. Dozens of virtual servers can be run on the same physical server, which already serve individual users.

Service is designed to bring joy to gamers, not pain and lagging picture. Therefore, our task is to tune in to users, assessing the speed and quality of the Internet connection on their part, hardware, their own resources, free slots on the server, the availability of the game itself, and instantly adjust to all these conditions.

At the stage of loading a game through Playkey, ping to the nearest servers and the presence of the selected game on a particular server are checked first. Understandably, no one remembers that we have already downloaded this game ourselves and installed it on the server SSD. Checkpoint is not relevant for all games, because the most popular titles like GTA 5 are spread on all servers at once.

What happens if the service load is close to critical and only long servers with high ping remain? The player is queued up for launch. You have to wait a short time, 3-4 minutes, but, fortunately, recently, such episodes occur very rarely and only at the most peak moments. We understand that the lag picture is a situation that is much worse than having to wait a couple of minutes, so we chose this scenario.

This is all your fault: we sort out the difficulties on the user's side

The main and first question asked about the Playkey is ping. Today we can unequivocally say that for large Russian cities we have achieved excellent value: when playing by cable, you can count on units of milliseconds. This is enough even for playing shooters.

Why can ping in playkey be above the playable level? According to our statistics, the main problem is the quality of the network directly at the user's home. The frequent problem of the “last mile” is connecting via Wi-Fi. With Playkey, we transmit realtime video, so we need to instantly respond to the “feedback” from the player. On the technological side, Playkey is far more complex than the standard video stream. We cannot allow a delay of a couple of minutes before viewing or reloading the sagging parts “in progress”. The video buffer in Playkey is minimal and amounts to milliseconds - as soon as it crosses over 20-30 milliseconds, it becomes uncomfortable to play.

The Wi-Fi router is a terrible headache where packet loss is unacceptable. Put a kettle near the router - Wi-Fi practically will not work. Move the router behind the wall - the signal will begin to lose twice as many packets. For the player, this means that the picture will start to lag, and the ping will grow. When connecting via cable all these problems are not, but what can you do, we have learned how to work with players via Wi-Fi. How?

To eliminate lags when playing via Wi-Fi, we made adaptive bitrate triggers, which we collect all the time. Based on the data obtained, we set the user to the highest possible quality and bit rate, depending on its conditions. The situation of the user may change: starting from the fact that someone started using the network in parallel with the game, or the player moved to the next room, and the Wi-Fi signal improved (or vice versa worsened?). At each step we take retaliatory actions.

Therefore, we strongly recommend that users connect via wire, this allows us to exhibit better quality.

Additionally, even before launching the game, the system collects statistics on the user's hardware: memory, operating system, processor. This data allows us to understand the quality of the video stream can be decoded by the user's computer. To decode streaming video, you need resources: if we are talking about the Core i7, then it can decode in realtime and 4K. And something old or laptops on the Intel Atom are practically incapable of doing it on the fly. We have to weaken the bitrate, leave fewer encoder chips - otherwise the software decoder simply does not have time to process the frame. Even here we are trying to find a balance in the quality of the image that is available to the user in his realities.

Our hyper goal is to launch Playkey on 3G / 4G networks, although they are sharpened for fundamentally different tasks. Now, when transferring data within the mobile network, the player receives a high ping, and although it is generally possible to play, here we still have a lot of work to do.

There are also more serious problems - on the way from the computer to the data center. It happens that traffic goes from Moscow to St. Petersburg via Amsterdam. Just because the operator at the backbone level, or in a specific data center, thought that it would be cheaper.

To combat this phenomenon is almost impossible, except by connecting to the traffic exchange sites. Most often the problem occurs when working with different operators. If the Perm user is connected, for example, via “XX-telecom”, and the servers are connected via “YY-telecom”, then it happened that the traffic was hooked through Moscow. between providers may not be direct peering, since at the moment this problem is mostly eliminated.

How games are ported

The appearance of the game on the Playkey - it is not just “downloaded-set”. This is a whole way to adapt each project to the cloud. Previously, it took months, today we have made some progress: many engines are supported “immediately”, and the game can be launched on the day of release.

In other cases, you have to connect a technical team and start setting up. “Guys, we work further”:

Fun Fact # 1: One of the main difficulties in porting games is anti-cheat protection. The behavior of the cloud gaming service looks like an emulator, and Tom Clancy's conditional : Rainbow Six Siege , for example, does not give us access to the online mode.

All games use WinAPI functions, there are a lot of these functions, some developers use little functionality, some - advanced. In our technology, we handle these challenges, but they are very many and they behave differently. Most of the scenarios we are already working out perfectly, but there are some new things. Yes, writing a file is not a problem, but how does the game work with it further? What markers does it put on this file? Only for reading? Does she close it? Did she leave a link to it to use later? The main scenarios, such as working with the file system, working with the registry, working with sound and video - we have, of course, already worked. Therefore, to launch a new game, it is often enough to specify the version of Direct X, select the audio library, and everything is ready. But there are also surprises, because of which setting up the game can take from a couple of hours to a couple of months.
')

How are catalog games selected?

We are repelled from the goal to select the hottest new items, fresh ones that have received a high rating and, of course, with high technical requirements. High priority is given to new games that we technically know that we can probably release on the day of release. Users themselves also make the choice - often in social networks we open the polls, we take the winners in a mass decision. That is, users here directly affect the library of games Playkey.

It looks like this:

(and by the way, our VK group: https://vk.com/playkey_net )

Two important “buts”: we don’t take to play games in beta and we don’t work with titles that are focused exclusively on online and have a complex anti-cheat system.

What servers are used?

The huge complexity of Playkey is that we need not ordinary servers, but servers with a GPU. There are not many of them in the world, and in Russia there is practically no such way at all.

Here’s what these exclusive guys look like:

The problem is that they are rarely available for rent. Service providers often boast that they have such solutions available, but, as a rule, they are put in the catalog “to be”. And if there is a request - there they are already organizing the delivery and price offer. There is simply no formed demand for servers with GPUs.

In fact, we have already blocked the amount of server resources with a GPU that was available for rent in Russia.

Fun Fact # 2: The main use case for such servers is the military industry and sophisticated mathematical calculations, where the GPU can do better than the processor. This is also a problem: the cloud architecture does not suit the military - they want to buy a server for themselves. At any price! And this inflates the cost even higher. We, on the contrary, need “clouds” and for rent. And yes, better is cheaper.

Yes, there are several large companies that provide the servers we need, but this is where the price problem arises. They make a one-stop solution for all industries - and primarily for those who work with professional graphics. They build a complicated and complicated infrastructure, and we need an architecture that would simply save money on storing games and reduce the check for players.

Now there are simply no massive tasks for servers with a GPU. And it fundamentally affects the issue price. We want a rental model, but service providers at the current stage have to put all financial risks on us alone. Because of this, it is easier for us to lease the server.

And it's all?

Of course not! This is only our first post in a series of publications about cloud gaming. Now, after we have dismantled the basic architecture of the Playkey service, we plan to tell in detail the statistics about gamers in Russia, about the performance of Playkey hardware (and the cost of building a similar PC at home) and much, much more. So feel free to subscribe to our channel, comment and confess - what else would you be interested to know about us?

Happy New Year, Geektimes! Play only the best games.