Why we didn’t do it perfectly: how the infrastructure of the War Robots servers was changing

The first prototype (for example, games in a new niche for you) is often done “on the knee” of sticks and oneself. And the sticks, as a rule, are also from this self. There are several reasons for this.

Firstly, it would not be a pity to refuse an unsuccessful idea. And secondly, in the pursuit of perfectionism, you can forget about the needs of end users or never finish the work even on the alpha version. But what if so many people began to get into your “clay” cart that rebuilding it on the move no longer seems such an attractive idea? That's about what happened to us.
')
I'll run ahead and tell you that now there are about 1.5 million DAU in our projects. But this was not always the case.

First commit

The first commit of War Robots in git was made on July 29, 2013. And it was not a soulless initial commit, a huge number of files arrived. How much time was spent on development before this date is unknown, but we can look at everything that happened after.

A commit is a commit, but on April 14, 2014, the expected “miracle” occurred - the launch of War Robots (then also Walking War Robots). For comparison, the first half year of the DAU game was about 10-15 thousand players.

So what was all this about?

Infrastructure 1.0

Profile server

Let's start with the simple. The profile server was a set of servers with a database (Cassandra). On the same servers the app-servers were located, with which the client directly communicated. These were tomcats, the client went to them via HTTPS.

Host server profile - player profiles - what they had in the garage, in the store, how the robot is equipped, etc. Looking ahead, I will say that since those times the structure of the profile server has not changed much. Only the app servers we brought to separate physical machines, and left the database on our own. Well, the number of servers only grew over time.

Prior to launch and for a long time afterwards, a single person from the Belgorod office was engaged in developing a profile server. Only when it became clear that the project is doing well and it is growing rapidly - we began to expand the team, because needed a high speed of development. Now 6-7 people are engaged in the “profile”.

So, we found out where and how the profiles of the players were kept, but the game is not only about the purchase of robots and weapons. We still have to beat our rivals. Let's figure out which servers were matches on the launch.

We did not have our own servers at launch, and we used Photon Cloud. The most frequent questions I hear after his mentioning are “where is the logic?” And “how did matchmaking work?”. There may be many options, but now it will be discussed how everything was arranged in War Robots.

Matchmaking

Any fight in War Robots begins with matchmaking. And with us (like many others) this was built on the principle of “it is necessary yesterday and for nothing.”

Imagine a room in which the oldest client, as in a courtyard football, gained 2 teams and went into battle with them. The captain's flag passed on to the oldest client left in the room and the story repeated.

For a while the solution worked, but what if the flag was not delivered to the most securely connected client? Imagine that as soon as someone handed over the flag, he immediately turned the game ... Believe me, the stupid guys do not work for us, they all knew it well and knew in advance. Just the principle of “yesterday and for nothing” is just this kind of decision.

With the growth of players, the scheme worked worse and worse. Now it was urgent to come up with a solution that would work the day before yesterday. No problems! We launch the client on the PC right in the office, prohibit him from taking himself into the battle and, behold, in tens of minutes - he is already the oldest client in the room! Brilliant and simple.

Now we need to divide MM into groups. What do you think we did? That's right - we launched four client copies on the same machine in the office. And so it worked for six months!

Let me remind you that we are talking about the first months of the game with real players - those same 10-15 thousand DAU.

Match

And where is the logic of the match? Answer: master client. The first who got into the game room, became such. It was he who decided when the match would start / end; appruvil point spawn; determined the victory / loss of teams on points. Moreover, each client controlled the captures of lighthouses in his own way (in the first War Robots “Superiority” mode, in order to win, it is necessary to destroy all opponents or to capture and hold the largest number of lighthouses scattered around the map).

And you quite in one match could observe different states of lighthouses. A lot of bugs in the first months and complaints about the "beacons are not captured" - it's all because of this (may our support forgive us), it's by design!

How were hits recorded? This is generally a separate and interesting topic. They have already told about it, but for the general picture I can only say that at that moment the hit was registered by the client himself, who was shot at. Just imagine an entry in the battle log: “Reporting: MegaKiller3000 dealt me ​​2324 damage” ...

Oh yes! Each client kept its own battle log! At the end of the match, he sent it to the profile server, the server profile waited for N battle logs and decided who to charge what and why. And when the server profile believed that it could decide the outcome of the battle (got enough logs), the client would get the result and that would render the screen with the battle statistics (known as “postkombat”).

Now imagine - not all clients played out the battle, someone turned off, someone was hysterical and closed the game himself, and somehow you had to live with it. It would seem that all this can not work like that at all. But in 95% of cases it worked. Supports of those times even say that it was tolerable.

Of course, everyone understood that it was impossible to live this way and we had to move on. But had it not been for all this, there would not have been what we have now.

Infrastructure 2.0

At that moment only a few developers in the company were engaged in server solutions. One of the most experienced were the guys from the AppMetr team.

Java developers installed Parallels + Windows + Photon Server SDK on Macs and started developing War Robots game servers. The result of their work, I called the infrastructure version 2.0. The development was conducted in parallel with the history described above.



As you can see, Photon Cloud is missing from the diagram. But there was a master server (hooray!). He was engaged in balancing, matchmaking went there and our game servers appeared that served the game rooms.

As a result, we have eliminated some of the horrors described earlier by desing. Matchmaking went from the computer in the office to the master server. The calculation of the results of the battle - from the profile server to the game server. Slowly transferred the logic of the match from the master client to the game servers. Capture lighthouses, victory conditions, damage calculation ...

Infrastructure 3.0. What's next?

The performance of the game was excellent and we had no doubt that the game would be a hit. It was necessary to accelerate, to increase the pace of development. Began to attract more guys from the Belgorod office. They launched a search for a developer who would take over the whole story. So the current head of server development at Pixonic joined the team.

A little later, our feature with clans arrived, under which the following 3.0 infrastructure appeared:



The key difference from everything that was before is services. There was a separate Communication-service. The client was taught to establish a connection with the “hangar” room on the game server and to make API requests (single entry point for interacting with services).

The clans also really wanted to have a chat, the first version of which was implemented on Photon Chat. But we were not satisfied with the loss of chat history when disconnecting all customers from the chat room, so we implemented our own. Cassandra is used just for storing history:



Well, now everything seems fine, but something is missing. Matchmaking! He was still on the master server. Therefore, we move on to the next (it is the current) version of the infrastructure.

Infrastructure 4.0. Current version

The differences are extremely simple: MM migrated to a separate service in each region. They also made leagues, viralku, etc., but this all works the same way as the clans, so we can skip.

What I want to say in the end: when developing a game or any other product, try to show it to end users as soon as possible. Do not immediately try to do everything perfectly. It is more important to get feedback as early as possible and turn in the right direction - after all, what is ideal for you may be indifferent to the player. Otherwise, you can always get stuck at the stage of polishing the project.