How we outbid ordinary players and their servers or DDoS: a practical guide
Talking about new projects is, of course, good, but everything does not always work out the way we want.
In general, we began to recall fakapas from the past, when the solution of one problem added new ones, they got carried away and decided to share a couple. How to ban innocent players, punch your own servers, make a mistake in one letter and catch tons of negatives from users - that’s all we love.
There was a time when our game War Robots was attacked by cheaters because of flaws in architecture. There were utilities that increase the stock of health to cosmic values, which made them almost immortal. We finally closed the task with the cheaters, but not immediately.
')
At first we wanted to solve the problem beautifully at the technical level: block the ability to change the parameters of the robot on the client. We failed to do this (then, of course, we found a way). And then the first working solution was banhammer, which calculated cheaters using a simple scheme:
The decision was a crutch, but effective. Problems started because of it a little later, but first you’ll have to tell a little about another bug and features of the development of synchronous mobile PvP.
When players have a bad internet connection (and this is a normal situation in mobile games), absolutely magic things can happen when data is exchanged between client and servers. Requests from clients come incomplete, not in that order or with a strong delay. In general, one server bug allowed that if a connection was poor, the client could send the results of the battle two or even three times in a row. Accordingly, players could receive 2-3 times more rewards or accidentally spend twice as many resources on repairs.
We solved this problem rather quickly: the server profile learned to ignore the extra results of the battle from one client. After successful testing, we have released a new version.
This is where we are covered.
A whole bunch of players every day began to ban Banhammer, about which we safely forgot, because the problem of unkillable robots with an endless supply of health was far in the past. It turned out that as soon as the player's client sent the results of one battle to the server in double, the bankhammer took it so that every dead robot was killed twice - that is, He took damage twice his health. And after each battle, the players flew to the packs in the ban.
All, of course, unban and even paid compensation, but the situation is so-so, not exactly positive.
Already wrote about the evolution of our server infrastructure, and now remembered one case of that time.
At the end of 2015, the release of the long-awaited feature in War Robots - clans. When the update came out (and it was late at night), we opened the champagne and everything would be fine. But it was not long to rejoice - the servers suddenly became ill. It turned out that we made a DDoS attack with our own hands.
How? Very simple. The client on the combat results screen in trying to get information about clans of players made too many requests. And when the server responded “leave me alone, error”, the client returned to the server without any timeout.
The same night, we flashed a flag (we did not have time to close the champagne), which was controlled from the profile server - it completely blocked the work of the Hangar Client API. To the players who have already joined the clans, we left this box enabled, that is, everything worked for them, because their number was not enough to secure the server.
As a result, we began to correctly process the server's responses in the game, and in case of an error - to increase the timeout for retry request.
A separate story is when a poor-quality realization meets the human factor. Only now no one was banned, but on the contrary distributed a rating to the left and to the right. In short, one night our monitoring (and we are monitoring everything at all ) recorded a very rapid growth in the rating of players.
Then it turned out that the implementation of the scoring itself in theory allowed duplicate data. But no one would pay attention to this if the administrator on duty due to a typo in one letter accidentally restarted the server, which should not have been working. It was he who began to double the players ratings.
I had to urgently release the fix and go through the base to remove all the extra points that had time to accrue. To prevent this from happening - on all servers, we cut out the old scheme for calculating points and ruled out the possibility of erroneous launching of services where they should not work. It was necessary to do this from the very beginning, of course, but it would be too boring.
There was another typo with a typo, but much more serious.
Somehow on Halloween we launched a new gucah lottery. If anyone does not know, Gach is the mechanic of getting an item from several different ones at random. In the lottery, the player had a limited visible set of prizes of different value. For each discovery, the player received 1 prize, this prize was removed from the set, and the opening price increased each time. Thus, the player could buy all the prizes of the lottery guaranteed, and the lucky ones took out the most valuable prizes at the first discoveries (and, accordingly, received them very cheaply).
In general, we dabbed a feature with sweat and blood for the event, tested it, laid it out. Launch, update graphics ... Hurray! They rushed up! .. And at the same time tons of negativity in the community hit us, that we allegedly deceive our players.
After half an hour, the lottery had to be turned off. Yes, we really cheated the players. But the matter was not at all in the chances or the prizes - the matter was in one letter.
The lottery interface shows the cost of the current opening (the one that increased each time), for example, PRICE: 100 Gold. Here's what the concept looked like (note that the idea on each card also additionally indicates the price of opening it):
And this is how it went on sale, when as a result of a series of "improvements" PRICE (price for participation in the lottery) for the designer suddenly turned into a PRIZE (prize):
At the same time, because of the confusion before the release and the bad check from each individual card, the price for discovery was lost, which all together confused the players.
And they reflexively pressed the button until they spent all the hard. Well, what, the "prize" is increased with each purchase. And so it was in 18 languages. At the same time, we also had locales on the client, so even one letter could be fixed only through a hotfix.
As a result, the situation was unfolded, additional check points were entered, gold was returned to the players and the locales were written down on the server so that nothing like this would happen again.
It's time to enter the hashtag # kosyakinaprode
In general, we began to recall fakapas from the past, when the solution of one problem added new ones, they got carried away and decided to share a couple. How to ban innocent players, punch your own servers, make a mistake in one letter and catch tons of negatives from users - that’s all we love.
Lost - get a ban
There was a time when our game War Robots was attacked by cheaters because of flaws in architecture. There were utilities that increase the stock of health to cosmic values, which made them almost immortal. We finally closed the task with the cheaters, but not immediately.
')
At first we wanted to solve the problem beautifully at the technical level: block the ability to change the parameters of the robot on the client. We failed to do this (then, of course, we found a way). And then the first working solution was banhammer, which calculated cheaters using a simple scheme:
- Each robot after the match was tested for damage.
- If the damage received by the robot exceeded the value of its maximum health, then the player was recognized as a cheater, and his account was blocked.
The decision was a crutch, but effective. Problems started because of it a little later, but first you’ll have to tell a little about another bug and features of the development of synchronous mobile PvP.
When players have a bad internet connection (and this is a normal situation in mobile games), absolutely magic things can happen when data is exchanged between client and servers. Requests from clients come incomplete, not in that order or with a strong delay. In general, one server bug allowed that if a connection was poor, the client could send the results of the battle two or even three times in a row. Accordingly, players could receive 2-3 times more rewards or accidentally spend twice as many resources on repairs.
We solved this problem rather quickly: the server profile learned to ignore the extra results of the battle from one client. After successful testing, we have released a new version.
This is where we are covered.
A whole bunch of players every day began to ban Banhammer, about which we safely forgot, because the problem of unkillable robots with an endless supply of health was far in the past. It turned out that as soon as the player's client sent the results of one battle to the server in double, the bankhammer took it so that every dead robot was killed twice - that is, He took damage twice his health. And after each battle, the players flew to the packs in the ban.
All, of course, unban and even paid compensation, but the situation is so-so, not exactly positive.
How to arrange yourself DDoS
Already wrote about the evolution of our server infrastructure, and now remembered one case of that time.
At the end of 2015, the release of the long-awaited feature in War Robots - clans. When the update came out (and it was late at night), we opened the champagne and everything would be fine. But it was not long to rejoice - the servers suddenly became ill. It turned out that we made a DDoS attack with our own hands.
How? Very simple. The client on the combat results screen in trying to get information about clans of players made too many requests. And when the server responded “leave me alone, error”, the client returned to the server without any timeout.
The same night, we flashed a flag (we did not have time to close the champagne), which was controlled from the profile server - it completely blocked the work of the Hangar Client API. To the players who have already joined the clans, we left this box enabled, that is, everything worked for them, because their number was not enough to secure the server.
As a result, we began to correctly process the server's responses in the game, and in case of an error - to increase the timeout for retry request.
"Free" rating
A separate story is when a poor-quality realization meets the human factor. Only now no one was banned, but on the contrary distributed a rating to the left and to the right. In short, one night our monitoring (and we are monitoring everything at all ) recorded a very rapid growth in the rating of players.
Then it turned out that the implementation of the scoring itself in theory allowed duplicate data. But no one would pay attention to this if the administrator on duty due to a typo in one letter accidentally restarted the server, which should not have been working. It was he who began to double the players ratings.
I had to urgently release the fix and go through the base to remove all the extra points that had time to accrue. To prevent this from happening - on all servers, we cut out the old scheme for calculating points and ruled out the possibility of erroneous launching of services where they should not work. It was necessary to do this from the very beginning, of course, but it would be too boring.
Priceless prize
There was another typo with a typo, but much more serious.
Somehow on Halloween we launched a new gucah lottery. If anyone does not know, Gach is the mechanic of getting an item from several different ones at random. In the lottery, the player had a limited visible set of prizes of different value. For each discovery, the player received 1 prize, this prize was removed from the set, and the opening price increased each time. Thus, the player could buy all the prizes of the lottery guaranteed, and the lucky ones took out the most valuable prizes at the first discoveries (and, accordingly, received them very cheaply).
In general, we dabbed a feature with sweat and blood for the event, tested it, laid it out. Launch, update graphics ... Hurray! They rushed up! .. And at the same time tons of negativity in the community hit us, that we allegedly deceive our players.
After half an hour, the lottery had to be turned off. Yes, we really cheated the players. But the matter was not at all in the chances or the prizes - the matter was in one letter.
The lottery interface shows the cost of the current opening (the one that increased each time), for example, PRICE: 100 Gold. Here's what the concept looked like (note that the idea on each card also additionally indicates the price of opening it):
And this is how it went on sale, when as a result of a series of "improvements" PRICE (price for participation in the lottery) for the designer suddenly turned into a PRIZE (prize):
At the same time, because of the confusion before the release and the bad check from each individual card, the price for discovery was lost, which all together confused the players.
And they reflexively pressed the button until they spent all the hard. Well, what, the "prize" is increased with each purchase. And so it was in 18 languages. At the same time, we also had locales on the client, so even one letter could be fixed only through a hotfix.
As a result, the situation was unfolded, additional check points were entered, gold was returned to the players and the locales were written down on the server so that nothing like this would happen again.
It's time to enter the hashtag # kosyakinaprode
Source: https://habr.com/ru/post/417441/
All Articles