Delayed processing of commands in a social game

Hello!

Today I want to share with you the solution of the problem that we faced when developing a social game. The game client was written in flash, and php was chosen for the back-end. The game belongs to the time management games.
The scheme of work was chosen as follows:

1. the player performs an action on the client
2. the client checks the possibility of an action
3. sends command to server
4. the server checks whether the action is performed, executes the command, making changes in the database
5. the client is notified that everything is OK or is informed about the error

Everything worked fine until a sharp increase in the number of players occurred.
First started the brakes from php. The main problem with this implementation was that for every action a player is jerking a server that performs quite a lot of calculations on calculating objects on the map before executing the command. This problem was solved by adding additional servers with php handlers.
Then we came up against the performance of mysql. There were too many requests. Since the sharding was not incorporated into the system, they were twisted as best they could. Something moved in mongodb, somewhere improved the work with the cache.
')

By the way, mongodb was not such a simple repository, as it may seem at first glance. Considering the fact that we have sharding turned on and the correct indexes were set, we still caught the brakes there, which we could not figure out at that time. Periodically, just a packet of requests spilled out of the log, which suddenly blunted. Although a second later the same requests worked fine with the same number of requests. But this is a topic for a separate post.

Actually for a new similar project, it was decided to use a different scheme of interaction between the client and server.
That's what I want to tell you.



The principle of operation is as follows:

1. when you start the game receives the full player state of the player and the current game balance
2. the player performs actions on the client
3. the client validates them, but does not send them to the server, but accumulates them
4. upon reaching a condition, the client sends all these commands to the server in a packet
5. the server verifies that the commands are in the correct format and simply informs the client that everything is ok
6. while all the commands are saved in the queue
7. a daemon is also running on the server, which periodically pulls out a stack of commands from this queue and executes them

It turns out the scheme with deferred processing commands, which has the following advantages:

1. the number of server requests is much reduced
2. decreases the number of calls to mysql when processing commands, since saving to the database occurs only after processing all commands from the pack

Of course there are also disadvantages, which are more likely solvable problems. And here you need to look at the requirements of the project.
Our list was as follows:

1. getting the current game state at the moment, until the whole queue for this user is processed
2. some teams require execution in realtime (for example, buying real)
3. players cheating on the client (the command was executed on the client, but failed on the server)
4. multi-thread processing

Gearman is used as the queuing server on the project. Otherwise, everything is standard: php + mysql + memcached.
A sharding for mysql and memcached is laid into the current implementation (there is a small number of memcached servers).

Let's talk about how to solve the above problems.

On the server went packs of commands. The player closed the game and immediately launched again. The queue is not processed yet.

Since Gearman is not a DB and it doesn’t allow to somehow search for data stored inside it, a module was written in php that allows you to connect the database to command processing and always know what state the processing of a particular user is in: how many packs commands in processing and whether there were any errors in processing.
When a client requests a profile for a player for whom the entire queue has not yet been disassembled, he receives in response a message asking him to wait and request a profile after 10 seconds. This is repeated until the profile is received.
On fitting, it will take up to 20 seconds to execute all the player’s commands, which is permissible.

Execution of certain commands in realtime

It is probably not even a problem. Just implemented the ability to execute certain commands here and now. It is necessary so that the player does not lose the money that he brings into the game in case of problems in processing the queue.

Cheating player on the client. Errors due to the difference in logic on the server and client

Situation: the player spun the client and threw money to himself. The client then allows him to buy the building. Then the player performs some actions with the building. It creates a chain of events that could not occur, because the player actually has no money.
The server receives 4 packs of commands. In the second, there is a team buying a building.
Queue processing begins. The first pack of commands is processed successfully, and in the second there will be a logical error, which leads to the fact that the building cannot be bought.
At this moment for the user in the database is put a timestamp when an error occurred. Since all the command packs contain information about when they arrived, at the moment when the handler receives 3 and 4 packs, he will miss them, since their creation time is shorter than the error time.
At this point, the client sends 5 pack of commands, but instead of answering that everything is ok, it receives a request to restart the game state. We know when the client last received the game state. And if this time is less than the time of the last error, then the server does not refuse to accept commands for processing.

Processing in multiple threads

We have 3 threads. And 3 packs of teams in the queue for different users. Handlers simply select these bundles at the same time and process them. All perfectly.
The situation becomes more complicated when we have 3 packets in a queue for one user and more than 1 free handler. In order to avoid a situation in which parallel processing of two or more packs of teams of one player begins (they need to be processed sequentially) a kind of traffic light is implemented, which allows you to say that the user is currently being processed. And if so, then the second handler will put the data back into the queue, but with a higher priority . The priority is changed so that the 2 task remains ahead 3. Otherwise, after processing 1 pack, the handler will receive 3 pack. This traffic light is handled by handlers. At the moment after receiving the commands, we say that the user is processed, and at the end of processing - ready.

This is actually all that I wanted to tell. I do not want to delve into the implementation, as there is nothing nontrivial.
As a framework, the Yii framework is used. For the daemon, use the command for yiic, which runs as follows
nohup ./yiic que work> / dev / null &

The team monitors the number of descendants. Launches new ones if they fall for some reason.
Descendants register GearmanWork to parse the queue.

Thanks for attention! Put your thumbs up and subscribe :)