Description of the prototype of my game multiplayer server
Hi, Habr. I am glad to present my first article: a description of the prototype game multiplayer server.
→ Source code (under Apache 2.0 license)
Content:
')
The entry point is a websocket controller that accepts all sorts of requests from users: from login and game requests to game moves and writing chat messages. This controller serves thread-pool (about 20 threads).
One of the most important things in the game is the fast processing of game actions (priority business case). That is, ideally, the game should instantly respond to user actions and not “hang”. While for many other non-game actions, such as authentication, writing messages in a chat or matching players (for playing together), a longer or a smaller delay is quite acceptable for the user.
Therefore, I tried to design the architecture in accordance with this requirement as well (see picture):
Incoming authentication requests are queued in a special thread-pool. After that, the thread is immediately released and again ready to take user action.
The authentication process itself will be already in asynchronous mode. It includes:
- authentication on Facebook;
- create or update user data in the database;
- sending information to the client in case of a successful login.
The size of this thread-pool can be adjusted already depending on the server load.
When a player requests to create a game request, the solution to the forehead would immediately be to call a functional (function / method) that matches the players. And for the purpose of thread safety, add some variant of blocking a shared resource (in our case, this is a list of applications for the game). That is, we take a lock, then we match the players, then we create a game between them and send notifications to customers. If these operations take a relatively long time, then all other threads awaiting the removal of the lock will be wasted. As a consequence, the server may potentially not accept inbound game actions that must enter the Game Loop immediately.
This option is also potentially poorly scalable: with an increase in the number of threads, they can all get up (block each other) when accessing this shared resource. From here there will be little benefit when scaling up and increasing the number of threads. Therefore, I came up with another option (by the way, if this approach has a name - write in a comment):
In the new version, game requests are added to the Concurrent Map, where key is the user and value is the application for the game.
All - after that the incoming stream is immediately released.
Streams will not even always block each other (when writing to it), since ConcurrentMap does not patch the whole map, but a segment.
Every n seconds, exactly 1 thread is called to process incoming game requests (Matching players). He calmly and handles this map. This ensures flow safety without blocking and quick release of incoming flows.
This solution has another advantage - it allows you to “fill up” the applications and then match them in a more appropriate way. The logic and implementation became simpler.
Here is a bit more complicated:
1) Incoming game actions (moves) simply add up to a special queue (each game has its own queue). This queue simply stores the actions performed by the players. After that, the stream is traditionally released.
2) As usual, we have a GameLoop (game cycle). It bypasses the list of all games. Next, for each game, he pulls a queue associated with it. And already from this queue gets the moves made by the players. It then processes each action sequentially. It's simple.
In principle, it is also possible to parallelize the processing of different games on a thread pool. This is possible, since the games are not related to each other. This functionality can also be made non-blocking: it is enough, for example, to use non-blocking locks from the java.util.concurrent library. Or, if we have a distributed solution, use Hazelcast as a distributed cache with the ability to lock the keys in a map ... However, this functionality is not required, since the processing of game actions is too fast. Therefore, GameLoop runs in one thread.
3) There is one more thing - if the game has changed, then you need to send notifications to clients and, if necessary, update the data in the database. This is also done in asynchronous mode (No. 4 in the picture), so as not to slow GameLoop.
Summarizing
The architecture is designed to:
- Requests with game actions had the highest priority over other types of requests (for example, before login, or application for a game).
That there were no such that 100 requests for authentication came and "hammered" thread-pool (serving user requests). At the same time, incoming game actions would stand in a queue, and all games would “slow down” at once for a few seconds.
- Everywhere was non-blocking multithreading.
This approach supports vertical scaling and an increase in the number of threads. An article on the topic, see the section "Scalability Issues"
One of the low-priority tasks was to make a weak binding between the server module and the game module . In other words, so that you can easily “pull out” the game itself and attach it to the Desktop-UI. Or put another game of the desktop type to a multiplayer server.
This is achieved by defining the areas of responsibility. The project is divided into three modules: server, game API and game implementation.
All game logic is “sewn up” in the game module. The server simply redirects the game actions to the game controller via the code API (Java code). Answers from the game come either immediately or postponed - through a subscription (Subscriber template).
The game module knows nothing about who will call it via the java API and subscribe to events. For a better understanding of the interaction between the server and the game, a separate module is highlighted - the game API is a set of contracts (interfaces). The server only calls them. A game module provides an implementation.
There are both unit and integration tests.
In general, tests are used where there are difficult / long / tedious test cases.
For example, these may be different disconnect options: let's say if the user has connected from a new device, then the old connection needs to be closed. Or, for example, this is a disconnection check, if the user was inactive for 15 minutes (so as not to wait that long - many parameters are put into environment variables and “locked” for a few milliseconds for a quick test run).
There is also a chat check: that different users see each other's messages.
There are checks for game requests and game creation.
Under the aforementioned cases, integration tests well suited up the server and the IoC context (external systems are locked).
Unit tests are also used. For example, where there is no need to raise the context; or where you need to check many variations of input parameters.
For example, unit tests are used to cover game rules. Each game wheel is implemented by a separate class with a single public method. It is a pure function and is easily covered with dough.
And further - from these rudder functions business logic is already composed in a separate class. So much easier to read and understand the code.
In general, when choosing the type and methods of writing tests, I liked this report: “Hexlet - Testing and TDD” .
It is similar to the fast processing of game requests - it is also calculated with caching of calls to the database . During authentication, the user’s data is read into the cache (if there is no cache yet). After that, all the rare requests for this data come from the cache. At the end of the game (which happens not often) there is an entry in the database with the update of information in the cache.
It is better not to look at the client’s code and functionality: for the prototype, very little functionality was needed there, so it was written in a quick way. All functional extension points, the generalized code is laid on the backend.
Not all moments are disclosed in the article. In particular, about the management of connections and disconnects (for example, in the case of opening a session of a new device). The game also has a ranking system and the top 100 players on the main table. There is not only multiplayer, but also a game with bots. Embedded functional expansion points for various aspects of both the server and the game.
The game is written in Java. With the active use of the Spring Framework, which out of the box gives you work with Websockets (Spring WebSocket), integration tests (Spring Boot Test) and a bunch of other buns (DI, for example).
Horizontal scaling for websockets is not so easy to do. Therefore, in order to speed for the prototype, it was decided not to do it.
The server is hosted on a free account on Heroku. According to this free tariff, the server is cut down if there were no requests for it within 30 minutes. An elegant solution was found - I just registered on the monitoring site, who periodically ping the server. As a bonus - getting additional information on monitoring.
There is also a free Postgre with a 10k line limit. Because of this, you have to periodically run the deletion of irrelevant accounts.
→ Source code (under Apache 2.0 license)
Content:
')
- Inbound Processing Architecture
- Brief description of other moments
- Modules and interactions of the main classes
- Different types of tests
- Caching when working with the database
User Inbound Processing Architecture
The entry point is a websocket controller that accepts all sorts of requests from users: from login and game requests to game moves and writing chat messages. This controller serves thread-pool (about 20 threads).
One of the most important things in the game is the fast processing of game actions (priority business case). That is, ideally, the game should instantly respond to user actions and not “hang”. While for many other non-game actions, such as authentication, writing messages in a chat or matching players (for playing together), a longer or a smaller delay is quite acceptable for the user.
Therefore, I tried to design the architecture in accordance with this requirement as well (see picture):
Authentication Requests (# 1 in the picture)
Incoming authentication requests are queued in a special thread-pool. After that, the thread is immediately released and again ready to take user action.
The authentication process itself will be already in asynchronous mode. It includes:
- authentication on Facebook;
- create or update user data in the database;
- sending information to the client in case of a successful login.
The size of this thread-pool can be adjusted already depending on the server load.
Requests for the game (№2 in the picture)
When a player requests to create a game request, the solution to the forehead would immediately be to call a functional (function / method) that matches the players. And for the purpose of thread safety, add some variant of blocking a shared resource (in our case, this is a list of applications for the game). That is, we take a lock, then we match the players, then we create a game between them and send notifications to customers. If these operations take a relatively long time, then all other threads awaiting the removal of the lock will be wasted. As a consequence, the server may potentially not accept inbound game actions that must enter the Game Loop immediately.
This option is also potentially poorly scalable: with an increase in the number of threads, they can all get up (block each other) when accessing this shared resource. From here there will be little benefit when scaling up and increasing the number of threads. Therefore, I came up with another option (by the way, if this approach has a name - write in a comment):
In the new version, game requests are added to the Concurrent Map, where key is the user and value is the application for the game.
All - after that the incoming stream is immediately released.
Streams will not even always block each other (when writing to it), since ConcurrentMap does not patch the whole map, but a segment.
Every n seconds, exactly 1 thread is called to process incoming game requests (Matching players). He calmly and handles this map. This ensures flow safety without blocking and quick release of incoming flows.
This solution has another advantage - it allows you to “fill up” the applications and then match them in a more appropriate way. The logic and implementation became simpler.
Processing game action (number 3 in the picture)
Here is a bit more complicated:
1) Incoming game actions (moves) simply add up to a special queue (each game has its own queue). This queue simply stores the actions performed by the players. After that, the stream is traditionally released.
2) As usual, we have a GameLoop (game cycle). It bypasses the list of all games. Next, for each game, he pulls a queue associated with it. And already from this queue gets the moves made by the players. It then processes each action sequentially. It's simple.
In principle, it is also possible to parallelize the processing of different games on a thread pool. This is possible, since the games are not related to each other. This functionality can also be made non-blocking: it is enough, for example, to use non-blocking locks from the java.util.concurrent library. Or, if we have a distributed solution, use Hazelcast as a distributed cache with the ability to lock the keys in a map ... However, this functionality is not required, since the processing of game actions is too fast. Therefore, GameLoop runs in one thread.
3) There is one more thing - if the game has changed, then you need to send notifications to clients and, if necessary, update the data in the database. This is also done in asynchronous mode (No. 4 in the picture), so as not to slow GameLoop.
Summarizing
The architecture is designed to:
- Requests with game actions had the highest priority over other types of requests (for example, before login, or application for a game).
That there were no such that 100 requests for authentication came and "hammered" thread-pool (serving user requests). At the same time, incoming game actions would stand in a queue, and all games would “slow down” at once for a few seconds.
- Everywhere was non-blocking multithreading.
This approach supports vertical scaling and an increase in the number of threads. An article on the topic, see the section "Scalability Issues"
Brief description of other moments
One of the low-priority tasks was to make a weak binding between the server module and the game module . In other words, so that you can easily “pull out” the game itself and attach it to the Desktop-UI. Or put another game of the desktop type to a multiplayer server.
This is achieved by defining the areas of responsibility. The project is divided into three modules: server, game API and game implementation.
All game logic is “sewn up” in the game module. The server simply redirects the game actions to the game controller via the code API (Java code). Answers from the game come either immediately or postponed - through a subscription (Subscriber template).
The game module knows nothing about who will call it via the java API and subscribe to events. For a better understanding of the interaction between the server and the game, a separate module is highlighted - the game API is a set of contracts (interfaces). The server only calls them. A game module provides an implementation.
There are both unit and integration tests.
In general, tests are used where there are difficult / long / tedious test cases.
For example, these may be different disconnect options: let's say if the user has connected from a new device, then the old connection needs to be closed. Or, for example, this is a disconnection check, if the user was inactive for 15 minutes (so as not to wait that long - many parameters are put into environment variables and “locked” for a few milliseconds for a quick test run).
There is also a chat check: that different users see each other's messages.
There are checks for game requests and game creation.
Under the aforementioned cases, integration tests well suited up the server and the IoC context (external systems are locked).
Unit tests are also used. For example, where there is no need to raise the context; or where you need to check many variations of input parameters.
For example, unit tests are used to cover game rules. Each game wheel is implemented by a separate class with a single public method. It is a pure function and is easily covered with dough.
And further - from these rudder functions business logic is already composed in a separate class. So much easier to read and understand the code.
In general, when choosing the type and methods of writing tests, I liked this report: “Hexlet - Testing and TDD” .
It is similar to the fast processing of game requests - it is also calculated with caching of calls to the database . During authentication, the user’s data is read into the cache (if there is no cache yet). After that, all the rare requests for this data come from the cache. At the end of the game (which happens not often) there is an entry in the database with the update of information in the cache.
It is better not to look at the client’s code and functionality: for the prototype, very little functionality was needed there, so it was written in a quick way. All functional extension points, the generalized code is laid on the backend.
Not all moments are disclosed in the article. In particular, about the management of connections and disconnects (for example, in the case of opening a session of a new device). The game also has a ranking system and the top 100 players on the main table. There is not only multiplayer, but also a game with bots. Embedded functional expansion points for various aspects of both the server and the game.
The game is written in Java. With the active use of the Spring Framework, which out of the box gives you work with Websockets (Spring WebSocket), integration tests (Spring Boot Test) and a bunch of other buns (DI, for example).
Horizontal scaling for websockets is not so easy to do. Therefore, in order to speed for the prototype, it was decided not to do it.
A couple of funny moments
The server is hosted on a free account on Heroku. According to this free tariff, the server is cut down if there were no requests for it within 30 minutes. An elegant solution was found - I just registered on the monitoring site, who periodically ping the server. As a bonus - getting additional information on monitoring.
There is also a free Postgre with a 10k line limit. Because of this, you have to periodically run the deletion of irrelevant accounts.
Source: https://habr.com/ru/post/351738/
All Articles