How not to write large servers
Those who could see my last article (and it is rather related to this topic) know that for more than a year and a half I have been developing my own implementation of the Minecraft server, designed primarily for high loads. However, in our work, we also use the standard server (Bukkit) for several mini-servers, just to have a variety. And now, faced with the next version of the server, which was 5 times worse than the previous ones, I could not stand it any longer, and decided to write this article.
The article is more like a story than a teaching material, so you are unlikely to learn from it useful coding skills, but I hope it will seem interesting to someone or even useful. But if you go to see a bunch of code and examples, then do not open the article, it’s not about that. I hope this will be the next article.
You do not need to know anything about minecraft and especially about its server, in this article I just want to tell how the original Minecraft server works, as well as its “binding” - Bukkit, to tell why such a system does not work and should not. I do not claim to be perfect knowledge of server development and do not claim that my server is written correctly and better than anyone. I just share my experience based on two years of working with a server from the well-known Mojang and on the one and a half years of developing my server. All the information presented here is my personal opinion, and the article is intended to broaden horizons or even learning and may be of interest to both beginners and advanced professionals.
')
Let's start, perhaps, with the cleanest ("vanilla") Minecraft server, or rather, with what he does. These things are handled by the server:
You ask, what is the problem here? So many people do: the main application logic in one thread, it is very convenient to program, no need to worry about synchronization and other problems of parallel applications. The problem here is that if there are more than 40 people on the server, instead of the standard 20 cycles, he is already doing 15, if 70 people, then 10, if 100, then he sinks to incredible values. This is despite the fact that I actually have a powerful 6-core Core i7 and 64Gb of RAM! And where should I put these resources now, if out of 12 streams two are employed at most?
I will not gossip, give an example:
On server 223 of the player, while the visibility radius is chosen rather small, there are 46577 chunks in memory, 524 "urgent" blocks, 87 blocks of redstone and pistons, 11240 Entity objects, 4274 Entity animals, 19 carts and boats, 717 other Entity, players, which are also Entity and require appropriate processing.
The number of tiles and light updates my server does not display in the information (I do not need it), but you can believe there are a lot of them.
Just processing animals is a terribly difficult process - they regularly search for a path, search for other Entity around, they have AI (quite advanced in recent versions), so it’s already a lot of work to process 4,000 animals.
To bypass 3 million blocks (approximately as many random blocks are processed with so many chunks) is also not a trivial task.
11 thousand items need to move, do some other actions, send updates about their position to players and so on.
And all this needs to be done in 50 milliseconds, otherwise everything will slow down, because the speed is calculated in cycles. If the server does less cycles per second than it should, then, for example, mobs start walking slowly and jerking. Plus calculations in cycles is obvious - if the server hangs or there is a huge garbage collection (the server is in Java), it will not happen that the cart traveling at full speed on the next cycle will turn into a fast moving small object, and you will have to calculate its movement over complex algorithms.
There is also a Bukkit!
Bukkit is such a “wrapper” for a vanilla server. It adds an API for creating plugins, it is super-convenient for plug-in developers and is really well made. But, roughly speaking, it only gets worse. If a player sends a packet, he has moved a little or turned his head ... an event is created and sent to all the plugins that process it. And with this, the motion processing function is already quite complex. When a block is broken or installed, the same thing happens, as well as about a hundred other actions that the player or the server itself creates, including waving a hand, changing the state of the redstone, flowing water, spawning the mob, AI, thousands of them ... That is, The system is good, but it creates a bunch of extra calls when handling everything.
Fortunately, some plug-in developers have learned how to pull the heavy logic of their plug-ins into a separate thread. Vivid and good examples are the OreObfuscator and Dynmap plugins. The first “cleans” blocks sent to the player from unnecessary data so that the player cannot look through the walls with cheats. It does this in a separate thread, folding the packets into a queue and processing them separately from the server logic. The second generates a dynamic map for the browser, is also very well made. In general, praise them for not loading the main thread even more.
There is also a plugin that reduces the number of things that the server processes per cycle. Combines the objects lying nearby, unloads mobs, limits the processing of chunks. This is very cool, no server can do without this plugin - NoLagg.
How to do it right (in my opinion)
We suffered for a long time with all this, when a year and a half ago our online grew to 100 people, and the speed of work slipped to 0.5-1 cycles per second. We tried to optimize the server, correct the code, try to remove as much as possible, changed the work in some places not by cycles but by seconds (for example, in a furnace. In Bukkit, this was also added ... after a few months). In the end, we reached a terrible server instability and decided to spit on it all.
The only option that could provide us with a comfortable online as many players as we wanted was a scalable server. I don’t think that it’s necessary to explain that the process’s thread is not scalable, it can work only on one processor core at the same time and its performance is limited by the core performance. The cores in the processors are now quite productive, but there is a lot of work, and the processors are now being made multi-core, not the time not to do something multi-threaded.
It is not possible to split an existing server into several threads. Multi-threaded programming is a thin, complex thing that requires a lot of knowledge of the code you work with, and it is practically not embedded in the existing application. The code must be written from scratch.
Thus, a server was born, based on as many streams as possible: the world is divided into pieces of 64x64 chunks and each such piece processes chunks in one stream, one stream for processing urgent blocks, one stream for redstone and pistons, one stream for mobs, one stream for items, one stream for carts, one stream processes other Entity and other information about the world, one stream recalculates the light, four streams from different parts of the world save the world to disk, one stream renders a map, one stream maintains the server and a team d console, updates statistics. For players, a system is used that allows packet processing to be placed either on a separate stream for each player, or on a pool of streams, or on separate streams for each player. In this case, everything can be divided into several more streams: process the same type of objects in at least 20 different streams. As well as Netty (NIO) as a network engine, as opposed to standard I / O.
The development of a stable version of such a server, which does not have all the functionality, cost about 8 months of work for me alone without having experience. All code is designed for asynchronous access to all data. But it was worth it - quite recently we set a record of 559 people, who not only stood in one place, lagged and filmed on fraps, but passed a very big event with redstone, and at the same time felt comfortable.
The moral of this story is: if you expect that your project will be at least a little popular and you think that at least a little theoretically it is possible that there will be at least a few people on the server ... do not skimp on creating scalable architecture.
Waiting for your rotten tomatoes, suggestions for improving this article, as well as suggestions for what you would like to see in the next article, which someday will be.
The flow of thoughts may contain spelling errors, because was written in one breath.
The article is more like a story than a teaching material, so you are unlikely to learn from it useful coding skills, but I hope it will seem interesting to someone or even useful. But if you go to see a bunch of code and examples, then do not open the article, it’s not about that. I hope this will be the next article.
You do not need to know anything about minecraft and especially about its server, in this article I just want to tell how the original Minecraft server works, as well as its “binding” - Bukkit, to tell why such a system does not work and should not. I do not claim to be perfect knowledge of server development and do not claim that my server is written correctly and better than anyone. I just share my experience based on two years of working with a server from the well-known Mojang and on the one and a half years of developing my server. All the information presented here is my personal opinion, and the article is intended to broaden horizons or even learning and may be of interest to both beginners and advanced professionals.
')
Let's start, perhaps, with the cleanest ("vanilla") Minecraft server, or rather, with what he does. These things are handled by the server:
- Chunks - for those who do not know, the whole world of Minecraft is divided into pieces with an area of ​​16x16 cubes and a height depending on the settings. All chunks in the radius of visibility of the players are loaded into the server’s memory and are located in the HashMap, each server “tick” is processed. At this time, the following is done: all active chunks (those that are within a certain radius of the players) are moving one by one. For each chunk, the weather is processed (pour snow, strike with lightning), as well as random processing of blocks — several dozen random blocks are selected from the entire chunk, it is checked whether these blocks need to be updated (by block type) and a special function is called on the selected block.
- Tiles are special blocks that are processed every tick, and not by chance. These units include furnaces (updating the burning status of the material, the remaining fuel, this should be done evenly, and not by chance, like the tick of the other blocks), as well as there are spans of mobs (blocks that spawn mobs around themselves), potions boilers and similar things. They are all stored in the List (List), which is filled when a chunk is loaded or when a new tile is installed during operation, and are looped through each cycle.
- Urgent blocks - they, of course, are not called that, but nevertheless, these are blocks that need to be processed “urgently”, that is, on the next cycle or with a small delay (also in cycles, everything is counted in cycles, even time,
even Allah), and not by chance, because random blocks are processed every few minutes on average. They are processed approximately as tiles, only they can have a delay, after how many cycles they need to be processed. Processing tasks are usually generated while the server is running due to the actions of the player or other blocks. In particular, the redstone is processed in such a way that it must very quickly react to external changes, blocks of fire, flowing water and the like. - Light Update - Minecraft uses static lighting divided into blocks. Each block has its own level of illumination from 15 to 0. When changing blocks, their illumination should be recalculated, the algorithm is not very complicated, but recursive, and there are also two types of lighting - from blocks (torches, fire and etc.) and from the sky, they must be calculated independently, that is, twice for each change (if there is a sky in the world, there is no sky in Nether).
- Entity is almost all objects. Mobs, players, items, lying on the floor, carts, boats, paintings, lightning, arrows, and more. All of them are stored in one large list and the function tick () is called in turn. Before this, it is checked whether they died, if they died, then they are removed from the list and from the server memory, respectively.
- Spawning mobs is also a separate action. Mobs spawn in a certain radius from the player, at that a random point is selected in the chunk and on the basis of several shifts in different directions it is selected whether it is possible to put a mob there and it is created
- Handling players - all the packets that the players sent need to be processed, obviously.
- Download and generate chunks - if an attempt is made to access a chunk block that is not in memory, the chunk must be loaded from the disk, if it is not on the disk, it must be generated. No need to explain that the hard disk is almost always a very narrow place. Chunk generation is even more complicated than loading it.
- Saving a chunk — during a general server save or simply when the chunk has not been used for a long time and can be unloaded, chunks need to be saved to disk — converted to stream and written to a file.
You ask, what is the problem here? So many people do: the main application logic in one thread, it is very convenient to program, no need to worry about synchronization and other problems of parallel applications. The problem here is that if there are more than 40 people on the server, instead of the standard 20 cycles, he is already doing 15, if 70 people, then 10, if 100, then he sinks to incredible values. This is despite the fact that I actually have a powerful 6-core Core i7 and 64Gb of RAM! And where should I put these resources now, if out of 12 streams two are employed at most?
I will not gossip, give an example:
On server 223 of the player, while the visibility radius is chosen rather small, there are 46577 chunks in memory, 524 "urgent" blocks, 87 blocks of redstone and pistons, 11240 Entity objects, 4274 Entity animals, 19 carts and boats, 717 other Entity, players, which are also Entity and require appropriate processing.
The number of tiles and light updates my server does not display in the information (I do not need it), but you can believe there are a lot of them.
Just processing animals is a terribly difficult process - they regularly search for a path, search for other Entity around, they have AI (quite advanced in recent versions), so it’s already a lot of work to process 4,000 animals.
To bypass 3 million blocks (approximately as many random blocks are processed with so many chunks) is also not a trivial task.
11 thousand items need to move, do some other actions, send updates about their position to players and so on.
And all this needs to be done in 50 milliseconds, otherwise everything will slow down, because the speed is calculated in cycles. If the server does less cycles per second than it should, then, for example, mobs start walking slowly and jerking. Plus calculations in cycles is obvious - if the server hangs or there is a huge garbage collection (the server is in Java), it will not happen that the cart traveling at full speed on the next cycle will turn into a fast moving small object, and you will have to calculate its movement over complex algorithms.
There is also a Bukkit!
Bukkit is such a “wrapper” for a vanilla server. It adds an API for creating plugins, it is super-convenient for plug-in developers and is really well made. But, roughly speaking, it only gets worse. If a player sends a packet, he has moved a little or turned his head ... an event is created and sent to all the plugins that process it. And with this, the motion processing function is already quite complex. When a block is broken or installed, the same thing happens, as well as about a hundred other actions that the player or the server itself creates, including waving a hand, changing the state of the redstone, flowing water, spawning the mob, AI, thousands of them ... That is, The system is good, but it creates a bunch of extra calls when handling everything.
Fortunately, some plug-in developers have learned how to pull the heavy logic of their plug-ins into a separate thread. Vivid and good examples are the OreObfuscator and Dynmap plugins. The first “cleans” blocks sent to the player from unnecessary data so that the player cannot look through the walls with cheats. It does this in a separate thread, folding the packets into a queue and processing them separately from the server logic. The second generates a dynamic map for the browser, is also very well made. In general, praise them for not loading the main thread even more.
There is also a plugin that reduces the number of things that the server processes per cycle. Combines the objects lying nearby, unloads mobs, limits the processing of chunks. This is very cool, no server can do without this plugin - NoLagg.
How to do it right (in my opinion)
We suffered for a long time with all this, when a year and a half ago our online grew to 100 people, and the speed of work slipped to 0.5-1 cycles per second. We tried to optimize the server, correct the code, try to remove as much as possible, changed the work in some places not by cycles but by seconds (for example, in a furnace. In Bukkit, this was also added ... after a few months). In the end, we reached a terrible server instability and decided to spit on it all.
The only option that could provide us with a comfortable online as many players as we wanted was a scalable server. I don’t think that it’s necessary to explain that the process’s thread is not scalable, it can work only on one processor core at the same time and its performance is limited by the core performance. The cores in the processors are now quite productive, but there is a lot of work, and the processors are now being made multi-core, not the time not to do something multi-threaded.
It is not possible to split an existing server into several threads. Multi-threaded programming is a thin, complex thing that requires a lot of knowledge of the code you work with, and it is practically not embedded in the existing application. The code must be written from scratch.
Thus, a server was born, based on as many streams as possible: the world is divided into pieces of 64x64 chunks and each such piece processes chunks in one stream, one stream for processing urgent blocks, one stream for redstone and pistons, one stream for mobs, one stream for items, one stream for carts, one stream processes other Entity and other information about the world, one stream recalculates the light, four streams from different parts of the world save the world to disk, one stream renders a map, one stream maintains the server and a team d console, updates statistics. For players, a system is used that allows packet processing to be placed either on a separate stream for each player, or on a pool of streams, or on separate streams for each player. In this case, everything can be divided into several more streams: process the same type of objects in at least 20 different streams. As well as Netty (NIO) as a network engine, as opposed to standard I / O.
The development of a stable version of such a server, which does not have all the functionality, cost about 8 months of work for me alone without having experience. All code is designed for asynchronous access to all data. But it was worth it - quite recently we set a record of 559 people, who not only stood in one place, lagged and filmed on fraps, but passed a very big event with redstone, and at the same time felt comfortable.
The moral of this story is: if you expect that your project will be at least a little popular and you think that at least a little theoretically it is possible that there will be at least a few people on the server ... do not skimp on creating scalable architecture.
Waiting for your rotten tomatoes, suggestions for improving this article, as well as suggestions for what you would like to see in the next article, which someday will be.
The flow of thoughts may contain spelling errors, because was written in one breath.
Source: https://habr.com/ru/post/157921/
All Articles