Bomberman Online and habra effect - 450 players on the same map. Report and details of the game engine

As promised in the topic-announcement of our game , we post a report on the habraeffect and details of the game engine.


First of all, we want to thank everyone who participates (both with the game and with the practical suggestions and ideas) in our testing of the first and current versions of the engine and gameplay. As practice has shown, during the habraeffekt bugs are and fix much faster :)
')

Statistics for 3 weeks

Players registered: 4 663
Played rounds: 2 252
Number of players (including guests): 13 402
Number of frags: 469,534
Deaths: 545 957
Number of goals scored: 1,707 ( WTF ?)

[during habraeffekta]
Online Players (Maximum): 453
CPU load (according to the munin schedule): 168% (16 cores - 1600%)
Memory Usage: 1.65 GB
Traffic (maximum): outgoing 24 Mbps | incoming 9 Mbps

(The graphics are clickable)

[Engine] Connections [ 1 ]	[Engine] Main cycle time [ 2 ]	[Engine] Bytes per send [ 3 ]
[Server] CPU usage	[Server] Memory usage

TTX

Hosting: a dedicated server in the data center Vladimir (which courtesy of Parking.ru for the time of habratestirovaniya)

1. Java & Jetty game engine
2. HAProxy - proxying websockets
3. Nginx - statics and proxying to the node
4. Web functionality - Node.js (packages: express, mysql, request, nodemailer, validator, moment, forever)
5. ClientSide - GWT, broadcast in javascript

Details of the game engine

Next - from the face of Jedi_Knight (the author of the engine).

Development of gaming applications on canvas is a promising direction in which many development teams are now moving, for example, the Estonian project mightyfingers.com

Some ideas are taken from the article www.html5rocks.com/en/tutorials/canvas/performance

HTML5 canvas Basics

Terminology: a tile is one square, for example 32 × 32; chunk - 8 × 8 tiles.

Consider the features of the tile engine. Here is the model I came to:

1) Subpixel drawing for tiles is simply not needed.

2) Several transparent canvas-s located on top of each other are used. If terrain was not modified, there was no scrolling, then you can not redraw it - this gives a small performance boost (compared to cache optimization).

3) As in the game model, the whole field is divided into 8 × 8 chunks. Since 64 drawImage calls for 32 × 32 images cost much more than a single call for 256 × 256, visible chunks are drawn into a cache consisting of two layers: under the player and above it.

4) The cache is associative, that is, a pair of canvass is used by several chunks on the map: a chunk with coordinates (X, Y) uses a cache with coordinates (X mod A, Y mod B) where (A, B) is the cache size. At the same time, cache sizes should be such that two chunks with a common cache do not appear on the screen at the same time, there will be a race.

5) In addition to the canvass, cache numbers in the cache are stored in the cache, which are generated by the logical map. By the number you can uniquely draw the tile.

6) When changing the visible chunk, the corresponding object in the cache is marked as dirty, requiring redrawing. At the same time, neighboring chunks can also be marked as dirty: you never know, suddenly a new shadow falls there, or there you have to draw a cut of the stone.

7) In case if it is necessary to redraw a dirty or new chunk, the table 10 × 10 is put in the buffer and the surrounding tiles. This buffer generates an array of tile numbers - it is already 8 × 8. The tile number can also contain information about the shadow falling on this tile. In those places where the tile number has changed since last, a redraw occurs.

8) For tiles, it is more convenient to use not drawImage, but createPattern and fillRect. fillRect is also more convenient because they can draw several identical tiles in a row, and this will be faster.
This test reveals a strange effect: that for textures 32 × 32 there is a significant performance increase on the iPad 2.5 times:
jsperf.com/canvas-texture-draw-vs-fill
Comments with the addition of TheShock

9) And finally, why all this is necessary: ​​when scrolling instead of drawing a heap of small images 32 × 32, drawing from the cache takes place. The drawImage call is much smaller, it seriously affects performance.

Green, yellow, shadows - are drawn on the first layer, each 32 × 32 square can be drawn independently of the others, since the tile number contains information about the substrate, wall and shadow.
The second layer is drawn in red, it is shifted 8 pixels up.

Found bugs

So, during the collective testing the following problems were discovered:

1) On a large map of a certain size, the tile cache suddenly started up, which led to wild brakes that only chrome could handle. The whole thing was in the periodic map: the card sizes (W, H) should be divided by the cache sizes (A, B).



Let W, H = 7 A, B = 3, corner chunks fall on one cache position, and on a periodic map they fall on one screen, which leads to a race. The same situation is observed throughout the “seam” of the card.

2) Client profiling revealed bottlenecks in code using java-collections. In these places, the collections are replaced by samopisnye structures.

3) Changing the network protocol and switching from utf8 to binary web-sites in those browsers where they are supported (Chrome, Firefox) allowed reducing the average client traffic to 12 Kbps. The native ArrayBuffer is used, I note that Firefox does not implement the DataView class, so we manage with the usual Uint8Array.

The graphs at the beginning of the article show:

[1] Connections - the number of active connections to the server.

[2] Main cycle time - how long the different stages of the main cycle occupy. Tick ​​lasts 100ms, preprocess - checking commands from clients, tick - update game state, Observers - forming messages for clients. Total - all together, plus the time it takes for messages to go to the network buffer. It is measured in milliseconds within one second, so that when Total approaches the 1000 server badly, overhead starts to appear.

[3] Bytes per send — net traffic sent to clients, without websocket frame headers, in bytes / sec.

Gameplay Changes

Almost all the time it took to accelerate the engine, client and network protocol. Serious work was done on optimization and refactoring, eliminated a lot of bugs.

But I managed to do a little more by the gameplay: it became easier to lock up the enemy. But most importantly - added a ball and gate! Those who score receive a rescue shield. Particularly agile collect 3 shields and run like superheroes :)

New load habratesting. The goal is 1000 players online on a single card.

For the first time, the server without problems and lags kept 350 players online. Today it is ready for 1000.

We invite everyone on June 8, 9 and 10, from 17 to 22 to try to achieve a new bar of players online. Join us , call friends / colleagues - we are waiting for you!
(if someone doesn’t have a domain, go directly: 141.101.245.117 - not everyone has updated their dns after the move)

Especially for habrayusers, we have prepared a Habramonitor , which shows graphs of the current load on the engine and server (with an interval of 5 minutes).

And may the Force and the Fan be with you.