Game client download tracking system

Downloading a game is a rather complicated mechanism that takes place in several stages, each of which may fail for one reason or another, whether it is a disconnection, a hardware failure, a banal firewall or restrictions on the provider side.

Diagnosing errors that occur during the download of the game client, in such a variety of conditions, is not at all a trivial task. And even the use of advanced solutions such as Google Analytics, does not allow to fully solve the problem. That is why we had to design and write our own system for tracking the download of the game client.

First, let's take a closer look at the download process of the game client. In a social network, our application is an iframe element with an HTML page (hereinafter referred to simply as a canvas), which is physically located on our servers. After loading the canvas, the game client is initialized, at the same stage we determine the availability of the flash player and its version. Then the game client is embedded in the page as an object tag. After that there are 3 main stages that are most interesting to us:
')

1. Actually downloading a file from a CDN (about 3 MB).
2. Connecting to the social network API and getting the necessary data.
3. Authorization on the game server and download resources.

Force majeure situations can occur at every stage, so we keep track of them. The game client, using the external interface, informs the canvas on the result of each of the steps (both successful and not). Canvas, in turn, contains a small javascript library, which stores the transmitted information. After all stages of the download data is sent to the server.

Based on this, you can briefly describe the main tasks in designing the system and highlight the following requirements for it:

• processing a large number of simultaneous requests — notifications and the fastest response to the request;
• identification of the country and region of the user by IP-address in real time;
• logging of various data that can be expanded or supplemented over time, and the provision of aggregated statistics, including in real time.
• providing statistics for many simultaneously connected clients in real time.

Obviously, the system must be able to cache data in order to quickly provide information for online monitoring, as well as dynamically scale. Since the data is received continuously, and statistics on countries and regions may be needed at any time, it is necessary to determine the geo-information as they arrive.

There was also the issue of updating data on the client side and the load on the server with simultaneous requests. To view statistics from the server, you must somehow get the data, and for this usually use the so-called technology of long polling.

When using long polling technology, the client sends an Ajax request to the server and waits for new data to appear on the server. After receiving the data, the server sends them to the client, after which the client sends a new request and waits again. The question arises: what will happen if several clients simultaneously execute such requests and the volume of data processed on them will be significant? Obviously, the server load will increase greatly. Knowing that in the overwhelming majority of cases the same requests are sent to the server, we wanted the server to process the unique request once and send the data to the users. That is, the push notification mechanism was used.

Selection of tools and technology

It was decided to implement the server part of the application on the .net architecture using classic ASHX handlers for incoming requests from gaming clients. Such handlers are a basic solution for processing web requests in the .net architecture, they have been used for a long time and work quickly due to the fact that the request is quickly passed through the IIS pipeline of the web server. We need only one handler, which handles requests from the client, sent via pixel. This refers to the approach when a 1x1 image is inserted on the client, where the URL is designed to transmit the necessary data to the server. Thus, a regular GET request is performed. On the server side, for each such request, it is necessary to determine the geo-information by IP address and save the data to the cache and database. These operations are “expensive”, that is, they require a certain time. Therefore, we use asynchronous handlers: during the request, only data validation is expected (performed quickly), after which the data is queued for asynchronous processing, and the client immediately receives a response.

For data storage, we chose the NoSQL database - MongoDB, which allows you to store entities on the principle of Code First, that is, the database structure defines the entity in the code. In addition, entities can be dynamically modified without the need to change the database structure each time, which allows you to save arbitrary objects transmitted from the client in JSON format and subsequently make various selections on them. In addition, you can dynamically create new collections for new data types, which is great for horizontal scaling. Virtually every unique tracking object is stored in its collection.

After analyzing the requests from the client, we realized that overwhelmingly the same requests are sent to the server from different clients, and only in specific cases they are different (for example, when the query criteria change). As an alternative to lond polling, we decided to use web sockets (WebSocket), allowing to organize two-way client – ​​server interaction and send requests only for opening / closing a channel and changing states. Thus, the number of client – ​​server requests is reduced. This approach allows you to update data on the server once and send push notifications to all subscribers. As a result, the data is updated instantly, as a result of an event on the server side, and the number of processed requests on the server is reduced.
In addition, you can organize work without using a web server directly (for example, in a service), and some trivial problems with cross-domain queries and security are also eliminated.

Notification processing

An asynchronous handler is a boxed solution, because in order to write your asynchronous handler, it is enough to inherit a class from IHttpAsyncHandler and implement the necessary methods and properties. We implemented a wrapper as an abstract class (HttpTaskAsyncHandler) that implements IHttpAsyncHandler. IAsyncResult was implemented through Tasks. As a result, HttpTaskAsyncHandler contains one mandatory implementation method that accepts HttpContext and returns Task:

public abstract Task ProcessRequestAsync(HttpContext context); 

To implement an asynchronous ASHX handler, it is enough to inherit it from HttpTaskAsyncHandler and implement ProcessRequestAsync:

  public class YourAsyncHandler : HttpTaskAsyncHandler { public override Task ProcessRequestAsync(HttpContext context) { return new Task(() => { //  }); } } 

Actually, the Task itself is created through the overridden ProcessRequestAsync in context:

  public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback callback, object extraData) { Task task = ProcessRequestAsync(context); if (task == null) return null; var returnValue = new TaskWrapperAsyncResult(task, extraData); if (callback != null) task.ContinueWith(_ => callback(returnValue )); //  . return retVal; } 

TaskWrapperAsyncResult is a wrapper class that implements IAsyncResult:

  public object AsyncState { get; private set; } public WaitHandle AsyncWaitHandle { get { return ((IAsyncResult)Task).AsyncWaitHandle; } } public bool CompletedSynchronously { get { return ((IAsyncResult)Task).CompletedSynchronously; } } public bool IsCompleted { get { return ((IAsyncResult)Task).IsCompleted; } } 

The EndProcessRequest method checks that the task is executed:

 public void EndProcessRequest(IAsyncResult result) { if (result == null) { throw new ArgumentNullException(); } var castResult = (TaskWrapperAsyncResult)result; castResult.Task.Wait(); } 

The asynchronous processing itself takes place via the ContinueWith call for our wrapper. Since ASHX handlers are far from new, standard calls to them look ugly: ../handlerName.ashx. As a result, you can write HandlerFactory, implementing IHttpHandlerFactory, or write a routing, implementing IRouteHandler:

 public class HttpHandlerRoute<T> : IRouteHandler { private readonly String _virtualPath; public HttpHandlerRoute(String virtualPath) { _virtualPath = virtualPath; } public IHttpHandler GetHttpHandler(RequestContext requestContext) { var httpHandler = (IHttpHandler)BuildManager.CreateInstanceFromVirtualPath(_virtualPath, typeof(T)); return httpHandler; } } 

After that, during initialization, you need to set routes for handlers:

 RouteTable.Routes.Add(new Route("notify", new HttpHandlerRoute<IHttpAsyncHandler>("~/Notify.ashx"))); 

When receiving a notification, the definition of the geodata by IP address is performed (the determination process will be described later), the information is stored in the cache and database.

Data caching

The cache is implemented on the basis of MemoryCache . It is cyclical and automatically deletes old data after the specified period of the instance has passed or when the specified amount is exceeded in memory. C cache is convenient to work and configure, for example, through .config:

 <system.runtime.caching> <memoryCache> <namedCaches> <add name="NotificationsCache" cacheMemoryLimitMegabytes="3000" physicalMemoryLimitPercentage="95" pollingInterval="00:05:00" /> </namedCaches> </memoryCache> </system.runtime.caching> 

The storage of data in the cache was organized in a distributed form, which made it possible to quickly obtain data for certain periods of time. The following is the cache layout:



In essence, the cache is a collection of key – value. Where key is a timestamp (String), and value is an event queue (Queue).

Cache events are stored in queues. Each queue has its own time key, in our case it is one minute. As a result, the key looks like [year_month_day_hour_minute]. Example format: “ntnKey_ {0: y_MM_dd_HH_mm}”.

When a new event is added (notification), a temporary key is determined in the cache and a queue is located, then the notification is added there. If the queue does not exist, a new key is created for the current key. Thus, to obtain data for a certain period, it is sufficient to determine temporary keys and receive queues with events. Further data can be converted as needed.

Data storage in MongoDB

In parallel with the caching, the notification is saved in MongoDB . You can read about all the features and advantages of this database on the official website. Now there are many different NoSQL databases, so the choice must be made based on personal preferences and tasks. From myself I want to say that it is easy and pleasant to work with Mongo, it is developing dynamically (version 3.0 was recently released with the new WiredTiger engine), there are .net providers for working with it, which are also optimized and updated. However, it is worth noting that the use of NoSQL databases is suitable for solving certain tasks and should be used as a replacement for relational databases only after in-depth analysis. Nowadays, an integrated approach is increasingly common - the use of two types in large applications. Typical applications include replacing the logging mechanism, working in conjunction with NodeJS without a backend, storing dynamically changing data and collections, projects with the CQRS + Event Sourcing architecture, and so on. If you need to select data from several collections and / or perform various manipulations with samples, it is better to use relational databases.



In the specific case, MongoDB came up perfectly and was used for quick uploading with grouping statistics, cyclic storage (the ability to set the lifetime of records using a special index), as well as geoservice (which will be described later).

The MongoDB C # /. NET Driver was used to work with the database (at the time of writing - version 1.1, version 2.0 was recently released). In general, the driver works well, but is embarrassed by the cumbersome use of objects like BsonDocument when writing queries.
Fragment:

  var countryCodeExistsMatch = new BsonDocument { { "$match", new BsonDocument { { "CountryCode", new BsonDocument { {"$exists", true}, {"$ne", BsonNull.Value} } } } } }; 

In version 1.1 there is already support for LINQ queries:

 Collection.AsQueryable().Where(s => s.Date > DateTime.UtcNow.AddMinutes(-1 * 10) .OrderBy(d => d.Date); 

But, as it turned out in practice when analyzing through a profiler, such requests are not converted to native (BsonDocument) requests. As a result, the entire collection is loaded into memory and moved, which is not very good for performance. Therefore it was necessary to write requests through BsonDocument and directives of base. Hopefully, the situation is fixed in the recently released .NET 2.0 Driver.

Mongo supports bulk operations (inserting / modifying / deleting several instances at once in a single operation), which is very useful and allows you to cope well with peak loads. We just save the events in the queue and have a working background-process that periodically gets N notifications and saves it via the Bulk Insert to the database. All you need to do is synchronize the streams or use Concurrent collections.

The database works fairly quickly in read mode, but indexes are recommended to increase the speed. As a result of proper indexing, the sampling rate can increase by about 10 times. Accordingly, when indexing increases the size of the database.

Another feature of Mongo is that it is maximally loaded into RAM, that is, when prompted, data remains in RAM and is unloaded as needed. You can set the maximum value of RAM available for the database.
There is also an interesting mechanism called Aggregation Pipeline . It allows you to perform several operations with data upon request. The results are transmitted through the pipeline and transformed at each stage.
An example would be a task when you need to select and group data and present the results in some form. This task can be represented as:

sample grouping provision ($ match, $ group, $ project and $ sort).

Below is an example of an event selection code grouped by country code:

 var countryCodeExistsMatch = new BsonDocument { { "$match", new BsonDocument { { "CountryCode", new BsonDocument { {"$exists", true}, {"$ne", BsonNull.Value} } } } } }; var groupping = new BsonDocument { { "$group", new BsonDocument { { "_id", new BsonDocument {{"CountryCode", "$CountryCode"}} }, { "value", new BsonDocument {{"$sum", 1}} } } } }; var project = new BsonDocument { { "$project", new BsonDocument { {"_id", 0}, {"hc-key","$_id.CountryCode"}, {"value", 1}, } } }; var sort = new BsonDocument { { "$sort", new BsonDocument { { "value", -1 } } } }; var pipelineMathces = requestData.GetPipelineMathchesFromRequest(); //   var pipeline = new List<BsonDocument>(pipelineMathces) { countryCodeExistsMatch, groupping, project, sort }; var aggrArgs = new AggregateArgs { Pipeline = pipeline, OutputMode = AggregateOutputMode.Inline }; 

The result of the selection will be an IEnumerable, which if necessary is easily converted to json:

  var result = Notifications.Aggregate(aggrArgs).ToJson(JsonWriterSettings); 

An interesting feature of working with the database is the ability to save a JSON object directly into a BSON document. Thus, it is possible to organize the storage of data from the client into the database, and the .net environment does not even need to know which object it is processing. Similarly, you can make and requests to the database from the client side. More details about all the features and features can be found in the official documentation.

In the next part of the article we will talk about the GeoIP service, which determines the geodata by request IP-address, web sockets, polling server implementations, AngularJS, Highcharts and will conduct a brief analysis of the system.