Browser Development - Functional Level

Quite a lot of questions came after past articles regarding my direct role in the life of the project - it all came down to the desire to learn technical details that did not constitute the basic logic of the world, but directly support the existence of everything that was planned. Some questions have already been answered, but some moments remained behind the scenes. For a long time I tried to figure out what is so good I can tell about the system, which would not be “banal solutions”, but it would be really unusual. Those, really archival and unusual, in my opinion, moments were not found.

Of course, the source contains some interesting places, but they are specific specifically for our project and are not suitable for everyone. I want to tell about these “places” that stand out slightly from the general mass of the functional, but one should not expect the level of “monsters” of the industry - all solutions are deeply integrated into the logic of the project itself and are its conclusions from the tasks set.

Like last time, the story will be conducted from the point of view of one of the team members involved in the development of the project from its inception - the programmer, that is, this time - me.
')

Developer labor

No matter how much I looked at the block from “hantim” on the pages of “habr”, no matter how many suggestions I had to take to participate in projects, I was always skeptical of them. Usually there is a “complete team”, but only a developer is missing, and the offer consists of a tiny share and a huge amount of work.

Until now, I can not answer the question - what did move me when I agreed to participate in our project? It was clear right away - the work would be delayed not for one year, but the real physically tangible profit may not follow at all. Apparently, the stars so formed that I was unexpectedly loyal to the team, and the team accepted me.

Distant acquaintances called close and distant acquaintances, as a result a team formed, as a result close acquaintances left the team, as a result distant acquaintances became close friends.

Thanks to this project, the amount of experience gained was always stable regardless of my main job, where the flow of new knowledge usually ended in a couple of months and the routine began. At least, I hoped that this project would be an additional advantage for me when applying for a new job, if necessary. However, later it turned out that employers are not interested in this - they were always interested only in previous jobs, for which I received a direct reward.

The only thing left to do was to lead your project with the hope that one day it would become the main one, and I would work side by side with my comrades in spirit. Well, experience means experience.

Do not trample, I just swept

The first thing I want to say about the development of the project - no matter what its scale and complexity - keep it as clean as possible. If not for other developers who will come after you, then for yourself. How many times I came across in the drafting of other people's projects, as many times I was surprised at how people can live in such a mess.

Directly at the start, we had only one module - the “administration panel”, the rest (user part) was slowly “sawed” in the group of controllers outside the module. But this was the limit when the number of controllers exceeded a dozen, and the module of the administration panel with its set of directories ceased to fit into the screen when viewing the tree in the IDE. The categorical solution was to completely rework the structure of the project and pack everything into various modules, including embedded modules for the administration panel. This resulted in a project of a certain “Zen” and easy accessibility of each section. Thanks to the quick “line-by-line replacement,” they also aligned the base, although later they still had to get their hands on it.

Such global manipulations are much easier transferred when the amount of code just comes to a critical point, and the project is not yet in combat conditions - this will save from further complex refactoring, and will allow you to continue to maintain a strong project structure.

With a database size of 135 tables, it is also impossible to arrange a heap in the model folder - from the very beginnings of the modelka project they are placed in directories with a triple nesting so as not to create chaos (even sorting alphabetically does not save from searching in 30 files starting with Bird ~). Someone might think that 2-4 clicks when searching for the right file for editing is a lot, but for me and the project coordinator this procedure turned out to be much better than trying to scroll with the mouse wheel in search of one file from one hundred and fifty.

Any strict project structure that complements the basic architecture will always make it easier to navigate in it, and the code automatically becomes a mark higher in quality. Even with dubious solutions in a hurry, I still don’t consider the project poorly written and I believe that when it comes time to take on an additional programmers team, they will easily find their way around the code.

At the same time, it was clear to me from the beginning that our coordinator would play his part in writing the project code - in the past he made his websites, how much he could and how much his knowledge was enough, but he still cannot replace a full-fledged developer. However, at the same time, he can easily save me from the task of spot-changing coefficients or correcting minor flaws in the layout, and this was one of the reasons for additional improvements in the structure of the code.

All formulas in which there are numerical coefficients, and which are involved in critical calculations, are fully rendered into a separate Formula class. First, we got rid of the presence of constants in the code - we just write numbers in formulas, and there is nothing wrong with that, since the class and its methods are intended for this purpose. Secondly, we got a single point of the system, which allows you to edit the balance of the game, without running through the rest of the project.

In addition, Yii has ready-made mechanisms to support multilingualism - in our project several languages ​​are not supposed yet, however this mechanism helped us to separate literary texts describing errors from the “mechanical” ones written by me in the process of creating a functional. And again - all the texts of errors or successful actions are collected in several files, and there is no need to search for texts throughout the project.

On the other side of the code

Since our entire project is built on the user’s waiting time for “accomplishing an action” (end of work, completion of the timer, updating the list of something, etc.), undoubtedly, most of this work should be performed by the server regardless of the user's online presence .

Some assumed that the work of the “server” part of the project required additional applications written in a more “correct” programming language, newfangled technologies, and so on. However, in conditions of limited baggage of knowledge and, certainly, not limitless nerves and patience, we could not afford to introduce technologies that we had not studied until the end, and were based on what we have. Accordingly, since the site’s “face” is based on PHP with the Yii framework, the project’s backside works on the same wheels.

At the same time, it is not at all necessary that we will feel a performance drawdown soon enough - we can afford some tricks that temporarily neutralize the lack of technology. And if we really have such a problem, it will mean the popularity of the project and the availability of funds for development.

Currently, the project works on the basis of forty-five entries in the list of background tasks (with some repetition, but not the essence) - this is a lot or a little, everyone will decide for himself, but this is clearly not the limit. Here is our list:



The most important tasks - the calculation of the results of deferred user actions - are placed in a single task flow, called tick, which is running 100% of the time the server is running. Of course, the PHP daemon is not something that can be made to work normally with the standard thinking of a PHP developer, when there is confidence that the application should die after its launch, so the tick works 10 minutes apart, after which restart the task. To protect against some unforeseen circumstances, there is a setting of mutex flags in work cycles, and the start of a new process is slightly superimposed on the end of the previous one in order to avoid idle time.

Speaking of “tick” - his work must be fast, and it does not tolerate any delays even for a fraction of a second, since on the other side of the system the user can wait for the page to automatically reload after the action timer expires, respectively, the calculation time accuracy is very important in this situation. We cannot calculate in advance and charge the in-game currency immediately after the start of the task - by definition, the task can be abandoned, at the time of the work the player can be attacked by another player and can take away some of the funds available at that time. In this case, we resort to a small trick and calculate the results in the background task for a couple of seconds before the expiration of the work - during these seconds, the user simply does not have time to understand what happened if he suddenly opens the page before the appointed time and sees the letter of accrued reward. For these two seconds at the time of peak load, the server will be able to create a queue of players waiting for the completion of tasks, and have time to complete them before the onset of time "x".

In order not to load the server with constant samples from the database in order to search for new users who are about to get the result of work, the “tick” task has a waiting mechanism that allows you to dynamically calculate the idle time. In a simpler language, the server does not make unnecessary samples during the operation of a “tick”, if there are no users who are soon waiting to complete the work.

In order to have the structure of a list of background tasks and not lose anything during a deployment, the list is automatically compiled from a separate config where data about each task is stored in a multidimensional array. The list of commands is generated in a readable form using a separate console command “php yiic.php crontab”, which is actually manually deployed by one command via “php yiic.php crontab | crontab –u production - “, rewriting the server’s task list for the desired environment.

Remember everything

Who would know that the cache will be one of the most cruel things on the project - since the appearance of its primordia in the code, we constantly ran into unconventional situations when the cache interfered with normal functioning. Gradually gaining experience, we had to solve caching issues with almost radical methods.

Firstly, about half of the project is kept on the cache - it is impossible to operate the system adequately quickly, where so much information is shown on the character's main page that half of the database is used for this. Part of the data updates the server in the background by periodic calculations and forced rewriting of the cached data (for example, part of the statistics). Part of the data simply loses its relevance and is loaded as the user requests (for example, the number of people online). And a piece of data is updated as certain events occur - in particular, “the arrival of a new letter” updates the list of recent letters in the “sidebar”.

Secondly, a huge reservoir of sampling from the database is kept on the cache. There is information on the project that does not change at the user's request, but only by a strict administration decree - the characteristics of items (except for the invention of the invention), the methods of currency extraction (texts for them, settings for them), characteristics of buildings, weather, awards, gifts - about 35 table names in total. The correct decision, in my opinion, was to use them in a permanent cache until the “above” command came to change the data. Models will resist, put a stick in the wheel, but the speed increase is worth it.

For the work of the permanent cache of selected models, a tiny interlayer was added for ActiveRecord, which, when trying to access CDbConnection, determined the need to indicate the use of the cache - and, when convenient, substituted the corresponding directive with standard methods. In fact, I created a sufficient amount of headaches for myself on the subject of how caching is established in the interlayer. Check out the Yii source and see what happens when you install “Model :: model () -> cache ()”. For the lazy, I’ll say that in this case, the instruction comes in “cache the next query in the database”, and there is no guarantee that the “next query” will be from the source of the instruction.

An additional problem of such a cache operation scheme was the entry point in the cache - the CDbConnection call occurred both for the main search model, and for all models that go in the relational bundle (if such are specified in the query), and the call from related-models occurs later than from the main one, respectively, all requests with join inevitably fell into the permanent cache.

Having put a few crutches in the form “I’m absolutely sure you don’t need to cache this wonderful query with the join of the static model,” we got a workable solution, but only under one condition - all the samples for the “static” models should be made strictly without join'ov where this is possible. That is, if a table with a list of items of the character is displayed on the page, this means that one request was made to the bird_item table for the bird_id <-> item_id bundle, and the darkness of individual requests for obtaining individual data for each item that magically lies in the cache. For reference, before the cache “warms up,” more than 230 queries are sent to the database on the character’s main page. After "warming up" - only 19.


The principle of such a cache was introduced not from the very “diaper” of the project, but approximately several months before alpha testing. The found “bugs” of the hooked up joins of the “static” model were corrected already on the working version of the project. Therefore, it is necessary to operate with such things with extreme caution.

Template code

Despite the fact that I have some experience with large projects, I have never had to tinker with design patterns (well, except maybe standard MVC and singleton, but these don't count). I even studied this topic more than once, but it just didn’t put off in my head, and if I needed to implement some kind of architectural feature, I never popped into my memory.

However, as it turned out a little later, if you “re-invent” the pattern yourself, it will constantly migrate from project to project, but it’s far from the fact that it will be understood once that it is actually “he” - it will simply be “good decision "that will become a habit.

With this pattern, I had a “strategy”, with the help of which so-called “switchable behaviors” were implemented in my projects. In Yii, there is a wonderful thing - behavior - which allows you to organize "horizontal" class inheritance (roughly speaking, trait, but implemented at the level of PHP code inside the framework for a long time). It is on such classes-behaviors that certain elements of the system are based that require different results of work, depending on the state of the object.

I will give an example. The project implemented the weather conditions - fog, sunny, rain, cloudy, storm and several other options - which carry various effects that affect the behavior of the game. For example, in cloudy weather, a player can be given a bonus of “additional chance” to find game currency, or when sunny weather sets in after a storm, there is a chance of sending a “storm ration” to all players, including a small cash allowance “for restoring after bad weather”.

Thus, each type of weather can contain one or several effects, which should have an iron base in the form of a code and ordinary parameters in the database. The canvas of conditional operators in this situation will not succeed. In our case, everything was decided through “switchable behaviors”.

First, a basic class-behavior is needed, which by default will always be connected to the operated model (it is possible and not to the model, but for the current task — specifically to it), which should contain methods for choosing the desired final behavior and methods stubs for returning default values ​​if the final behavior is not used. Secondly, we need a set of behavior classes that actually perform the necessary functions.

For weather conditions, a list of methods was selected that actually constitute effects checks - this is “obtaining a coefficient” (respectively, there is a call and use of an additional factor for calculations in the right places of the project), “whether the building is active” (with appropriate checks in the formulas) , "Whether the skill is active," "execute when activated" (method to perform when a new weather effect occurs). In the base class, all these methods are represented with checking whether the final behavior is connected and returning the default values, while the final behavior classes contain the actual code (which is small enough and easy to navigate, easy to edit). When checking (trying to perform one of the functions for the first time), the current base class is disconnected from the operated model and the final behavior class is connected.

In fact, a certain toggle switch works, which connects the required class depending on the state of the object.


Yes, it complicates the structure of the model, adds additional checks, connecting and disconnecting classes (and, of course, it gives more load to the interpreter), but it makes it very, very much easier to operate with the same type of code, which differs technically and repeats structurally.

System collapse

I have already told earlier that a sufficient part of the time spent working on the project was spent on thinking about possible critical situations that arise in the system. Whether it is a deliberate user attack or a code error that has occurred, this must somehow be handled by the system.

There are especially important parts of the system, without which the normal functioning of the game is almost impossible - for example, the work of the “tick” team, on which the calculation of all users' performance depends. If an error occurs in the operation of such an operation \ method \ function - it does not matter - the system should react just as critically.

In our case, the “curfew” mode was added, which implies the complete inactivity of all players, the closure of all game functionality from users and the complete cessation of any background project tasks. It would seem that it is a simple solution to implement, but it completely helps to prevent the appearance of “damaged” data (for example, incorrect multiple charges of funds that occurred due to a failure in the middle of the accrual function). In the process of debugging on alpha testing, such protection worked more than once during the night, when no one from the administration monitored the state of the system - it’s better to get an inaccessible game for a couple of hours than to skew the data.

This same mode of operation was later useful to us for creating a project deployment system. We implemented the hot “update code” function, which allows you to automatically start the timer before closing the game, then turn on the curfew, update the code from the repository, start the timer before opening the game, and after the specified time, let the waiting users into the game . For the sake of security, it is impossible to update the project code on behalf of the user operating the web server, therefore a specially trained individual user of the system does it for the rest, checking for the presence of such an indication with a certain periodicity.

This approach to updates allows you to enter painlessly small iterations of the code into the project, without interfering with the administrators to preliminarily check the integrity of the update, and notify users about the event.

In addition to the above, a possible “crutch” of data synchronization between versions of a project — for example, the data from the test server, if changed, is copied to the repository every night as possible against the occurrence of problems with project settings that are stored in “static” models. where they immediately get to the development server in order to have an up-to-date environment for preliminary testing of new code. That is, if the data has been changed, then in addition to resetting their permanent cache, the task to save them to the repository is put “active”. And only if the project server is marked as “source” (for example, the test server is one and the dev server is not) - so we can restore the old values ​​or transfer the values ​​to other versions of the project.

Data storage in the repository is carried out without any tricks - these are ordinary files with serialize () - the contents of the models, where one file has one table entry (the amount of data at the moment is such that one file is not enough for all the data in order to avoid problems with lack of memory during serialization-deserialization).

Divide to recover

Until the team had a system administrator, we had to rely only on our own strength and knowledge in the matter of creating, storing and restoring backups in case of unforeseen situations.

Now we have implemented a database backup bypassing mysqldump and locking the database, which allows you to restore the archive much faster than it would be with the help of a packet of requests from a dump. But also recovery is possible only entirely, at once all available bases on the server. Soon, we are promised another backup-recovery principle - already with the help of special utilities from Percona, but I can’t say anything about them yet.

For your own comfort at the start of the project, a system for dividing the project database into two parts was invented - one full-fledged self-contained part (including the entire set of tables), and the second one, including only data that can be lost for a short time without disturbing the entire project. In other words, the second part - the “den” database - contained tables with letters, system logs, a history of battles and other similar data that are accumulated in very large numbers, but do not have real value for the overall system. By default, all the “log” data is recorded in a separate database if access to it is available, and if for some reason access to the second database disappears, the system switches to the main database (where, by default, there is no data in the required tables, but in they can temporarily record new data that will later be transferred to an additional database).

Thus, if we suddenly burn down the server and don’t have a hot swap, but only dumps of two databases, we will restore the main database (with important information that supports the entire game) in 5-20 minutes and launch the project further, warning players that a collapse has occurred, and the rest of the data will be pulled up later. And in a few hours, when the additional base dump is loaded to a new location, it will be possible to switch the system to work with two bases.

According to preliminary calculations, the ratio of the sizes of the two bases was 1 to 70 — that is, the history of battles, letters, and other massive “husks” on a project is many times more than critical data.

The implementation of this idea turned out to be not very difficult - just need to redefine the CDbConnection search method in the model, which, depending on the type of model, tries to connect to the required database. But at the same time there are some minuses (or maybe not even minuses) - any “manual” spelling of SQL queries must be written using the tableName () method from the table model in use, where the correct database name was automatically substituted into the table name .

Bike or other people's slippers

It is imperative that during the development a question arises whether to use someone else's work in some part of the system or do it yourself. Even with a large catalog of extensions for the framework, this question is still not easy in view of the questionable quality of the code of other extensions.

From what is actually used on the project, in no way did only the new-fangled debag-toolbar, which had carefully transferred the framework from the new version of Yii2 to the version code Yii1.1, did not finish it in any way. Everything else was either superficially written to get rid of the appearing "notices" during the work of extensions (for example, swiftMailer, where variables were not periodically initialized or were not checked for the presence of array elements), or were almost completely rewritten (as in the case of XDbParam, EMutex, where, in principle, the code did not suit the content and quality, but at least allowed to be based on it in terms of general ideas).

A great example of the choice of agony can serve as an introduction to the draft forum, and in this matter we chose to create our own, rather than installing one of the many ready-made options. Firstly, I do not have enough experience and perseverance to easily create a design theme for the forum, if we chose the finished version. Secondly, a third-party forum would still have to be completed in order to implement its functionality linking the player and the forum (avatars, names, ratings, links to players, etc.) - yes, it is quite possible that there are “bridges” for the connection between the framework and forum, but no matter how much time we spent on bringing it to mind.

Creating our own forum allowed us to make the minimum necessary functionality based on the finished project, and the time spent on developing the basic framework turned out to be so insignificant that it can be ignored. Moreover, the forum code was completely in the project repository, in the migration to databases, in the division into two databases and the automatic archiving of “static” data. The benefit of our own development has blocked for us "the ability to immediately have ready-made functionality in a large volume." The forum is being gradually added (new permissive BB codes are introduced, new message samples are selected, and other improvements), but the project did not stop due to possible difficulties arising from creating a typesetting theme.

All other elements of the system go through a similar approach to the selection. - , - , , .

–

« » « », . « » , , , .

- Hetzner – vq7 vq12, memcache «-». , . «» « » – «» . «» – DNS- ip-, .

«» EX6S Hetzner. - nginx php-fpm ( PHP 5.4, 5.5), – Percona Server ( MySQL, «» «» ), Memcache. APC-, PHP 5.5 OPcache – , .

Pinba Zabbix. «» , Zabbix .



2.5 – , 1 – ( ), 500 – ( ).

, , — . CActiveRecord, «master-slave» — . , .

nosql – . , .

– , «» sql- , .

, . – , .

( , ) – . , . , – , , .

, — 9500 , 2 , . - , , , .

– . , , .