uid.me - service of personal pages, based on MongoDB and Mojolicious

Good afternoon, Habr!

We want to make a review post dedicated to our new project. The review will affect both the functionality and the technical part, we hope this will make the article interesting for both professional developers and those who read Habr in order to keep abreast of the Technology.

For those who are interested only in the technical side of the project, we recommend immediately moving on to the second part .
')
PART 1. Lyric

We are the development team of the personal pages service uid.me.
The personal page is, for example, like this:


http://uid.me/dikaya


http://uid.me/pavel_kudinov

Those who are not familiar with the western counterpart of our service should admit: the uid.me project begins its history as a clone-localization of the English-language service about.me

History of creation

Here is how it was. The company website builder uCoz, in which we work, for 8 years of its existence has accumulated in the depths of its data centers more than 35 million profiles created by webmasters, as well as numerous visitors to webmaster-created websites, forums and blogs.

All these people are united by the global authorization system uID:



Until today, every person registered in uCoz had a profile of this type:

http://3707164671.uid.me/



The about.me project was chosen as the best existing prototype of an individual page for each user of uCoz, which corresponds, in our opinion, to the modern trend of self-expression of the inhabitants of the Network of the beginning of the XXI century.

As in the case of about.me , we give the user:

1. The URL of the form uid.me/name_ , which has become quite useful for printing on a business card, can be specified as a homepage in skype, as well as mentioned on any media carrier.

2. The ability to combine a personal photo, high-resolution background image, basic information about yourself (such as a biography and area of ​​interest) into a single visual image.

3. Constructor, with which you can quickly and captivatingly give your personal page a unique look and overall visual consistency.




4. And finally, the most interesting: today, many of us are actively present in social networks. Someone closer formats Facebook and Vkontakte, someone is limited to microblogging Twitter and Instagram, some have their own popular channel on Youtube.

And here the rule is fair - the more social activity a person manifests, the sharper the question arises: “which social network is considered“ main ”?”.

We suggest using uid.me as a kind of personal business card online. Our service allows you to tie the most common social networks to your profile, and then you don’t have to choose which link to give with a new valuable acquaintance, indicate on your skype profile or put a signature on the forum.

Your posts, tweets and photos will automatically appear in your profile, and the general view will be used to display, and the “stream” function will create a common tape of events, combining everything that happens to you in chronological order.



By the way, if you want to create a personal page on uid.me , we recommend using automatic registration through a social network. When you click on any of the “ Login through ” buttons - a personal page will be instantly created without having to enter registration information!



Looking ahead, we want to say that close integration with social networks in the near future will significantly shift the project development plan.

The second version of the uid.me profiles, which is already under development, will have the main focus on the function of combining information from social networks into a single stream with a customized presentation of data.

In addition, it is planned to develop several interesting infographic widgets in the form of colorful graphs and charts representing information about your friends, travels, musical preferences and other interesting interesting facts that the system will be able to extract from your profiles and process automatically.

Perhaps it will look something like this:




PART 2. Technical

While developing uid.me under the wing of uCoz, we found ourselves in a rather unusual position: on the one hand, the entire project code was supposed to be written from scratch, on the other hand, on the day of release, the project automatically became highly loaded, as it had to import over 20 million profiles, even taking into account the fact that bot registrations and very ancient profiles did not pass the competition.

Nevertheless, we decided to do everything beautifully and with the use of fashionable technologies, thus conducting reconnaissance in force, having gained a lot of experience and potential leveling in the end.

The following were selected as success components:

0. Nginx. Where without him.

1. Database, out of the box is a decisive question of distributing data to several servers + fault tolerance when a server is physically dropped from a cluster for any reason. In this capacity, despite active holivars, MongoDB was chosen.

2. Flexible data scheme, which allows to pass the primary and subsequent phases of the functional prototyping without losses. Again, MongoDB helped, although it had to pay with resources for convenience, so getting the main answer to the question: “BSON - is it a luxury, or a modern means of transportation?” - is yet to come .

It is worth noting that the original mysql database of user profiles when converting to MongoDB format has grown 5 times. However, each profile has been enriched by an impressive amount of new data related to the uid.me functionality, so it’s not just the gluttony of the BSON flexible data scheme.

3. Honestly speaking, given the current trend towards the active use of dynamic JS interfaces (as well as immense respect for the technological breakthrough made by Google engineers in the development of V8 Javascript, it goes into the order bypassing all existing scripting languages ​​by the performance of dynamic compilation into machine code) crazy idea to apply node.js and close the circle of web development in JavaScript, while getting some fatty buns along with it ...

But they decided that “ one project is one new technology, and so far MongoDB is enough for us ” (c) Alexander Solovyov. By the way, whoever has not seen this report is a hit, we recommend it to the whole team!

As a result, we decided to leave corporate-familiar Perl as a server technology, but we managed to gain a second cosmic speed, leave the fast_cgi gravitational field and apply Mojolicious - a modern autonomous and adequate (not counting the author) web framework with routs, helpers, bridges, embedded support for asynchronous requests and other sweets that the modern developer should have.

4. Total asynchrony and data caching when interacting with social networks.

Speaking of the prototype of the project, it was noticed that the data from social networks obtained by the about.me service is not updated, loading only once - at the time of connecting the service. Probably, the cache update option is available to VIP users, but we were not able to get information from about.me . This made us think that it is possible to organize inter-server interaction and a caching system as efficiently as possible in order to minimize the risk of similar problems in the future.

Almost universally implemented OAuth2 and the similarity in the API organization of various social networks made it possible to successfully generalize the interaction.

Of course, at the prototype stage, all work with the API was synchronous, but blocking Hypnotoad workers for implementing API requests in a high-load project is an unambiguous luxury and waste. Fortunately, Mojolicious is built on a very decent, both interface and implementation, event-driven machine, thanks to which, by the way, every worker in the pool is able to simultaneously process not one (as in, say, mod_perl), but dozens of parallel requests , of course, provided that they contain a significant amount of asynchronous code.

By the way, considering that one of the main “frightening” arguments against the use of node.js is its total asynchrony, - Mojolicious can serve as an excellent mental bridge when you start developing within the framework of the classical synchronous paradigm, and complete, at least, having a significant part of the hybrid code (sync + async). To admit, now we are afraid of node.js much less and we hope to apply it in the subsequent projects.

Generally, uid.me was made on the principle of “no bicycles”, and a whole layer of fossil homemade works was sacrificed to Shiva, headed by the widely known kilobyte macro “ dw ” in narrow circles, since 2005, faithfully serving us and our close developers and which allowed DBIx :: Class to escape the transcendental horror at a difficult time. Bright memory.

And yet, when developing uid.me , one entertaining hack was born - this is a macro

take { … $take->('named_callback_slot_1') ... } process { my $taken = shift; … }, 

built on Mojo :: IOLoop-> delay and radically simplifying the entire cycle of operations related to the organization of named cascade asynchronous API interactions, including cascading exception handling (if you are interested, write to personal, share).

Returning to MongoDB

To put into practice something similar to NoSQL, the solution has been wanted since the times when it was not mainstream. Within the framework of those highload tasks that we had to deal with at the time, the following understanding gradually emerged:

1. Classic LAMP project starts with classic SQL database.
2. If the project becomes popular, it acquires the status of "highload", otherwise goto 1.
3. The “highload” status obliges us to think closely about caching, sharding, replication and
backup of what is stored in SQL DB.
4. The evolution of the data scheme of a living project becomes all the more painful, the more data is accumulated, and the more in demand, the more popular the project is.
5. As a result of all this, the ORM code begins to perform the functions of mutex, serialization / deserialization of data for memcached, primitive sharding, in particularly cruel situations - patches to ensure backward compatibility of the data schema (for it was not always possible to afford a large pass-through update of data) .

However, enough about the sad, the yard was harsh 2000'e.

The beginning of 2010's was illuminated by the emergence of several NoSQL solutions that promised to eliminate most of the problems of the growing highload project "out of the box." The appearance of open, ready-to-use NoSQL solutions was predicted by many, but, nevertheless, we were pleasantly surprised by the actual acquisition of a beautiful future.

After consulting with more extreme colleagues in terms of innovations, we decided to try MongoDB .

Studying a new technology for ourselves, we considered it logical to apply its capabilities to the maximum, hoping for the best (and therefore, a silver bullet from the box), hoping, however, to roll back to more classical techniques in those places where excessive insolence would have pushed us with interesting underwater stones.

By applying the possibilities to the maximum, we mean the following:

1. JSON data storage format allowed not to mess with the usual parent / child / x-relations in the data scheme with or without, limited to common sense. As a result, the nested structure of the main user object turned out to be greasy, but convenient. They boldly put in a bunch of checkboxes, display settings, small linked lists and all the other things that had previously led to the creation of a pack of near-user SQL tables.

2. In the data model, we added a general purpose code, which, at the interface prototyping stage, made it extremely pleasant to increase the JS functionality: by URL / profile / save, it became possible to send any JSON that extend the user object with new data, for example:

 user.save({ 'style.profile.top': '20px', 'style.caption.tags.color': 'rgba(30, 29, 38, 1)', 'info.first_name': '' }); 

All operations related to the activities of an authorized user were packed into a general dispatch function with a latent collector of 500 ms, combining various atomic edits into general packages.

As a result, the client-side developers were able to easily expand the structure of the user object, just starting to use new fields.

Of course, after the prototyping phase, the server part of / profile / save was equipped with contextual data filters, which cut off unknown fields and filtered values ​​for correctness.

There is only one problem left - users who have not had some fields at all could be stored in the database, since the last time they edited their profile before these fields arose. Ideally, I would like to have default values ​​for each field that will magically appear in any object retrieved from the database.

At the ORM level, we added forced extend of all extracted default data with values ​​for all non-existing fields.

The circle is closed.

We were able to dynamically expand the structure of the object, without resorting to end-to-end updates of the database, to work transparently with it not only from the server side, but also on the client-side, while the process of adding a new one turned out to be quite pleasant, and the transition from prototype to release was accompanied by exactly two actions:

1. Adding a rule for the data of the new field in the extend_rules filters.
2. Adding the expected default value for this field to default_user.

Here, perhaps, that's all. Thank you for your attention, we are waiting for you!

PS for sweets, nudity lovers: