History of one highload project
How to write a highly loaded, multifunctional project together? What if there is no money and time, and need to open? Under the cut a bit of interesting information from personal experience.
The idea to develop a project that facilitates life in the management of Internet sites, came up with a small company from Minsk. Our team is small but ambitious, so it was decided to develop such a large-scale project together.
Since the days of the webmaster are heavy and clumsy, a lot of functionality was announced for the implementation, namely:
In general, it was necessary to write "Dominder" (from the English. Domain + reminder).
Useful, interesting, promising project, but bad luck - not enough time and resources. There are as many as two developers (the second only 3 months after the start of development was connected).
')
Ordered over the hill steep, imported design. They decided to write on Ruby on Rails, PostgreSQL was the main database.
They began to look for compromises and technical solutions, thanks to which the implementation with the current material grips can be realized.
Theydecided to spend less time on the quality of the implementation of the frontend than we would like. We'll finish it on the sly - the main thing is that nothing is straining the customers.
Since there is no time, then this or that method of implementation is selected on the basis of the parameters:
Sometimes, gritting his teeth, you have to write this or that piece of code, just because it's faster. Crooked, ugly, but efficiently and flexibly.
In parallel with the development was conducted PR product. How did this happen?
2 “planning streams” are built: development and PR. They should be filled uniformly based on the current needs of the project and the possibilities for its current implementation. For example, if a functional piece of the project is ready - it was immediately tested, tested and popularized. Or, if the current number of users is less than predicted, then beautiful baubles for clients were first developed (for example, a demo account), and the optimization of current algorithms was secondary. And vice versa, if clients began to come in a shaft, they would throw everything and begin to code severely. It turned out a kind of swing, but in another way, in my opinion, nothing.
Whois information on all domains is available through public pages, which are constantly, ruthlessly indexed by search engine bots. Pages were cached, and domains that are not related to current clients are moved to a separate server (the so-called vertical sharding).
Due to the fact that the history of pings must be stored indefinitely - all records are pressed in the "archives" according to the display algorithm in the graphs.
Using your own samopisny proxy server, for pulling whois from recorders. Some of them at first simply banned ip due to very frequent calls. As a result, it was possible to obtain information for half a million domains per day. Not the limit, but there was simply no time for revision.
Everything that needs quick, frequent access is ruthlessly shoved into NoSql.
After the launch of the project, there was a question about attracting users.
The same monitoring sites in the West has long been popular and it is used by millions. But to explain to our Soviet person that it was easier, more profitable and more convenient to pay a few dollars a month and be able to immediately find out that something had happened to the site turned out to be a rather difficult task. Many did not understand that monitoring sites (and our other services) is like insurance. Well, if nothing happened. And if it happens?
For tests we have added 140 Belarusian sites. As a result, it turned out that about 40% of sites at least once a week cease to be accessible (for a period of 5 minutes to several hours). And some sites just wildly sausage and obviously lose their customers.
Of course, there is such a formidable competitor in Russia, like Yandex.Metrica (and they often ask me what our differences are). But it happens that Metric does not immediately send notifications (sometimes it is several minutes late), there is no possibility of checking the availability of sites from different countries, it does not guarantee monitoring in some cases and it is necessary to install additional code on their sites.
Having limited resources after the opening of our project, we began to act on our own on all fronts: e-mail marketing, affiliate program, partners in the form of hosting companies and web studios, etc.
In the end, after all the tricks and jumps, we opened up and began to receive the first profit, while experiencing childish joy from each newly registered user.
The idea to develop a project that facilitates life in the management of Internet sites, came up with a small company from Minsk. Our team is small but ambitious, so it was decided to develop such a large-scale project together.
Since the days of the webmaster are heavy and clumsy, a lot of functionality was announced for the implementation, namely:
- the project has to pull every week whois information on all existing domains (more than 300 million are planned)
- to keep all whois information for all domains along with a history of changes;
- notify on the expiration of the domain registration and hosting;
- monitor sites at least once every 3 minutes and at least from 6 countries (beautifully display the time, response codes and schedules with this whole thing for each country);
- check the site for the presence of blacklists, as well as the presence of malicious filth;
- manage finances with the addition of cost / benefit transactions across sites (again with charts);
- manage notes, with the ability to add tags and attach files;
- notify customers via e-mail and sms, according to user settings;
- fasten an interesting affiliate program with discount coupons and links;
In general, it was necessary to write "Dominder" (from the English. Domain + reminder).
Useful, interesting, promising project, but bad luck - not enough time and resources. There are as many as two developers (the second only 3 months after the start of development was connected).
')
Ordered over the hill steep, imported design. They decided to write on Ruby on Rails, PostgreSQL was the main database.
They began to look for compromises and technical solutions, thanks to which the implementation with the current material grips can be realized.
Decision number One
They
Solution number two
Since there is no time, then this or that method of implementation is selected on the basis of the parameters:
- time costs;
- scalability;
- (current number of users in the system) implementation must keep.
Sometimes, gritting his teeth, you have to write this or that piece of code, just because it's faster. Crooked, ugly, but efficiently and flexibly.
Decision number three
In parallel with the development was conducted PR product. How did this happen?
2 “planning streams” are built: development and PR. They should be filled uniformly based on the current needs of the project and the possibilities for its current implementation. For example, if a functional piece of the project is ready - it was immediately tested, tested and popularized. Or, if the current number of users is less than predicted, then beautiful baubles for clients were first developed (for example, a demo account), and the optimization of current algorithms was secondary. And vice versa, if clients began to come in a shaft, they would throw everything and begin to code severely. It turned out a kind of swing, but in another way, in my opinion, nothing.
Solution number four
Whois information on all domains is available through public pages, which are constantly, ruthlessly indexed by search engine bots. Pages were cached, and domains that are not related to current clients are moved to a separate server (the so-called vertical sharding).
Solution number five
Due to the fact that the history of pings must be stored indefinitely - all records are pressed in the "archives" according to the display algorithm in the graphs.
Solution number Six
Using your own samopisny proxy server, for pulling whois from recorders. Some of them at first simply banned ip due to very frequent calls. As a result, it was possible to obtain information for half a million domains per day. Not the limit, but there was simply no time for revision.
Solution number seven
Everything that needs quick, frequent access is ruthlessly shoved into NoSql.
After the launch of the project, there was a question about attracting users.
The same monitoring sites in the West has long been popular and it is used by millions. But to explain to our Soviet person that it was easier, more profitable and more convenient to pay a few dollars a month and be able to immediately find out that something had happened to the site turned out to be a rather difficult task. Many did not understand that monitoring sites (and our other services) is like insurance. Well, if nothing happened. And if it happens?
For tests we have added 140 Belarusian sites. As a result, it turned out that about 40% of sites at least once a week cease to be accessible (for a period of 5 minutes to several hours). And some sites just wildly sausage and obviously lose their customers.
Of course, there is such a formidable competitor in Russia, like Yandex.Metrica (and they often ask me what our differences are). But it happens that Metric does not immediately send notifications (sometimes it is several minutes late), there is no possibility of checking the availability of sites from different countries, it does not guarantee monitoring in some cases and it is necessary to install additional code on their sites.
Having limited resources after the opening of our project, we began to act on our own on all fronts: e-mail marketing, affiliate program, partners in the form of hosting companies and web studios, etc.
In the end, after all the tricks and jumps, we opened up and began to receive the first profit, while experiencing childish joy from each newly registered user.
Source: https://habr.com/ru/post/183700/
All Articles