“Evolution of Development” // Reports from the Mail.Ru Technology Forum 2011: text of the report, video, presentation
With the report “Evolution of Development” I open a series of transcripts from the Mail.Ru 2011 Technology Forum . This transcript was received on the day of the event, November 16, 2011. Details on how the report decryption system works - see the article “Reverse” of the Mail.Ru Technologies Forum: High-tech in event-management
( Download video version for mobile devices - iOS / Android H.264 480x368, size 204 Mb, video bitrate 500 kbps, audio - 64 kbps)
( Download video version of higher resolution H.264 624x480, size 610 Mb, video bitrate 1500 kbps, audio - 128 kbps)
( Download presentation slides, 2Mb)
Introduction by the Vice-President of the Mail.Ru Group, Technical Director Gabrielyan Vladimir
- Colleagues, I hope you enjoyed the dinner. Satisfied, kind, contented. Write to Twitter. Actually, I want to introduce our next speaker. You know, in every company there is a person who can be called “Mr.“ I know how ”, a person who is in the know of all technologies, a person who has very correct visionary thinking, and he largely determines our technological policy in“ Mail.ru Group ". This is Igor Ermakov , Deputy Technical Director, that is, me. Igor today will tell us about how we have evolved in our technologies over 13 years, and if you listen carefully to this report, he will actually help you skip the stages of development that we have already passed, and go straight to those technologies. that we use in Mail.Ru. Igor!
')
Ermakov Igor:
- Hello everyone again, very glad to see you all here, thank you for coming. Today we will talk with you about our evolution, about our development. Despite the presentation of Volodya, thank him very much, I will not tell you how to do something, I will tell you how it was, why it was and where we are now. What is evolution? The classic definition from Wikipedia is a process of development, and it is important that this is a process of development without sudden changes.
This is not a revolution, but a smooth change. I want to talk to you about the evolution in data storage, about tools. When I talk about tools, I do not mean editors and all sorts of utilities, I talk about languages and technologies. I want to talk with you about client-development and about changes in approaches to mobile development.
A tour of history.
On screenshots - our first designs. Maybe someone remembers? Even here is the design with our octopus. Yes, the first design with an octopus.
DATA STORAGE
MySQL , why MySQL ? We are not talking about other repositories, because the classic web project was then built on MySQL. Of course, the mail has its own service for storing letters, the first versions were written at that time. Data letters stored in the database is extremely inefficient. I knew one such postal service, he ended up very badly. They kept all the letters in the database. Everyone remembers its advantages - low cost, simplicity, compared with, of course, large corporate solutions, for a regular web developer - free, there was quite a lot of documentation. Nevertheless, it was familiar, because it was built on some kind of MySQL foot. Everyone remembers that MySQL on flat plates showed quite high performance, thank you very much. Nevertheless, there was a set of minuses. Just want to talk about the part of the minuses - this is an unmanaged cache, this is resetting the Query cache when updating the tables.
And it would seem, when we have a problem with the database and it would be good now to cancel the hard drives and put everything in HEAP. But the implementation of HEAP tables was just terrible in MySQL, and it remains very slow. Accordingly, the solution that everyone had in his head was to put everything into memory, memcached appears. What does he give us? It provides very efficient caching, very fast storage and data access in memory. Very simple and performs only one function. And quite often, despite the fact that memcached is still a hash, and it is not required to store data, memcached begins to be used as a kind of data storage. This is bad because memcached is not a data warehouse. He is not obliged to remember what you put in him, he can delete the key at any time, even if it has not been cached. The most important thing is everything in memory. If you have something wrong with the data, with the machine, everything - you have lost the data, and where it is unknown, there is no way to roll back. Of course, a good solution - it would be easy to take and get, on the one hand, at least some reliability of MySQL data and a memcached rate of fire. I remember there was such a project MemcacheDB , terribly slow, everything slowed down, did not pull. As a result, there are several solutions that we call: NoSQL .
NoSQL . They brought us performance, the lack of unnecessary functionality, which were burdened with implementation databases. Some of them already offer out-of-the-box solutions for scalability, for automatic scalability, and of a wealth of choices. Kostya talked about our " Tarantula ", probably compared with Redis , many people know Apache Cassandra , MongoDB , these are all kinds of NoSQL solutions that cover certain problems.
We, accordingly, use mostly, not even in the mail, but in the portal, Tarantool / BOX , our development, Kostya talked about this a lot.
DEVELOPMENT LANGUAGES: SERVER-SIDE
Let's talk about development languages. Back to the past - we have C ++ , Perl , somewhere in the environment C # , Java , PHP , ASP.NET are used , all the technologies are beautiful, good, over time new languages of Python and Ruby appear.
Although, to say that only appear - this is somewhat deceiving, because Python , if my memory serves me, is the year
The most interesting thing in frameworks is that they involve developers in working on these frameworks, and thus developers begin to improve, improve, that is, such a snowball.
What is the result, what are we arriving at now at Mail.Ru? Now Mail.Ru is a language compiled: C ++, of course, this is Perl, we have a lot of Perl code, we continue to support it, this is Java, Roman was telling now - we have Java code. We write C #, we write in Python, we write in Ruby, we also use Objective C , if someone asked Kostya on the sidelines, he probably told why we use Objective C in Tarantool
On the one hand, it seems that this is a zoo of technologies, it seems that we have a lot of solutions and there is no single strategy. But, on the other hand, I want to tell you that each of these technologies is justified. Each of them solves some of its task and is quite ideal in the application. And in the development of our company, there was some change in recruiting new staff, we now have requirements for a programmer ... We really don’t like a certain “holivar” when developers come and say: “You know, I don’t like PHP, because it’s crap” and Python is cool because it is fashionable. ” And at this moment we have such a kind of light bulb during the interview, which says that there is probably a problem with this person, because there is a set of tools that you need to use at the right time at the right time, and we treat programming languages exactly as tools: let's use them exactly where they are needed, and not because we like them.
DEVELOPMENT ON "CUSTOMER"
The ninety-first year, the appearance of HTTP , text pages, the first browser, the first HTTP request. On the screen - my favorite question at the interview with a web developer: write me a simple HTTP request. Very often, developers start writing to me something from HTTP 1.1, from HTTP 1.0 with a bunch of all sorts of headers. I just ask for five characters and a line break. In general, all modern web servers that I know, they all perfectly support the 0.9 protocol. The ninety-second year, the appearance of one of the text browsers Lynx , and a little later it is Netscape Navigator 3 and Internet Explorer 3 . Maybe, who remembers, these were already powerful graphical browsers that allowed to show that HTML 3.0.
The end of the nineties is the emergence of Mail.Ru , this is HTML, Dynamic HTML , JavaScript , IFrame , which brings us the first opportunity to work asynchronously with the client on the client-side.
That is, we get into our own hands technology that allows us not to overload the page, to update the data, without overloading the page. Microsoft, by the way, first used it, if I remember correctly, to update stock prices on one of its services. XMLHTTP Control appears a bit later in IE5 . Thank you very much Microsoft, they beat us the AJAX track. Again, the increasing popularity of Flash . Flash brings some simplicity and richness to display on the client. Well, the turning point is the emergence of cascading style sheets, which puts some order in how new websites are designed.
The mid-2000s is a recent story, the emergence of WebKit , thanks to Apple. They go for HTML5 . WebKit brings us a Canvas element, which is basically the beginning of HTML5. This is all AJAX with all its buns and usefulness. The appearance, finally, of Mozilla Firefox , which was born on the Gecko engine , and brought us the currently popular plug-in technology in the browser, FireBug , which allowed us to significantly speed up web development. I know that many of you work not only with Firebug, there are various Developer Tools. JS Frameworks appear. Here I want to talk about JS Frameworks and java-scripts separately. Initially, how does a client programmer work? He gets the layout in the form of a PSD or JPG file, cuts it, wraps it with HTML binding, maybe tables, if you're lucky, there will be some divs, everything will be fine. First, we start JavaScript - these are some forms, this is the processing of forms, this is their validation. And gradually, at some point, JavaScript is so complicated, and it becomes clear that application on the client side is often much more than the server side.
I remember well that crucial moment when separate JavaScript developers appeared, separate vacancies of people who work only on the application side on the client side. Again, I repeat, often this application, it is much more complicated than the server side, which sometimes just stores some data on the server, gives it away. What do we have now? Now, in general, not everything is as good as we, as web developers, would like - there are more browsers, five browsers. Of course, it's great for the user, each user chooses the best browser. Now we are experiencing the battle of HTML 5 against Flash, in some of its form, I don’t know who will win, probably each of us supports someone there, so to speak.
Again, the flourishing of JS libraries: Prototype , jQuery . Take what you want and use. Probably, on every second site you will now find one of the most popular libraries. Here it is. And continuing the discussion about JavaScript developers, a rather interesting phenomenon appears: an attempt to work on JavaScript on the server side. A kind of technology appears, called NodeJS , for sure you have heard about it, it allows you to build servers, write in JavaScript. What does this give us? Cross-platform code, we can write in the same language both on the client and on the server side of the library, which are for the client, which part of them we can use on the server. Excellent object language with simple syntax, quite understandable. Sample JavaScript code taken from the NodeJS site, which gives some implementations of a seven-line web server.
The technology is quite interesting, promising, according to our tests for our performance, it does not pull, there are many problems in the engine, until it turns out on our loads. On a little more smaller ones, maybe he still lives, on ours - he does not. For these ten years, a rather interesting result: we started with the LAMP project and in ten years it turns out that the whole layer of technology changes.
We are now doing a new project, on it we do not use old technologies at all. We use an event engine there. We did not take MySQL, it is not there, everything is based on NoSQL-bases. According to our tests, according to our runs, we see that it lives and works, and build a service is much faster than even we expected.
MOBILE DEVELOPMENT
Let's remember about mobile clients, let's talk about mobile development. If you remember, it was a long time ago, there was a low bandwidth of the channels, there was a rather strange markup language.
Strange it was that you could never guess this set of phones, who supports what. Who can show pictures who can not. What type of pictures can show, what type - can not. Whether form submission is supported or not. Why in some minor version it supports, and in the next one it has ceased. With a weak phone capability, the browser was completely none, there were no processors, everything was very slow and sad. Over time, XHTML appears - it begins to be supported by embedded browsers. Quite a big step from Microsoft in Windows Mobile. This is Internet Explorer, which somehow parses the page, parses HTML, and offers at least some kind of JavaScript support. Nokia with its platform 60 . Opera has done a lot in this market. They got in and continue to get into all mobile devices. Probably everyone remembers and knows the service that still renders large sites on the server side for small clients.
In my opinion, at some point a certain change occurs, thanks to the well-known company Apple, which brings us an iPhone with its Safari , and the idea is not even that there is a full-fledged browser that perfectly renders, perfectly supported, compared to its predecessor JavaScript, they bring quite an interesting idea. They bring an excellent usability solution for scaling pages and you now have no problems with how to zaskeit, rewind this big page, how to show it small, then big, then small again.
All the closest competitors and followers immediately pick it up, Android with its WebView , with access from the API, with an excellent JavaScipt engine and with Flash support.
What do you want to say separately about Android? Thanks to them, the support from the API is excellent, you can write browsers, sit down and write right now, based on what is already out of the box. This allows you to make interesting decisions, it allows you to write a web-based application, when you have a native proposal is just some kind of snap around WebView, and you already control the contents of this WebView from the server. That is, you do not need to write some kind of render. Again, the iPad with its next mobile browser, Samsung with its Bada , and Windows Phone with Internet Explorer, which is based on IE.
What do I want to say? The fact that they are all on their own is, of course, cool, but they bring one very big problem - there are a lot of them and we need to support them all and not forget. And plus the development of mobile devices, it gives such an interesting effect - there are services, popular services, which in principle are not designed for large devices. I will not enumerate, but there are those that you probably know, which are interesting and convenient to use, in principle, and they are only popular on mobile devices.
CONCLUSION
In general, I have bad news for you. In just ten years, we have completely changed all the technologies. If you now sit down to write a web service, you write it, you use completely different technologies. This means that you need to continue to learn, you need to continue to go to conferences, read books, read forums, listen to reports, because something new constantly appears. High-performance services, they are quite demanding on the choice, and the wealth of technologies that appear, often do not solve the problem, but create it. You have too many choices and it’s easy to make mistakes; it’s easy to choose the wrong thing, not the one that really solves your problem. The minimum set of today's typical web service, it has grown. If earlier, we speak on the example of a web server, if you used to take Apache , and you have it all there: images, scripts and everything, now you have a separate Apache, again, if it is Apache , a separate Apache, which will serve your applications, which will write in a scripting language, a separate web server that will give statics, you will definitely put the same NGINX , which will be in front of Apache, which will reduce the load. The bricks from which we build the service are becoming more and more, and they are becoming more and more specialized.
A good example is memcached , an excellent cacher, even though it is entirely based on text within itself, on the fact that everything is stored in a textual representation. Excellent caching, no longer need to use it. We took memcached and then forget about any other of its applications.
Again, the list of devices, permissions, browsers, engines has drastically expanded, rendering in each of them works differently. As for the resolution, there’s an interesting story in general, if you remember, the old web starts at 640 * 480, then it’s 800 * 600 for quite a long time, 1024, after 1024 - a zoo of permissions. And the most interesting thing is that now we are gradually returning to 640 * 480 and to 800 * 600, thanks to our new phones, which perfectly render the pages. Therefore, we return to the previous permissions and understandings about how to show the page in one form or another. The main problem, keep in mind that if you plan your service as successful for several years, then on average, once in 3 years you will completely rewrite it. By the example of the same mail, I can say that we rewrote the client, essentially, completely rewrote three times. And this is not due to the design, it is due to the emergence of new technologies.
New technologies give new speed requirements, and the user expands the channels, your slow service can no longer be justified by the fact that the user has a modem. It does not matter that you collect a page on the server in one second, it will download it five. Now half a second is already sensitive, for half a second be kind enough, but give the page. Accordingly, you constantly need to participate in this race, constantly accelerate and accelerate. Plus, new technologies - they are very good feedback. New technologies, the same AJAX, for example, offers new IT solutions. But in order for you to implement this AJAX, which essentially just changes the client’s website, you don’t have a page being overloaded, but it would seem that a bad page isn’t being overloaded, it’s very cool, there’s a minimum of data chasing through the network. But in order to write normally, you need normal AJAX support on the server side. This means that you are going and actually completely refactor the server-side code. In general, get ready that you will rewrite your services, somewhere on average, once every three years, if this speed does not increase.
Thank you very much, let us have your questions.
QUESTIONS FROM THE HALL
You talked about how you developed. I would like to know if you were waiting for some technologies, did you have any preparations for them, for example, that a touchscreen would suddenly appear? That is, it is clear that it was invented back in some distant year, but it was not on the devices, and you waited for it to appear, in order to use it for something, or you were drifting: “oh, cool! there was a touch-screen ", or" oh, even cooler! there was an accelerometer, and it can somehow be used. That is, were there any plans for waiting for technologies in your plans, in which something was already invented, or was it all spontaneous, did any new solutions "move" more manufacturers?
Ok, I'll try to answer, I understand your question. Not that we sit and come up with a cool thing, but in order to give a cosmic example, to count cancer cells, but it would be great if you had a cool battery on your service, then we would all together consider cancer cells , fully load the CPU, but let's wait until you have a small nuclear reactor on your phone for it to be all your life. We do not do that. We work with manufacturers, with manufacturers of hardware, manufacturers of software, with large manufacturers, the same Samsung, the same Nokia. They share with us some of their best practices, we discuss how the next versions of our products will work on them, so we know them a little more.
Good afternoon, my name is Ilya Pyatin, Line Media company. We have now heard, in fact, a certain history of the development of Internet technologies, but the report is called “The Evolution of Development”, right? And the development, because it consists not only of technology? These are developers, some teams you have, yes, deploy, version control, and so on. Can you hear how it all evolved? Because, in many cases, it is even more important than the technologies that are used, somehow nothing was said about it. And which of the technologies mentioned do you still prefer, at least not for the entire period of history, and now ... Did you use everything that was listed or did you give any priorities? Thank.
See about the technologies we prefer. That, apparently, poorly expressed his thought. We love the tools in our application, therefore ... And since we have a lot of services, you cannot use the same technology to store mail, to show a social network and to provide VoIP communication between your two messenger subscribers. Therefore, there is no technology alone. We have evolved for a long time and finally grew and now we have no use everywhere, we are evolving together with colleagues from other companies, we look at all the developments that are on the market, we are trying to use them somewhere, somewhere else we try, we see that there are no chances for us and they do not apply to us. And there is no technology. If we talk about deploy, monitoring and operational things, this is a very long discussion. I didn’t want to talk about it at all, because this is a separate discussion about how we evolved in monitoring ours, how we evolved in our deploy, how we apply, where we don’t apply, what rules we use there, calculations, rules of commits and patches. This is a very long discussion, we can talk with you separately.
Igor, hello. My name is Alexey Shishkin. I have the following question for you. In your opinion, what is missing in augmented reality in order for it to fully enter the market. At one time, the iPhone came out with Safari, it became possible to watch websites from mobile devices. What is missing for augmented reality in your opinion? Thank?
Can you tell me a little more about this technology?
Augmented reality is the display on the screens, say phones, of objects that are not present in the real world, but are generated by the data received at the client’s location in the application.
You know, I understand what you mean. I think we need to wait a bit. Iron is not ready yet.
Allow me to ask a question? If I were asked to evaluate your report, in terms of a5- point system, I would not evaluate it, but I wrote it brilliantly!
Thank.
That's why. Because you need to know the story. And what you said about the interview, it even morally supported me a little, since I sometimes took exams from students and approached the answer with the same approach. When you need to argue some options. But, really, the terminology is changing all the time, sometimes you do not have time to learn everything. But back to the databases. Around of which everything is spinning in the main. The fact is that Konstantin also spoke in front of you, saying that the implementation databases will still have their niche, no one cancels them, but there is also a new SQL. Now I would ask you to decide on the definitions, perhaps you can already change them. Because, as far as you remember, relational databases, the relational data model won in due time, somehow prevailed, evolved before the hierarchical network. But there are two definitions that appear: simple, when it was meant that the evolutionary database interacts with flat tables, and more mathematical, which is relational from the word relation relation. This is because we are modeling real-world objects that interact with each other. Firstly, about this relation and the second SQL language also has its rich history of development and when ... It is known that the SQL language consists of several sublanguages that define the main work with the database. Here you are, please.
As for determining the applicability and so on, well, I’m probably going to repeat every tool is good where it is. Thank you, you listened very carefully to my report, in which I said that we started with the C language, we continue to write on it and we will write. , , , , , . , . , . 13 , , , . , .
. : - ? , . ?
, . - , . . , , : , . , , , . . ?
?
, , , , , .
Hello, Igor. . «-». , , , WinBse-, noSQL, , ? ?
, Python
-?
Not
, .
, ,10- . , -, mail.ru Perl-.
. ....
, mail.ru, -Perl.
, . - Perl. : « - , Perl». , , , Perl, , . , .
, . . , , : « ( ), ?»
. , ...
, , . - ?
. , , Tarantool. Tarantool . , Tarantool. . memcached, memcachedb, Tarantool, memcached, memcachedb , , , , . Redis, , . : , , … Tarantool , Tarantool!, , Tarantool.
Tarantool, , , , , , Tarantool , . . : , , , , . MySQL. Let's. - ? not. , MySQL . . , . , , . , , , , , , , , , , , … , . , 9 . MySQL - - . . - , «» , , . . , , .
Hello, Igor. . . , , . , ? WML, - .
. , WML- , . . WML-, , WML, XHTML .
! Good day. Question. , , , . , , - . , , . , , , - . , . , ? , , 20%. , - , . . - ?
. , , .
, , , ? - ?
, «-», , Tarantool — open-source , , - , «» , . , open-source , , , .
, , - ?
?
, , , , .
, , .
, «». . ?
, ? ?
? NodeJS, . , -, «» NGINX, «». , API, , .
, - , « », «API », « NGINX», , , , API , , , Nginx, - , … ,
Mail.Ru , NodeJS ? ?
- , -, . , , . ?
, PHP, enterprise-, , , .
, ? , - , - , , , , API, - . deprecated . PHP.
JS . , , , -.
, . , - , . , , ! , , , , , . , - .
, NodeJS, Java 1.0. 1.0 , Java 1.0
, .
, . .
, , , . , . «» ? : «-, NodeJS , ». , ? , , 20% , «» - , «» ?
-, , . . , , , «» , , . core dump, «» , , .
, .
, . . - , . - , - , - , - .
, .
, , .
Thank.
OK, thank you. Just a second. A person who wants to expand consciousness. who wants to expand reality, so here I am with him. Also really looking forward to come here, yes. Honestly, I’m also really looking forward to when our world will begin to change even more efficiently with our hands, thank you.
Igor, thank you very much.
This text is a transcript of the report of Igor Yermakov at the Mail.Ru 2011 Technology Forum , held on November 16 in the center of Infospace. Details about the technology of creating texts of reports based on video recordings can be found here: “Wrong side” of the Mail.Ru Technologies Forum: High-tech in event-management . Video versions of other reports (including versions for mobile devices) are available on the Forum website - techforum.mail.ru . Text versions of the reports will be published here and on the Forum website every week or less often in a similar format. Please report in the "lichku" about typos in the text.
Source: https://habr.com/ru/post/133193/
All Articles