Interview with the creator of NGINX Igor Sysoev

- Igor, tell us how your education was structured, how did you come to programming and in general got carried away with computers?
I was born in Kazakhstan in a small town. When I was about a year old, my father (he was a military man) was transferred to Alma-Ata, and I lived there until the age of 18. In 1987 I finished school and went to enter Bauman Moscow Higher Technical School, but I could not get in the first time, and I went back to Almaty, where I got a job as a laboratory assistant at the branch of the Institute for Advanced Studies of the USSR Ministry of Geology. There were old Iskra-226 computers, and I started to program something in BASIC. And at that time a series of articles was published in Radio magazine, how to assemble your own Radio-86RK computer, and thanks to their reading, I had a fairly good idea of ​​how a computer works and how it works. And the first experience with computers was a little earlier: in high school I went to the Palace of Pioneers, and there they installed Yamaha KUVT computers (MSX standard). I remember when I typed the first program, I confused the unit with the letter I. In general, it did not work for me because of such things.

- Do you remember your first program, which was used by other people?
My first major and alienable program is the AV antivirus, which I wrote in 1989-1990. It was written entirely in assembler, the amount of assembly code was somewhere around 100 KB. The program was able to find several viruses, having a database wired inside the program with several signatures of then-known viruses in the USSR, of which there were about ten strengths: the Marijuana, Sophia, Vienna viruses and several others, I don’t remember their names. So this was my first program that I distributed in binaries - I did not distribute the source code then. As a result, it spread across the country, it was installed even in several factories. There was also a feedback: people sent letters by mail with viruses written to floppy disks. For a while I supported this antivirus, but as a result, in 1992, I lost interest in this subject, and the program died.
In 1994 I graduated from the institute, and the year before I started working as a system administrator in one company that was associated with the sale of petroleum products. I worked there for almost 7 years, after which in April 2000 I decided to leave. Then the NASDAQ was blown away, the “dot-com bubble” burst, and just at that moment I decided to go online. For six months I worked in the online magazine XXL.RU, after which, as I remember, on November 13, 2000, I came to work at Rambler.

- What did you do at the Rambler?
I worked as a system administrator. However, apart from the direct work of the sysadmin, I again began writing programs in my free time. It should be noted that programming was not part of my job duties, but since there was time and thrust, the first thing I did was adapt the patch to compress the Apache responses. Unfortunately, at that time the mod_gzip name was already taken, so I called my version mod_deflate, it worked with Apache 1.3.
Then I was asked to deal with the mod_proxy module. I looked at it and decided that it was easier to write everything from scratch than to adapt some things there. Thus, the mod_accel module appeared - a module and a set of patches for Apache for reverse proxying. All this was also in the spring of 2001.
- That is, you made all these modules for the Rambler, at the same time putting it in public?
- Basically, yes. Mod_deflate actually came from the patch that Dmitry Khrustalev wrote while working for RBC. That is, this patch was taken as the basis, there my code, maybe half only.
In the fall of 2001, I had the idea to write a lighter and more productive web server than Apache. At that time there were already other similar servers, but they all could not proxy, they gave only static. They had one more common flaw - they worked in one process, and, accordingly, it was impossible to scale them, say, on a two-processor machine.
At that time, I already had quite a good experience working with Apache - both as a system administrator and as a programmer. Two modules I wrote added knowledge: I had to look at the source code for Apache and understand how it all works. Therefore, many things in nginx migrated from Apache ideologically. Not code, but ideology, all nginx code was written from scratch.
However, I did not like everything in Apache: for example, there it is very easy to make such a configuration that it will be extremely difficult to maintain. That is, the site grows, some new functionality is added, and in the end it becomes impossible to work with the site. It is necessary to add something, and you sit, you think: “And what will break with what I add?” In nginx I tried to avoid these things. In general, somewhere in the spring of 2002 I began to develop nginx.
')
- How quickly did your people find out about your developments that were external to the Rambler? How did the project evolve?
In 2003, my development was heard outside the Rambler, and, moreover, nginx began to be used on several sites. The first was the Estonian dating site Rate.ee, which still exists. This is, by the way, the most heavily loaded site in Estonia. Then nginx began to be used on mamba.ru and on zvuki.ru, where I distributed MP3s.
In early 2004, Rambler started the foto.rambler.ru service, and one of his colleagues, Oleg Bunin, asked me to finish the query proxying functionality in nginx in order to start using it fully, including at the Rambler photo service. Up to this point, the project was quite academic, I gradually wrote it, but it could never end in anything, that is, it would probably not have been put into production anywhere. In general, it turned out that I urgently completed the proxying. And somewhere in the beginning of 2004 a version with proxying appeared, and the foto.rambler.ru service started working on the basis of nginx.
On October 4, 2004, on the anniversary of the launch of the first space satellite, I released the first public version: 0.1.0.

- Now the share of nginx is growing very fast, and how was it at the very beginning?
Now he is really growing fast enough. At first, everything was noticeably more modest. In the first year, for obvious reasons, nginx gained the most popularity in Russia. In the future, about nginx learned outside of it, and some enthusiasts began to use it at their own peril and risk. An English-language mailing list appeared, third-party resources began to appear describing nginx, people sent me more and more wishes and comments, I made corrections, the product gradually gained popularity. Now the project is really growing very fast, and this was one of the reasons for creating a company. Alone, I just stopped working.

- So, there was no promotion at all, it turns out, did the product make itself?
For my part, there was no special PR. Although there is an opinion that the best PR is just a good product? .. That is, the entire height was due to the fact that nginx “just worked”, and people told their admins about their positive experience, and those to their acquaintances, and so on the principle of word-of-mouth radio. The popularity of nginx, in my opinion, is associated with several things. First, it is an effective and free software that allows you to significantly save hardware resources and money, and secondly, in principle, it works well.

- But there are analogues, the same lighttpd, for example.
In fact, there are a couple of reasons: it turned out to be a rather interesting combination of vital features for creating effective web infrastructure, which I added gradually and which made nginx such an indispensable tool. At the same time, nginx is not overloaded with unnecessary features and remains a very compact development. In addition, the modularity of nginx allowed many companies and third-party developers to build their extensions based on the nginx kernel. We can say that nginx has long been a kind of web platform.
About lighttpd (lighty). Once it was more common than nginx, and more famous in the world. Its author is German Jan Kneschke. The difference in popularity was due to the fact that Russia is an incomprehensible country with a balalaika and a bear, snow, and here is Europe. Again, it was better for him with English, including with English documentation? .. By the way, thanks to lighttpd, the FastCGI protocol gained a second wind. Until 2000–2001, it was exotic, everyone used interpreters that were inside Apache: PHP, Perl, Python. And since in lighttpd, executing the PHP code inside the process is unrealistic, the solution was FastCGI. And thanks to the lighttpd FastCGI has found a second life. Although back in 2000, people said: “Why, what is it - FastCGI? We have mod_php, and everything works fine there. ”


- What are the main cases of using nginx you see now?
The main use on busy sites is proxying. At the same time, nginx is installed as a front-end and proxies applications on backends via HTTP or via FastCGI or WSGI. In this case, the standard approach is to use in conjunction with Apache - for example, at my previous place of work, nginx worked for a long time just that way, only a couple of years ago switched to using FastCGI. By the way, in this case, the statistics shows that nginx appears, Apache disappears. Although both are actually used: just nginx is one of the components of the proxy system, visible from the outside.

- Explain clearly, why proxy requests at all?
Why do people actually use Apache with nginx? It would seem, why is there an extra link that will interfere. Apache is good and easy to use where you need to run an application, for example, using mod_php. Now imagine that this PHP is capable of generating 100 responses per second, and each answer has a size of, conditionally, 100 KB. Not all clients use fast connections: 10 years ago there were modem clients, the mobile Internet is now very common, someone just has a bad provider or a slow tariff. And here we have the answer with a volume of 100 KB and the effective speed to the client, for example, 80 Kb / s (10 KB / s). This means that this response will be transmitted to the client for 10 seconds. As a result, all this time, while the client slowly downloads the answer, Apache, along with PHP, “eats” 10–20 MB of memory per client. And instead of doing what Apache can do quickly, he waits for slow clients to download the answers. All this consumes a lot of memory, and the processor too. When we put nginx between clients and Apache, everything starts to work more efficiently: nginx takes all the answer as quickly as possible, freeing Apache, and then slowly gives it to clients, without spending a lot of memory. Nginx does not consume a lot of memory or processor, because a different web server architecture is used - non-blocking, based on asynchronous event processing, which allows you to process many thousands of connections within one process (unlike Apache, where each connection is processed by a separate process or thread. - Ed.).
Well, plus to this, we can take out all static files from the backend, this is a simple thing that nginx can handle very easily and as efficiently as possible - tens of thousands of files per second can be given to nginx at the same time if memory allows, and if it allows network connection to the server.

- Let's go back to sample scenarios.
So, the first scenario is when we just accelerate, maybe even a single site. We had Apache, we set nginx in front of it and - bang! - a miracle happened. People really put and are surprised, and then write on "Habr" that "it is necessary, as cool". The second option is proxying too, but we have a lot of back-end, that is, we can effectively scale the entire system horizontally, provided that the application itself allows it. Thus, nginx acts as a load balancer. One of the drawbacks of the current implementation is the absence of several balancing policies, but people use it, it works, and we will add functionality. What else? Another scenario, for example, is this: for some reason, many people don’t like Apache. They want only nginx on the server, they don’t want to install Apache. In this case, all the scripts they work through FastCGI for PHP or WSGI for Python.
For example, WordPress.com - they started using nginx as a balancer a long time ago, and a commercial LiteSpeed ​​acted as a web server. This year they have already completely migrated to nginx, now they have PHP running in FastCGI mode.
Another standard use case is when nginx just gives all the static, say, MP3, FLV, MPEG4 video, pictures.

- Let's talk a little about security. Have there been any serious vulnerabilities during the existence of nginx?
Vulnerabilities were different, but there were no problems with using them to get remote access, code to execute. It was possible to drop workflows, but just to execute the code — there were no such vulnerabilities. See, usually the exploit is designed for what? We wrote something down to the server, this case fell on his stack. The server is working, making a return and falling on this code. Accordingly, in order for the exploit to work, you need to know where the stack will be at this process. As a rule, when there is any Debian / Ubuntu package, there is a binary, you can reproduce an emergency, try to find out where this stack is and thus make an exploit. How did you begin to deal with this? They began to randomize the address space - in modern Windows, for example, it works this way.

- ASLR?
Yeah, right. This is randomization. We had a stack here, and now it is here. And, accordingly, we can not predict, that is, we took the package, but it’s impossible to understand where he now has a stack. In this respect, nginx is simpler, because there is practically no data on the stack that is read from clients. One can count on fingers several cases where it is used, but in these places the code is quite reliable. The data received from clients, nginx places "on the heap", allocating memory malloc'om.
Accordingly, if there is to write somewhere a little more, then we will not get on the stack pointer. This randomization in nginx was present from the very beginning. In general, it is very difficult to write a working exploit if possible. In addition, the processes that are engaged in processing requests do not work from the "root".
Security advisory were, they can be viewed on the site. I believe that all these error messages need to respond adequately, calmly and professionally. For example, to hide the fact of the bug, when everything is already published, say like “What? There was nothing, everything is good ”- it simply undermines the credibility of the project.


- How many people worked before and are now engaged in the development, development of the project?
For a long time I was engaged in one, almost all the code I wrote alone. About four years ago Maxim Dunin began to help me more and more. In addition to the two of us, as the product developed, people sent patches. And often they send just letters with a text description of problems or requests. They say to me: “There is a mistake, you can solve it like this.” Just words. We do it as much as we can.
We also have a separate person now - Ruslan Yermilov, who is now engaged in documentation. It performs several tasks: it is the translation of the current Russian documentation into English, the updating of information and the adaptation of documentation, so that it is understandable and unambiguous for people reading it for the first time. A frequent problem when the author writes documentation, he has a certain context in his head, and he repels himself from it. He thinks that it goes without saying, but in the end he misses a lot of details. With this we are actively fighting: Ruslan looks at nginx "from the side", with a fresh look, therefore he is able to write so that everything is clear to everyone. In addition, Ruslan has extensive experience in developing and documenting complex software projects.

- I propose to go to questions related to the company Nginx, Inc. and with how you came to create a business.
I'll tell you everything now. So, probably, in 2008, the first letter from an investor came to me, I don’t even remember who it was. In general, over the past two years there have been about a dozen such letters. People wanted to do something with nginx, make a company. But I refused, because I am not a businessman in general. But in the end, I began to understand that something needs to be done, otherwise I just can’t continue to develop the project alone, I don’t have enough strength for everything. It took quite a lot of time to figure out how and with whom I would like to make the company “around” nginx. In general, I very rarely change the direction of life: for example, until Rambler, I worked for seven years in the same company, I also worked in Rambler for ten years. Changes are hard for me. But, nevertheless, by the spring of this year I finally decided to establish a company that would help the further development of the project. Partly, this step was inspired by Sergey Belousov, the creator of Parallels and the fund Runa Capital. We talked to him informally several times, and as a result I gradually became much closer to the idea of ​​creating a company.

- Sergey is able to convince, yes?
Sergey is a very interesting person in general, it is always exciting to discuss matters with him and not only, he is a very energetic person. Sergey is also a rather domineering manager - I think he influences so many decisions in his companies, this is an owner who likes to control what is happening, to directly participate in the business.
In general, the process of negotiating with investors, signing the terms of the transaction, a lot of everything is a difficult thing, because, firstly, there are a lot of boring details, a huge amount of paper in English, legal, it’s hard to read in Russian, but -English - even more so. Negotiating everything, again, reconciling all things: we want this, that, they want that. Psychologically it is hard. But then, if investors understand your business, everything becomes much easier.

- Interesting: you worked in the Rambler and worked on nginx. Did the Rambler have any rights? This is such a subtle question. How did you manage to keep the rights to the project?
Yes, this is a rather subtle question. Of course, he is not only interested in you, and we have worked it quite thoroughly. In Russia, the law is designed so that the company owns what has been done as part of employment duties or under a separate contract. That is, there must be an agreement with the person where it would be said: you need to develop a software product. In the Rambler I worked as a system administrator, I was engaged in development in my spare time, the product was from the very beginning being released under the BSD license, like open source software. In the Rambler, nginx began to be used already when the main functionality was ready. Moreover, even the first application of nginx was not in the Rambler, but on the sites Rate.ee and zvuki.ru.

- Who else works for you in the Nginx company?
We also have Sergey Budnevich, a system administrator, who is involved in supporting the company's infrastructure. Our infrastructure is not very big, but it is there. We have mailing lists, we have a mail server, automatic assembly, testing of pekedgej, error tracking, etc. Sergey helps us with this very much. We are now going to prepare packages for several more Linux distributions: CentOS, Ubuntu. Sergey is busy automating a variety of processes related to the development, testing and maintenance. There are two more people: one person is engaged in marketing - Andrey Alekseev, and Maxim Konovalov is the chief of everything, he makes the company work.

- What is the official title of your position in the company?
Formally, I am the technical director. I do not know how to lead people, I focus more on the architecture of future products and the transfer of development "to the team." It’s pretty hard to delegate work, but the company was created just to improve development and product, so now I'm trying to teach myself this. Colleagues deal with organizational issues, communication with customers, marketing, relationships with partners, documentation, hiring staff, etc. We have many different difficulties, learning to communicate at different levels - this is not so easy. In fact, we all participate in all the affairs of the company, because the company is not so big, but there are many things to do.

- It was difficult to delegate, because it seemed that everyone is doing poorly, easier for yourself?
Well, yes, the approach is such that I would rather do it myself, because it will be better, or because it takes a long time to explain what needs to be done, or it is psychologically hard to say: “Do this.” Personally, it was hard for me to delegate authority for a number of reasons. Now, as a technical director, I am mainly responsible for the architecture and quality of development.

- Igor, thank you very much for the interview! It is evident that you still learned to delegate: with all our business questions you sent us to Maxim Konovalov.
By the way, this is the first interview I give. He agreed only because he created a company. Literally in the spring I was asked by people from another IT publication, I said to them: "Sorry, I do not like, do not want and can not."

- Thanks again! Maxim, did you present any formalized business plan in negotiations with investors? What do you plan to make money on?
Funds mainly invested in nginx as a very promising product. A detailed business plan, of course, was important, but American investors are approaching the issue of investment, based not only and not so much on the business plan, where it will be written that we will earn so much in a year with an accuracy of tens of cents. It was important that nginx is now very popular, it is a ready, existing product.
About the fact that we have for ideas for making money: we want, first of all, to achieve the right balance between free and paid functionality. We want to do something that a number of companies have not quite succeeded in the past. There are several examples of business based on open source development, where companies could not keep the right balance, had to close some features in the product itself, ask for some ridiculous money for them, it upset everyone, and the products stopped developing.

- So you want to make a separate commercial product and find a balance between an open product and a commercial one?
We do not want to make a separate commercial product, we want to make commercial add-ons over the main open source product. It will develop, there will be features that are required by the community. The money we received will help us to put the entire production of the product to a new level. Now Igor is no longer working alone on the code, a team development is under construction. We recruit people in Russia, the engineering team in Moscow will remain.
Accordingly, the focus on the open source product is very strong and will remain so.
At the same time, we know that there are customers, large companies, medium-sized companies, even small companies that have been using nginx for a long time. They built a business on this and are grateful to us. When we meet, we hear something like: “Excellent, great product - thank you so much! But we lack this and that. Can you do it - are we ready to pay you? ”From such conversations, we gradually develop a chain of what we could sell without grieving the supporters of the free open source product and not undermining the credibility of the project as a whole. That is, we collect similar requests and compare them with the wishes that come from the user community. We are looking at where there are intersections, and if we understand that some functionality is actually necessary for everyone, not just a separate company,we implement it in the free version of the product.
There are even companies that say: “Let us pay you for all these features so that they appear faster in the product. We want everything to be in open source, we don’t want the feature to be exclusive and / or paid. ” This is called sponsored development.

So far, we have developed ideas that commercial add-ons will be more relevant to large examples of nginx applications: for example, using commercial add-ons it will be easier to manage thousands of instances, there will be advanced performance monitoring, additional functionality designed for hosting, cloud and CDN infrastructures.

- So you have a focus on the product. Will you not sell services separately, for example, for implementation, consulting?
The fact is that the company is small, it will remain small - we do not want to grow to a company of several hundred people. We actively work with partners, with system integrators, with vendors of software and hardware, we are actively looking for channels to work through partners. Consultations will be carried out partly through partners, partly through us. Unfortunately, we ourselves, directly, will not be able to provide consulting and technical support services to all users.

- And what awaits ordinary users in the near future, are you planning any new features?
From the history of changes in the code over the past three months, from the list of releases that we released, it is clear that since the inception of the company we have seriously intensified the process of developing and making corrections. We have integrated quite a lot of improvements and new options. For example, they added streaming MP4, about which Igor was asked for several years. The work goes on, the functional develops.

- I understand correctly that Igor Sysoev is the main shareholder of the company, and the rest, a smaller share belongs to investors?
Yes, Igor is the main shareholder, in total there are three founders of the company, and there are, naturally, investors as owners, a group of investors - they jointly own some part. By the way, the process of receiving money from investors is technically very simple - securities are issued under the relevant legislation, investors buy them for some amount. The amount goes to you, you use it for the development of the company. That's how everything is arranged with us.



Hacker Magazine, December (12) 155 .

Subscribe to "Hacker"

• 1 999 . for 12 numbers of paper version
• 1249r. for an annual subscription to iOS / iPad (Android'a release soon!)
• "Hacker" on Android