Do I need to experiment in the development process?

Each successful software company project begins with an idea, but before it becomes a real product, you have to go through a lot of trial and error. Today we want to talk about some of the projects Virtuozzo, which could (or could not) become popular in the market products

Software companies constantly need to make decisions about the direction of development. We in Virtuozzo have such a unit - the program management department. It employs people who are responsible for the roadmap of existing products, the release of releases and updates, development planning. Thus, the activities of program managers can be divided into three large sectors:

• Development according to market needs. This is the creation of products for existing trends, the adaptation of releases for industry trends. For example, when the market formed a need to work with Docker containers, we began to develop a mechanism for their support at the level of our hypervisor.
  ')
  
• Development for the needs of current customers is the revision of releases, according to feedback received from users. It can be as obvious “Guys, help us solve the problem with the stack” or come in the form of periodic requests for technical support, which help to outline the obvious need for improvements.
  
  
• And the most ambiguous is the experimental direction. Very often, we do not know whether it will be possible to solve a specific task or not, and how much this solution will be in demand. Sometimes, if an employee has an idea, and the team regards it as a good idea, we try to make a prototype and make a product out of it.

Experiments and reality

The presence of such a direction in the company creates a chance for the emergence of spontaneous, often successful products and tools that not only succeed in the market, but also change the industry to a certain extent. By the way, this is how the project Virtuozzo Storage was born, which was decided to develop in the form of a distributed data storage system. In 2010, it was proposed to use server disks to integrate virtual machines into a distributed storage system, and today Virtozzo Storage is one of the three cornerstones of our ecosystem.

There are several more examples where a product that was on the market began with a prototype. For example, the completely redesigned architecture of the Power Panel product, which we released in March, also began with a similar project. But not always experiments lead to the appearance of the product. However, a negative result is also a result. Indeed, without it, it would be impossible to find more promising projects. In order to check this or that opportunity, we select 1-2 developers - those people who are interested in this area - and we conduct research for several weeks or months. Further, it is possible either the appearance of a separate product, or "chips" in one of the existing solutions, or the closure of a direction as unpromising.


Unsuccessful samples

At the turn of 2006-2007, our developers had the idea to create a virtualization solution for MAC servers, similar to the product PSBM (parallels server bare metal). Interestingly, the product came out of the research-state, went through the development phase and even got to testing. But then it turned out two unpleasant nuances. Firstly, Apple has always been famous for the closeness of the part of the critical elements of systems, such as the BIOS, which in turn led to problems on certain variants of hardware configurations. Fully solve these problems could not succeed. In addition, it became known, first at the level of rumors that Apple will not develop a server line, and we decided to close the project. Well, it turned out that the decision was the right one - and in 2010 Apple really abandoned the servers, and the product we developed was never announced.

The second project, which I would like to talk about, alas, did not even come out from the research stage, although initially it seemed very attractive. We worked on creating a prototype of the so-called Meta-PC - a tool that would allow creating a runtime environment for simultaneous work on several servers, using their total capacities for a single application. If successfully implemented, such a product would provide an opportunity to vertically scale applications, as well as to provide one process with more CPU and RAM than a single computer or server can provide.

The project was closed, and the reason for this was market trends. Having studied the situation, we realized that such problems are solved differently today - when the power of individual machines is not enough for an application, the developers begin to change the approaches to the design of the application itself, use distributed computing technologies and methods of horizontal scaling. As a result, technologies such as MapReduce and container management environments have begun to grow rapidly. After several months of research, it became obvious that such a solution with the possibility of vertical scaling would have a very small market, and therefore the project was simply closed.

There are examples of projects that are closed and at later stages, after many successful years on the market. For example, the solution for working with OS containers in the Microsoft Windows ecosystem has been part of our product line for about 10 years! The system was developed, tested and launched, it had its users, the project was developed. But the further - the more, working with the closed Windows kernel was becoming more difficult, and more complex customer requests turned out to be simply impossible to implement. Yes, Windows has a lot of APIs, but they are not intended for such tasks. And attempts to use the elements of the kernel to launch the OS containers began to resemble attempts to find something in the black box, when all you can do is put your hand through the hole and blind through the contents. Therefore, in 2015, the project was decided to close, focusing on containerization in Linux, where the kernel source code is available and therefore can be easily used from operating system containers.

Conclusion

However, finally, I would like to give one more very illustrative example - this is a system of live application migration. In 2005, our developers decided to try to do an interesting thing: freezing a set of processes, saving them to a file, and thawing them in any other conditions and on a different hardware is about the same thing as live migration of virtual machines, but in essence “cutting out” these processes from the operating system kernel.

Even then, many said that such a scheme could not be implemented, but we did it! When it worked, we tried to offer tools for “live migration” to include in the Linux kernel. But practice has shown that the community does not always accept all the proposed innovations, and making them into each subsequent version of the kernel itself is very costly. And then it was decided to develop a backup system in user space. This is how the CRIU project (www.criu.org) turned out, which not only became a leading tool in the field of “live migration”, but was also able to make its contribution to the Linux kernel so that the toolkit worked in all versions of Linux. So experimenting is useful, even if you do not know whether there will be a return on the project.