Ingram Micro and Odin Automation

Hi Habr! My name is Timur Khakimyanov, I am the vice-president of Odin Automation and I lead the development of a platform for the distribution of cloud services and applications. Information about our company in RuNet is extremely scarce, despite the fact that our products are used by more than 150 telecom operators, hosters and providers around the world, some of which are very large and well-known. At the same time, almost all of our development department is located in Russia. Therefore, I want to fill the informational vacuum a bit and tell you more about who we are and what we do.

Our product is a platform that automates the sale of cloud services and applications. We sell the tool on which people build their business. It is necessary to clarify that this is not a boxed product, it cannot simply be bought and delivered. To do this, we need to work with us to determine the platform configuration depending on the existing infrastructure, work out the final architecture, customize the platform (Custom Software Development) in accordance with it, then properly configure and integrate it with various back-office systems. And one of my responsibilities is to oversee the development of key components of the platform. I am also responsible for releases, that is, I build the process of the work of our entire team so that with a certain regularity (now it is 3-4 times a year) we have product assemblies that we can give to customers. Since we became part of Ingram Micro, our main customer has been ourselves. We have a very hard and fast rule: when we release a release, we are the ones who are updated to it first. Previously, we rolled out the release very carefully, that is, first gave small customers, then more customers, and the largest customers were updated in about a year. Now it's the other way around. After the release of the new release, one of our biggest installations - Ingram Micro - switches to it within a month. To understand what tasks and problems we face, we must first tell about the history of our product, which is made up of smaller product histories.

A long time ago

In 1999, the company Plesk appeared in Novosibirsk, which released the eponymous product written in PHP - the hosting control panel, which allowed to automate some internal processes. Immediately after creating the panel, Novosibirsk decided to write a new product, Plesk 2, but in C ++ and Java. It was the most fashionable technology in the late nineties. But while they were writing this second version, in 2003 Plesk was purchased by SWsoft. Around the same time, SWsoft purchased a billing system, as well as the German company Confixx, which created software for automating single web servers. After that, all of the above products began to integrate. The result is a set of tools necessary for service providers to manage services: the Plesk 2 web hosting platform, which later became Parallels Operations Automation, and the Parallels Business Automation billing system. The target audience was still hosters, we automated website and mail hosting, as well as virtual machines based on Virtuozzo container virtualization. We dealt with containers long before it became fashionable.

The magical world of telecoms

')
In the late 2000s, we discovered the wide world of telecom operators. The potential of the market for cloud services and SaaS services was recognized by the management of the company long before it became a trend, and we began to redo our business model. From an on-premise platform — when a provider buys a data center or rack in a data center, and puts its servers there — our product has become a platform that has automated the sale of cloud services provided by someone else. A key component of the platform is the APS standard , which describes how any external service can be integrated into our platform. An interesting history of the emergence of this standard. When we provided website hosting service, WordPress and Joomla were the most popular content management systems, and in principle, little has changed since then. And APS was originally made for the packaging of these web applications so that when selling hosting you could sell the WordPress installation with one button. But when the insight came about cloud services, we began to redo our product into a platform, and APS into a standard for integrating any services. Any - Office 365, Azure, or any anti-virus.

What changes were in the telecom market in the second half of the 2000s? Their services, firstly, began to cost much less, and secondly, the billing of users greatly simplified. If you remember, in the late 1990s — early 2000s there was a popular question: “From what second do you start charging, from the first, from the fifth?”. The first minute of the call could cost some money, the next could be cheaper. As a result, telecoms needed a lot of sophisticated software, specialists to develop and operate it, so that subscribers could be billed correctly. In recent years, all wrong. Now I pay 400 rubles a month and get, in fact, no limit. The complexity of tariffing has decreased, corporate IT resources have been freed up, which need to be used somewhere. In addition, the flow of money generated by voice and SMS began to dry up dramatically due to the emergence of communication programs such as Skype and WhatsApp. Telecoms have felt the imminent threat of becoming a data pipe, a pipe for the Internet and zero margin. Therefore, it has become important for them to discover some new areas for themselves, to diversify their business, which they are doing now. For example, VimpelCom, the owner of the Beeline brand, recently changed its name to Beon and began to position itself not as a telecom, but as a company specializing in digital technologies. We understood this trend many years ago, like some telecoms. And as a result, we began to reorient the product to a new customer category. We have released the second version of the APS standard (and then also versions 2.1 and 2.2), expanding the possibilities of integration. Traditional services (website hosting, mail, virtual machines) were replaced by a variety of cloud services integrated into our platform.

What are the difficulties? Hosters and telecoms are fundamentally different. At the telecom, everything should be repeatedly duplicated, highly available, certified, documented, constantly backed up and the like. At the same time, there are many different departments, often at war with each other, subcontractors, nightmarish bureaucracy. In hosters, as a rule, a small team, only a few key people, and even the leaders are technically well-versed. In telecoms, techies do not reach the top echelon of power. Also in the case of telecoms already have ERP, CRM, some kind of their own billing, customers have a personal account and an account in the system. Accordingly, the requirements for our software have grown dramatically, especially in terms of configuration flexibility and integration capabilities, supported loads, billing complexity, ease of operation with rather complex processes, and documentation quality.

Our tasks

Scaling

Now we are increasing the number of users supported by our system by 2-3 times every year, while along with the growing number of users, the specific number of services per user is also growing, and there is a constant complication of the service sales model. In general, everyone is looking forward to (and I also have a little fear) the so-called hockey stick effect, when the growth of the system from a linear turns into an exponential. First of all, these hopes are connected with IoT (it is expected that at this moment the number of consumed services will start to grow very quickly) and the final death of desktops and the traditional model of selling software.

Ability to integrate with existing systems

As I have already said, many of our clients (telecoms) already have some kind of BSS systems, and our platform should be able to work with them. The simplest example is that any telecom already has a giant database of several tens of millions of users. Our product must be integrated into the existing system so that it can pull up user data using SSO. And then the options begin - how to write invoices? How to pay? You can use our component, you can use existing systems, such as SAP or Oracle Finance. We can deduct money from the user's credit card ourselves, or we can turn off our component responsible for payments altogether and use the existing infrastructure of the provider. Up to a certain point, we solved this problem with the help of settings that allow us to modify the behavior of the product, but several years ago we chose a more systematic approach - we divide the platform into microservices that communicate with each other using stable documented APIs, this opens up unlimited possibilities for integration , but it means total refactoring. And you need to understand that while we do not have to break existing customers.

Cost of operation

For example, I will describe one of the tasks we solve. As it was before? When the new release came out, the hoster first mentally prepared for the test called upgrade, then spent it on the weekend, was nervous for a while, if something broke, we fell asleep with ticks and curses, we repaired everything on the fly and ran all along. Telecoms are not so: - there are always three instances of the system - Dev, Staging and Production. Any change is first tried on Dev, and the process is documented. Then the documented process is checked for Staging, and only after that is applied to Production. The same thing happens when starting a new service or almost any configuration change.

Our software should support this approach. That is, it should help save the time of system operators. Because time, that is, money that is spent on operating our system, is deducted from the profit that our system brings. And for us it is important that our customers be profitable, because then they are happy with us. Therefore, we do a lot of work to adapt the platform to modern requirements - we automate everything we can, using Docker and Kubernetes, add the ability to synchronize settings between systems; We recently filed a patent application for a built-in self-test system that uses a cucumber that interprets gherkin, by the way, one of whose authors, by the way, I am.

Epoch Ingram Micro

In 2015, we were bought by Ingram Micro, which at that time, based on our platform, launched its cloud business. Ingram Micro is the world's largest distributor of IT products and services, the company in the Top100 Fortune list with a turnover of more than 40 billion dollars and a business in more than 130 countries. Ingram's traditional business is undergoing trials, somewhat similar to telecom problems — iron sales are falling, especially in the server segment (which is the most marginal). At the same time, a significant share of revenue was derived from related sales of software. Now companies are no longer buying server hardware and migrating to the clouds, end users are increasingly abandoning personal computers in favor of tablets and smartphones, software (such as Microsoft Office) is not sold on disks with laptops, but by subscription model as services. In order not to miss, but rather to lead this trend, the department of Ingram Micro Cloud was created.

Ingram is different from most of our other clients in that it sells its services not to end users, not small businesses, but resellers, who in turn resell these services. One of the features of our platform, which distinguishes it from other solutions, is that in our platform you can build an unlimited hierarchy of resellers, in fact, this made it possible to use our system for the needs of Ingram. Thanks to this model, there are many interesting tasks related to billing, since now we have to cheat sales from resellers to end users, from Ingram to resellers and from service providers to Ingram.

To make it clearer, I will explain with an example. Microsoft bills us once a month by several tens of millions of dollars. At this point, we have already billed our resellers, calculated discounts for them depending on the volume sold, prepared invoices for their clients, collected money from them, blocked the service for those customers who do not pay, and are ready to reconcile for all these financial documents . Multiply this by the number of countries in which we already work (about 30 and this number is constantly growing), add the calculation of taxes, various methods and models of payment, discounts, and you will understand something about what we are doing here. And, yes, even all types of users need to give a panel, and each has his own (the reseller has a panel in which it is more convenient to sell services, the end user has a panel in which you can manage your services, view financial reports, buy new services).

What is the most difficult in the development process?

One of the features of our work is that we make a product that is used by 150 clients with a total audience of over 10 million people. Some systems for more than a dozen years. At the same time, we do not have separate branches for customers, everyone uses the same product and upgrades to the latest versions, so a lot of attention is paid to design changes, testing, collecting feedback from customers, and analyzing.

More than 300 people work on our product, distributed among several teams that do not obey each other, but work in parallel. The description of such a company structure deserves a separate story, because at this scale many processes are built quite differently than when you work in parallel with 5-10 people. Through the years, I carried several convictions that have been confirmed in practice all this time. For example, I try to make my teams balanced. Virtually any of them can perform any tasks. Of course, they understand some components better than others. But at the same time I do not have an artificial separation, that one team can do this component here, and the other - only this one. This allows you to freely move employees between teams, if the need arises. When teams have narrow specializations, it is often a very harmful pattern, especially when there are people-components (programmers working with one component, about which nobody else knows anything). Alone, people tend to be mistaken and gradually go crazy, the power is in the team, however banal it may sound.

A very important part of my work is to let people know what is happening around them, that is, to share information. If we discard the periodic and, alas, inevitable moments when I act as a crisis manager, then most of the time I establish communication flows within the team, help information to spread freely. It sounds simple, but alas, not only everyone understands the need and importance of information exchange. Often, the most difficult thing for us is not to write to write or study some kind of framework, but to agree on the changes so that the final result will suit everyone. Yes, of course, in our sphere there are people who do not want to talk to anyone and do not think that they should tell someone. They insist that they simply be given a difficult task, and they will solve it in silence. I even need a certain number of such people so that they can take on specific problems, immerse themselves in a task for two months, and then emerge with a solution. But such people should not be too much, and one should not allow the idea of ​​separateness to infect others. We need to somehow reconcile these two categories, and the majority need to be taught to talk and explain what needs to be done.

But don't get me wrong, of course, technical literacy is a prerequisite. Personally, I look at people with perplexity who call themselves programmers, but have never read Design Patterns GoF in their life, do not know (at least approximately) how basic sorts work and what is an estimation of the complexity of an algorithm, do not know how to multithreaded or at least do not represent approximately how computers are arranged inside - what is a pn junction, a pipeline, registers. In the end, an engineer is a person who understands how things he uses (this is my own definition).

Constant analysis is an integral component of effective work, and it is necessary to analyze both the results and intentions, both short-term and long-term. In addition to fairly standard iteration review and stand-ups, we periodically organize an extremely useful exercise - teams analyze their achievements over the past year or two, how well the new functionality is being used. This makes it very cool to understand how things really are. The design and discussion of upcoming works is even more useful activity, unfortunately, prone to getting into a dead end in difficult cases and far from being as fun as cowboy programming, so here we are trying to maintain a balance between thought and fun.

In general, something like this, in future articles, we will continue the story about our company, with more concrete and more detailed details.