From sysadmin to man

At DevOps there are at least two established views - from the system administrators and from the developers. The former usually boast that they use Chef / Puppet / Ansible / Docker since 200X, the latter believe that DevOps has either become obsolete and leads to NoOps, or that "I wrapped everything in a container, and then how it goes."



Business at the same time reads about DevOps in articles and hopes that the guys from the bottom will figure out what to do with it. At the same time, DevOps itself does not happen, the business is not similar to Google, the company does not become turquoise, people do not create new approaches to test hypotheses in the world.

')

This article is about DevOps as a system. How it helps the business, what competencies on the part of engineers should appear for DevOps, what business problems can be solved by the DevOps method of software production, and what mistakes are possible on the way to DevOps production and how to avoid or stop them. How, after all, an engineer to become a Man and be a creator in this world, how to build a career path for this and how to start looking at technology humanly.



The material is based on the decoding of the report by Alexander osminog Titov from our October conference DevOops 2017.







For those who are accustomed to watch reports from conferences in the recording, we attach the video.







I work for Express 42. My story is about my career path, but I will provide it with interesting tips and my own conclusions. He has the catchy name "From sysadmin to man." I am separating the role of the system administrator from some human condition. DevOps moves us towards being not just artists, but creators of new digital products and other things.



Since my story is based on my experience, I’ll tell a little about myself. I started as a system administrator when I was at university. He worked on the night shift, then began working on the day shift, then rose to the lead system administrator. Next, I worked on the social networks of connect.ua and fakultet.ru, then as a technical director at Oversan-Skalaksi. It was one of the first attempts to launch cloud hosting in Russia, as it turned out, a very early attempt. The need for cloud hosting in Russia arose just now. We only understood how to use them, understood that these are flexible resources, that they need to write applications in a different way, and so on. At that time no one understood this at all, so selling it was very difficult.



Then I worked in a startup Qik from Silicon Valley, the development office which was here in Russia. In Qik, I really felt the benefits of the DevOps concept, because in two months we made a product that grew from 0 to 5 million users. If, from the service point of view, in Oversan-Skalaksi, I felt that DevOps is needed and people need to understand what DevOps is to use cloud hosting, then in Qik I felt it as a system administrator and as a head of the system administrators department. Then we bought Skype, and we began to integrate there, as well as integrate DevOps practices there, because it was not on Skype. And then Skype bought Microsoft. It seems to have come to a company with about 40 people, and after a few years you work in a large company with 100 thousand employees. It was a very interesting experience.



As a result, I did not find where to go further in these companies, and my colleagues and I opened Express 42, which brings DevOps to the masses. This idea originated in Oversan-Skalaksy, and it moves me. Among other things, I really support the DevOps community in Russia. I am the organizer of DevOpsDays Moscow and Moscow DevOps Mitaps.



For a long time, I cared that using Ansible could be bad or good. The tool as a whole does not solve anything. You can use Docker, Kubernetes, put the latest Prometheus, but at the same time what you are doing will not be different from what people using bash scripts have. I will try to answer why it happens. In many ways, this answer lies within us, so the article is called so. The sysadmin thinks of how to write bash-scripts, and the person thinks a little differently.

How does DevOps appear in the company?

DevOps developer

There are several ways that DevOps can appear in a company. One of these ways is when the developers, tired of asking the sysadmins to do something and having heard a lot about DevOps, try to implement it. But at the same time they have a peculiar understanding of DevOps. They think that they can use any technology, wrap everything in a Docker container and send it to system administrators, but they don’t think at all about how their code will work in production. They did not change anything in their head to switch to DevOps, but at the same time they began to use new technologies.



I have seen many such companies. You come to them - they have four development departments. Each department writes its own microservice, each department has its own database. Someone Redis, someone PostgreSQL, someone PostgreSQL and MySQL at the same time. And they accompany it until the databases grow to such size that they cannot continue.



At this point, everything starts to crumble and fall apart, and terrible consequences arise. This is a technology zoo, which is not clear what to do. And the feature of the developer is that he solves the problem by dragging a new library. I think that those who work with Node.js developers know this. When Node.js developers see that the database does not work well, they can jump from PostgreSQL to another version, add a new extension, or start using some JSONB. As a result, there is a terrible state of architecture, but in general they are all satisfied. The application is difficult to manage, you will not understand what is happening there, it has low stability, it is constantly falling. In response to the falling application, developers stick something else there, and it continues to fall further, and in the end nothing is clear at all.

DevOps sysadmin

There is such a thing as DevOps-sysadmin. Usually these are very powerful guys, well-proven. They come and say: “Guys, it’s impossible to live like this, we’ve gotten the pressure on this routine, We’ll automate everything now, make a delivery pipeline, so sweet, beautiful, well-functioning. We know perfectly well how the application should work in production. Much better than these boobies writing on Node.js. And we know what to use to make everything beautiful. ”



Several times I came across the fact that such people built DevOps on FreeBSD. It turned out a closed system, which they themselves wrote, and only they understood how it works. Despite my sysadmin experience, I could not figure it out, and if I could not, how should the developer understand how to deploy through this CI system? As a result, I administratively forbade the use of FreeBSD in the company, despite the fact that I really love and respect the system itself, especially I love OpenBSD. But a week later, among Linux servers, FreeBSD servers began to appear again like fly agarics.



DevOps-sysadmins, as well as developers, think that technology is the most important thing, without them nothing can be done. If they like the technology, they try to stick it everywhere.

In Oversan-Scalaxi we came up with a term such as write-only-configurations and scripts. This is when a person could write, but no one could read.



At the same time, sysadmins continue to repair something in the soap. You come to him and offer help, and he gives you something with a three-floor mat. You do not understand, because you need to figure out what he has done. Well, if the developer comes and says, for example, that he needs a fault-tolerant database? This three-story mat will give him something, he will sit down and will not understand what to do with it. Everything, sailed, developers and system administrators do not communicate. Although inside is used all the most modern, sophisticated.



This is where the traditional idea of ​​sysadmins emerged: this is such a many-armed man who does something, but cannot figure out what it is. By the way, most HR are now looking for just such. I can tell you that finding such a person in the company will not relieve you of problems by 100%.

DevOps for business

Another way to get DevOps is from the business side. Some of your top manager went to some business conference, for example, to the valley, where he was told that if you don't have a Docker, some continuous integration (CI) and something else, then you cannot be considered technology company. The manager returns, collects employees and reads words from a notebook: DevOps, Docker, Concourse CI. The guys start to understand, and further mixed scenarios occur, as mentioned above: DevOps-developer, DevOps-sysadmin, and this all leads to a mess that you can not figure out.



In fact, I constantly see such situations. Why is this happening: you come to the conference, everyone has a rational, normal look - then you go to the trenches, to production, and there is trash and waste? That is, everyone understands everything, but for some reason does not work. I have a hypothesis that everyone understands some part, not all of it. And therefore now I will try to tell holistically about DevOps.

From Enterprise to Agile

First, from a business point of view, we have such a change: we are moving away from the enterprise, which is implementing monumental projects to transfer the business itself from point A to point B (for example, a five-year strategy to capture 70% of the market) ...



... and come to the world of Agile. The meaning of Agile is not to be flexible, but that point A is known, and B is not. And this is the most amazing thing that can happen. Because neither we, nor business are used to working with it. Imagine that you don’t know what will happen in a week or two weeks, that the manager came to you and said: “So, you try to get me something that can’t be, write down your name so that you don’t forget it in a hurry . ” And you do not know what to do, but the world and business are becoming so, and you need to get used to it. And all these practices, like Agile, Scrum, Kanban - not about how to live with it.





We are moving from the way of enterprise-enterprises and corporations to the way of technological. Some kind of software starts to interact with us in the market. If earlier people, companies interacted with us, salesmen called by phone, and so on, now, to call a taxi, order a pizza, listen to music, we go to the application. And an application is software that must be written by someone and continuously adapt to the market. And we - engineers and those who are engaged in business - must understand by application what is happening with the market. And in the end it moves us towards DevOps.



DevOps does not appear because some of you should be comfortable, and not because you need to apply steeper technology. NoSQL is not better than SQL, moreover, by 3-4 years ago it is much worse than SQL. SQL databases have been developed for 20 years, and NoSQL databases for 1 year. And we remember the deplorable state of MongoDB 4 years ago, when it was not a database at all.





DevOps is the same as before, only now we do everything at the same time. If you are a developer, you write code and immediately write tests, a wrapper for Kubernetes or just a Docker container, as it should work in production. And at the same time you have to fulfill one minimum condition - to run this code. Because many developers in the previous era did not do this: the compiler compiled, but what started there and started working in the application container is already the tenth thing. At the same time write metrics, logs that should be collected. And then you’re going to bump it all into Delivery Pipeline, Jenkins, TeamCity or something else. This is a fundamental difference between a developer in a corporation and a developer in a technology company. Here the developer begins to write not just algorithms, but the entire product.

DevOps history

This is the time to look at the history of DevOps. How did all this come about? I lived it, constantly reading the news, watching the historical chain, how it all appeared. And now I have different people from different years telling different versions of what DevOps is and how it originated. And it seems to me important to go back to basics.



In 2003, Ben Traynor at Google formed the SRE team. Google is the world's first serious digital company. Already in 2003, they were faced with the problem that the classic way that all software developers are accustomed to will fail to do what they want. And they made an innovation by introducing the SRE team, and since then they have been developing this practice.



In 2009, John Alspou and Paul Hammond gave a talk about working together on Flickr and how they divide 10 times a day. At that time it was a sensation and an explosion. And Patrick Debois spied on this report and came up with the term DevOps, organized the world community in order to develop this practice further.



There are technology companies that have the need to quickly divide. The old approaches did not fit, and they began to invent new ones. And smoothly, in a natural way, they were rearranged to create a new practice of creating software.



We are in a very difficult situation because we did not have this natural change. Such technologies come to us when something already happened there, and we still do not know how to use them. There, people evolutionarily come to this, and we have to make a revolution, rethink all this and somehow shift it to our soil.

How to use DevOps?

It is very important when applying DevOps to really realize that you are making a digital product and companies need time to market (TTM).



It is important to turn all teams into development teams. Already there is no separate sysadmin, a separate developer. Because those whom we call system administrators write what is called an infrastructure platform, and developers write what is called a product, and in this way they provide each other with a service.



Another unobvious and very important thing that we all forget about is the organization of the accumulation and exchange of knowledge within teams and between teams. We have big problems with this. We do not like to share something that, for example, is not yet ready, and this is the key to the fact that DevOps existed. We need to talk about what is not ready, to test hypotheses, we must be open to someone saying that they are not ready. Because we are accustomed to, for example, if the sysadmins tested some cool thing, they came, told, they were answered: “Not, but how does it work in production, what did you do?”



Sysadmins, realizing that somewhere they did not understand, somewhere they did not test, go, close, and this knowledge is lost, and we do not take a step forward. And it would be correct to say: “Guys, look! You have done everything very well, great work, but there is one question that you really want to ask: how will this work in production? Have you thought about it? Next time you show us how to implement this function in production, it will be very cool! ”

They already leave with the task. And in the case of our classic Russian approach “yes no no no, all garbage” they leave with the thought “why do we need to do this if we are all refused” . And this is a big problem inside the DevOps community. We do not share with each other, because we are not sure that this will not be recognized as obvious or not as hardcore as it seems, and so on.



I have already heard from conference organizers that everyone requires a megahardcore. Well, maybe, you can do not a megahardcore, but so that a person shares his real experience and so that he can talk about it, it's also great.

Developer in DevOps

The developer in DevOps does not write the code, but the product. And this is not an obvious thing, because at institutes, courses, books, as programmers, we are taught to write algorithms, not products, they are taught to solve not a business problem, but an algorithmic problem. This is a huge problem. It is very important to track in your head at what point you are solving an algorithmic problem, and at what point is a real business problem.



In a company practicing DevOps, the developer immediately thinks how his code will integrate with other components. Immediately begins to communicate about this. For example, to clarify in the chat a roadmap of the API changes that it uses to check. This is the beginning of cooperation.



The developer has an idea about the architecture of the application - this is very important, because without understanding how the architecture is structured, what the technological layers are and how they interact with each other, it is impossible to think about integration.



The developer knows how the code works in combat, and understands what happens with this application. In my example, when a developer immediately writes both code and a Docker container, and monitoring, he should have an idea of ​​how the architecture works, how the code works in production and integrates into the application. Those of you who work as system administrators or infrastructure engineers, consider how to convey this knowledge to developers. Your task is to tell them about it. You can hire consultants, but better by yourself.

Infrastructure engineer

The next role that DevOps has is an infrastructure engineer, who used to be called a system administrator. I absolutely dislike the term “DevOps Engineer” because DevOps is a general practice that covers development, testing, and operation. As I said earlier, you can have a DevOps engineer, a DevOps team, the best technologies, and not have DevOps.



Infrastructure engineer creates a platform primarily for product development, but it should be convenient for developers. This balance must be tried to respect.

An infrastructure engineer uses several layers of abstraction to provide services. For example, there was a good report by Nikolai Ryzhikov about Kubernetes, where he showed how to isolate and use several layers of abstraction. He had an ideal model there that is applied in practice. Such a model can be cut into separate levels of abstraction. This is done in order to reduce the complexity of problem solving and integration. When you have clear levels of abstraction, you can flip between them and see where the inconsistencies are. Do not trust the layers of abstraction, but they are very useful in order to be able to talk about it. That is, they should not be an absolute, but a useful tool.



The infrastructure engineer designs the platform as a product, knows how to be a product owner, what is design thinking, knows how to collect requirements from developers. But this is a high level, and I am not sure that such engineers are present in the market in the form of infrastructure engineers.

Technological technician

The tester also changes a little and becomes a technologist who achieves an improvement in the quality of the software and turns this process into code.

Release Manager / Service Station

The manager keeps in mind the process of continuous software delivery, manages business expectations and technical capabilities, and produces releases and releases. He produces, but does not plan, since the process of transition from point A to point B is non-linear, and in such circumstances he cannot plan.



As a result, they all together build an infrastructure platform, which is a tool for everyone: infrastructure engineers, developers, testers.





Here it is important that the code goes through the delivery process to the right, and the information on how it goes - to the left. You constantly get information about how your code passes through the delivery pipeline, and using this information, make changes.



The main thing that you need to convey to each other, to the developers, to all - that all this infrastructure is a common tool (like git), which is being improved by all. We can’t say that this thing is Petina’s problem, since it will be overloaded, it will not be able to cope with the complexity of the information that comes from the code, and in the end you will get a poorly functioning continuous delivery pipeline.



Within such a scheme, it is necessary to divide not responsibility, but knowledge and experience. Some people (release manager or CTO) have the knowledge and experience about everything - they hold the whole picture. Others have the knowledge and experience about the resource orchestration system, others have about the database and so on, and these are different teams, different people who take up space here and are responsible for the entire infrastructure platform as a whole.





This division into levels we in Express 42 call the concept of base-service-app. There is some basic level - the level of orchestration (allocation) of resources and various low-level infrastructure services. Infrastructure engineers have more knowledge and experience. There is a service level: different databases, load balancers, monitoring, logging, CI engines (TeamCity as engine or Jenkins as engine). There is an application level that brings these levels together through all sorts of APIs, functions, and so on. Again, I refer to the report of Nikolai Ryzhikov, he perfectly showed how it all works together and how to implement CI over Kubernetes.





Processes and technologies are extremely important. The technologies that you use besides themselves have a way to use them. For example, a knife can cut bread, and you can kill a person. So it is here, only in our case it is still more difficult, even more levels of abstraction. The processes that you create around this are very important. And these processes, in principle, cannot create one other than you within the company, because they are highly sharpened for specific applications. Now, in principle, it is impossible that, as before, you hired an ITIL consultant, and he put all the processes to you. The application of the Internet Bank and Yandex. Music are different as the sky and the earth. There are general principles, but the process itself is completely different.



Each of the layers of technology has its own processes that need to be built. And here I refer to the words of Alan Kay, who once in an interview with Habra said an amazing phrase: “Computers can be compared with musical instruments, their music is ideas . ”

Accordingly, the technologies that we have should be included in the processes that create the idea (the idea of ​​improving the product, the idea of ​​changing the process, the idea of ​​creating a new product). It is very important.



Companies that practice DevOps should organize a platform within themselves and some value system that will allow us to discuss what ideas we create using the technologies we use and how much we use these technologies (Kubernetes, Prometheus, Docker, etc.) , in order to play music, and not to break the guitar on the head of a neighbor.



In principle, from the point of view of a person within a company, the DevOps process should be arranged as follows. There should be such organizational units as committees that are assembled from people who are important to discuss this. It should not be the architecture department or the integration department, or the continuous delivery department, or the quality department - these should be committees that get together and discuss how we work now and what new ideas we are creating on the basis of our technologies. They create and change ways of working, and on the basis of this they create knowledge, some of which will be informal, and this is normal. In general, Russian people know well how to transfer knowledge by metaphors, for example, when a mechanic says “give me this crap”. Through cooperation and communication, this knowledge is created and invested into the ways in which technology is used and the technology itself, and this knowledge standard is implemented on technology.



The current moment differs from the old enterprise by the fact that there the standards were nailed, and now they are changing continuously. Over the past 3-4 years, quite a few technologies and standards of use have changed, it is even useless to fix them in documents, only in the minds of people.

If you like this report, come to the DevOops 2018 conference: there you can not only listen to the reports, but also chat with any speaker in the discussion area. The conference will be held in St. Petersburg on October 14, the cost of tickets will increase from the first of September.