What is common between egg cleaning and DevOps?
This is a translation of the article by Patrick Lee Scott posted on hackernoon.com. The author suggests to get acquainted with several important principles that will help you to get pumped into DevOps.
A couple of days ago, I tried to clean the egg in a stupid way, and my girlfriend Angeli asked me why I was doing this.
')
She took another hard-boiled egg and hit it on the cutting board, then pressed on it and rolled it on the table. It took only 3 seconds to clean. And I stood and took off the shell piece by piece.
What I want to say by this: we exaggerate the value of the effort.
The best solutions are simple and effective. Is it difficult to clean a rusty bolt? Maybe yes. But only if you do not know that this can be done with the help of Coca-Cola. Still hard? Not. You just need to put the bolt in a glass and wait.
Without knowing the simple techniques, you cannot use them. Instead of realizing, you are experimenting; instead of replicating, do research.
If in programming you use an approach over and over again, because it allows you to simplify the problem to be solved, this approach becomes a template or the so-called best practice.
Despite complex and frightening names like Command Query Responsibility Segregation (CQRS) or Event Sourcing (ES), these practices help solve problems. Especially those that arise when building distributed systems.
If you look at the development as a whole, we will see that there are more universal principles, such as “Keep it Simple, Stupid” (KISS) and “Don't Repeat Yourself” (DRY). I would like to talk about similar patterns and principles applied to DevOps.
DevOps are often presented as the Promised Land on which birds sing and the sun shines. But without using the right methods, DevOps will turn into hell, and you will instil all your fingers with a shell (like me).
Creating DevOps systems, I found several solutions for myself in addition to universal principles like KISS and DRY. While they can not be called patterns, but they will help you quickly clean the egg. These solutions (in no particular order):
If you have the opportunity to purchase a finished product or in the public domain there is a necessary and convenient tool, use this.
Do not reinvent the wheel - buy it.
Did you know that you can use the same mail server that is used in Craigslist? And what is it for free? Need a mail server - do not create a new one, work with an existing one.
I like to look for tools in awesome-self-hosted lists or use Helm Hub for this.
Despite the fact that Helm is a fairly new tool for finding software, this site has already been created: https://v3.helm.sh/
With Helm, you can easily search for ready-made bikes.
Let's go back to the past when I was just starting to program. In grade 9, I learned my first "real" language - QBasic. Before that, I had been building HTML and CSS websites for a couple of years. Then the Internet was new, and, although people knew how to create packages, they did not share them, as we do now. I usually used floppy disks for storage - it would be great to find them along with an arkanoid type game that I wrote in Java Swing in grade 11 ...
All that I used then was the standard libraries, baby! At least in my thirteen, I only knew about them. Although I am sure that some java-pros (hello, Vlad and Nick!) Were already already cool. ;)
Now everything is wrong. Today you can find entire UI libraries. You can find a library to elegantly and easily deal with dates with one command.
It would be great if there was a tool to use and install fully functioning subsystems, right? Need a database? install database
Good news: there is such a tool! With Helm, you can also install fully functioning subsystems packed and ready for use.
The principle is the same as NPM and Gradle, but the packages we manage in Helm are called charts. Charts mark up containers so that they run in Kubernetes in different ways.
It turns out a turnkey solution, buying a bike. Charts are the steering wheel, and containers with descriptions that run inside Kubernetes are wheels.
What's cool about the charts is the ability to pack services and run them in any Kubernetes cluster and even share them with other people if you want.
This means that you can describe the whole environment with code:
Need to upgrade your marketing site to 1.2.0? Just make a change and commit.
Once I sat at the table, looked into my code and tried to track down the bug. Users have been complaining about him for several weeks already, and finally I had some free time to play around.
Tada-ah! Found! Fixed!
I sat behind the partition in my workplace in a sunlit room and shouted to the others: “I fixed the bug!”
Next Tuesday, when users see the release, they will definitely appreciate my efforts! But for this we will have to pack up and try to budge the release ... Maybe we will succeed if nothing happens ...
Well, okay, if the release is still next Tuesday, users will definitely appreciate it!
That's how I was deploying at my first post-college job, being a junior programmer.
Since then, much has changed.
Now I use trunk-based development and deploy modules many times a day. When I send a Pull Request, the bot will post a corresponding comment with a review code with the collected environment after the tests and builds have been completed.
Today you do not need to shout through the partition in the office.
The more freedom you give to programmers — the freedom to control the parts of the infrastructure they need — the easier it is for you to work as a DevOps engineer.
In the first lesson, you and I saw that in order to update production, it was enough to change one number and commit. In the ideal world in which we pack applications into charts, each programmer in a team has the opportunity to influence how the tool works at the production stage. Kubernetes understands charts as container descriptions, which is why they describe resource requirements.
The logic is approximately as follows: I cannot guess how much memory or what CPU settings will be required for a new service (and I think that I am not the only one). Therefore, I also deploy monitoring and alerts with the rules, according to which I and my team will be informed to Slack that these settings need to be tweaked. That is, the system will report the necessary settings when we deploy it. I used to sit for hours, sending requests through Prometheus and adjusting the settings, just like I used to otkovirival shell. And now I learned to do everything wisely.
I already hear you say: “It sounds too difficult!” No. Just set the chart.
If you can automate or simplify something, go ahead. For example, what if you could assign a DNS path by simply deploying a service with the label “expose: true”? Here are the operators. This is a more advanced Kubernetes tool, and it’s worth getting to know it. But let's not get too into details.
This was a real revelation to me. I had to look at things from a different angle, so listen carefully.
For more than ten years, I have thought that there are only a few types of environments in the world. In the simplest scenario, there is a staging and production environment. First I turned in staging, then I started, I moved to the next stage, I unrolled, I started, and so on. As soon as everything got stuck together - the son was integrated - you can go into production.
I followed this scheme year after year and never doubted it. It always has been.
And this scheme is another unproductive way to get rid of the shell.
In essence, a chart is a dependency graph. It can be used to represent the environment, and then the process of deployment in production is reduced to the deployment of a single chart.
If each team, project, or group associated with a common context has its own chart, there are several environments that allow you to group and update all services in a single transaction.
Do not take production as a huge all-encompassing environment, in fact it is a lot of small productions.
Major changes scare their scale. And if you have several small development environments, you can manage to isolate changes and make the system more flexible.
In addition, if everything is arranged in the form of charts, the variables will be set out and can be changed from the outside, uniformly. For example, to connect a less powerful kafka cluster to preproduction, where it is not required, you just need to change the configuration.
So, with sorted out. And what to do with the databases? With Kafka? With security issues?
If you read carefully, remember: I wrote that databases can be included in a chart package.
In Kubernetes, there is a separate API for launching databases and other stateful applications in a convenient form - StatefulSets.
The very essence of the existence of Kubernetes is to increase the reliability of the launch of containers. With it, and using the Velero tool (installed via the Helm Chart), you can create a backup of the entire Kubernetes cluster, along with attached disks, such as those created by StatefulSet, and restore all with a single command. In addition, it is easy to set up automatic backup on a schedule.
Backups, single-team recovery and the Kubernetes manager will help you deploy a completely new cluster and restore its backup using just two commands. A maximum of three if you want to create a new backup first.
Instead of thinking servers, you can operate with whole clusters.
Is the production annoying? Take it out of sight - at a distance of backup, restore and change DNS.
Have you ever enjoyed a VPN with pleasure?
No seriously. Is it even possible?
Google had such an attempt. But a couple of years ago, the company announced that it would not use a VPN. I think they also did not get pleasure.
Google switched to trustless networks. They do not have a master key, which is suitable for any network, but at the entrance to each service lies the key in the form of an SSO login screen. Want to go to the monitoring service? Log in using company login and password.
Only a small shift is needed: instead of using a VPN for authentication, use a proxy.
VPN also places a link to Kubernetes management on a private network, but SSO does not. But they have their own authorization mechanism. In the case of Google Cloud and AWS, this is Identity and Access Management (IAM) and the ability to whitelist IP addresses.
If it is possible to work with a less cumbersome architecture without much loss, why not? Be like Google: move from VPN to non-trusted systems.
Ah, it seems to you that systematizing is a long time? Nonsense! In the future, it will save you hours, days, and even weeks. For the work!
Systematization is the only realistic way to recreate the infrastructure more than once.
If each setting item can be improved by changing and committing something in git, in fact, your entire technological organization is declarative. Any developer with access to git can easily maintain in working condition any system.
To do this, use a few small repositories. Monorepos make people cut corners and depend on a structure with artificial importance. Instead, it’s better to use a lot of small repositories that can be linked later.
Me and my friend Matt are creating a tool called meta , which will help to do this at the development stage. Helm does it at the production stage: for him everything is a dependency graph!
Do not pick off the egg shell in pieces. Beat and roll.
A couple of days ago, I tried to clean the egg in a stupid way, and my girlfriend Angeli asked me why I was doing this.
')
She took another hard-boiled egg and hit it on the cutting board, then pressed on it and rolled it on the table. It took only 3 seconds to clean. And I stood and took off the shell piece by piece.
What I want to say by this: we exaggerate the value of the effort.
The best solutions are simple and effective. Is it difficult to clean a rusty bolt? Maybe yes. But only if you do not know that this can be done with the help of Coca-Cola. Still hard? Not. You just need to put the bolt in a glass and wait.
Without knowing the simple techniques, you cannot use them. Instead of realizing, you are experimenting; instead of replicating, do research.
If in programming you use an approach over and over again, because it allows you to simplify the problem to be solved, this approach becomes a template or the so-called best practice.
Despite complex and frightening names like Command Query Responsibility Segregation (CQRS) or Event Sourcing (ES), these practices help solve problems. Especially those that arise when building distributed systems.
If you look at the development as a whole, we will see that there are more universal principles, such as “Keep it Simple, Stupid” (KISS) and “Don't Repeat Yourself” (DRY). I would like to talk about similar patterns and principles applied to DevOps.
DevOps are often presented as the Promised Land on which birds sing and the sun shines. But without using the right methods, DevOps will turn into hell, and you will instil all your fingers with a shell (like me).
Creating DevOps systems, I found several solutions for myself in addition to universal principles like KISS and DRY. While they can not be called patterns, but they will help you quickly clean the egg. These solutions (in no particular order):
- Do not do what others have done before you.
- Let developers be as productive as possible.
- Production is a myth.
- Transfer everything to the cluster and save it all.
- VPN is too complicated, there are easier solutions.
- Systemize, automate and conquer!
Lesson 1. Don't do what others have done to you.
If you have the opportunity to purchase a finished product or in the public domain there is a necessary and convenient tool, use this.
Do not reinvent the wheel - buy it.
Did you know that you can use the same mail server that is used in Craigslist? And what is it for free? Need a mail server - do not create a new one, work with an existing one.
I like to look for tools in awesome-self-hosted lists or use Helm Hub for this.
Despite the fact that Helm is a fairly new tool for finding software, this site has already been created: https://v3.helm.sh/
With Helm, you can easily search for ready-made bikes.
Let's go back to the past when I was just starting to program. In grade 9, I learned my first "real" language - QBasic. Before that, I had been building HTML and CSS websites for a couple of years. Then the Internet was new, and, although people knew how to create packages, they did not share them, as we do now. I usually used floppy disks for storage - it would be great to find them along with an arkanoid type game that I wrote in Java Swing in grade 11 ...
All that I used then was the standard libraries, baby! At least in my thirteen, I only knew about them. Although I am sure that some java-pros (hello, Vlad and Nick!) Were already already cool. ;)
Now everything is wrong. Today you can find entire UI libraries. You can find a library to elegantly and easily deal with dates with one command.
It would be great if there was a tool to use and install fully functioning subsystems, right? Need a database? install database
Good news: there is such a tool! With Helm, you can also install fully functioning subsystems packed and ready for use.
The principle is the same as NPM and Gradle, but the packages we manage in Helm are called charts. Charts mark up containers so that they run in Kubernetes in different ways.
It turns out a turnkey solution, buying a bike. Charts are the steering wheel, and containers with descriptions that run inside Kubernetes are wheels.
What's cool about the charts is the ability to pack services and run them in any Kubernetes cluster and even share them with other people if you want.
This means that you can describe the whole environment with code:
- name: backup repository: http://jenkins-x-chartmuseum:8080 version: 0.0.2 - name: monitor repository: http://jenkins-x-chartmuseum:8080 version: 0.0.3 - name: marketing-site repository: http://jenkins-x-chartmuseum:8080 version: 1.1.10 - name: denormalizer-service repository: http://jenkins-x-chartmuseum:8080 version: 1.0.0 - name: mongo repository: https://kubernetes-charts.storage.googleapis.com/ version: 1.0.0 Need to upgrade your marketing site to 1.2.0? Just make a change and commit.
Lesson 2. Allow developers to be as productive as possible.
Once I sat at the table, looked into my code and tried to track down the bug. Users have been complaining about him for several weeks already, and finally I had some free time to play around.
Tada-ah! Found! Fixed!
I sat behind the partition in my workplace in a sunlit room and shouted to the others: “I fixed the bug!”
Next Tuesday, when users see the release, they will definitely appreciate my efforts! But for this we will have to pack up and try to budge the release ... Maybe we will succeed if nothing happens ...
Well, okay, if the release is still next Tuesday, users will definitely appreciate it!
That's how I was deploying at my first post-college job, being a junior programmer.
Since then, much has changed.
Now I use trunk-based development and deploy modules many times a day. When I send a Pull Request, the bot will post a corresponding comment with a review code with the collected environment after the tests and builds have been completed.
Today you do not need to shout through the partition in the office.
The more freedom you give to programmers — the freedom to control the parts of the infrastructure they need — the easier it is for you to work as a DevOps engineer.
In the first lesson, you and I saw that in order to update production, it was enough to change one number and commit. In the ideal world in which we pack applications into charts, each programmer in a team has the opportunity to influence how the tool works at the production stage. Kubernetes understands charts as container descriptions, which is why they describe resource requirements.
The logic is approximately as follows: I cannot guess how much memory or what CPU settings will be required for a new service (and I think that I am not the only one). Therefore, I also deploy monitoring and alerts with the rules, according to which I and my team will be informed to Slack that these settings need to be tweaked. That is, the system will report the necessary settings when we deploy it. I used to sit for hours, sending requests through Prometheus and adjusting the settings, just like I used to otkovirival shell. And now I learned to do everything wisely.
I already hear you say: “It sounds too difficult!” No. Just set the chart.
If you can automate or simplify something, go ahead. For example, what if you could assign a DNS path by simply deploying a service with the label “expose: true”? Here are the operators. This is a more advanced Kubernetes tool, and it’s worth getting to know it. But let's not get too into details.
Lesson 3. Production is a myth
This was a real revelation to me. I had to look at things from a different angle, so listen carefully.
For more than ten years, I have thought that there are only a few types of environments in the world. In the simplest scenario, there is a staging and production environment. First I turned in staging, then I started, I moved to the next stage, I unrolled, I started, and so on. As soon as everything got stuck together - the son was integrated - you can go into production.
I followed this scheme year after year and never doubted it. It always has been.
And this scheme is another unproductive way to get rid of the shell.
In essence, a chart is a dependency graph. It can be used to represent the environment, and then the process of deployment in production is reduced to the deployment of a single chart.
If each team, project, or group associated with a common context has its own chart, there are several environments that allow you to group and update all services in a single transaction.
Do not take production as a huge all-encompassing environment, in fact it is a lot of small productions.
Major changes scare their scale. And if you have several small development environments, you can manage to isolate changes and make the system more flexible.
In addition, if everything is arranged in the form of charts, the variables will be set out and can be changed from the outside, uniformly. For example, to connect a less powerful kafka cluster to preproduction, where it is not required, you just need to change the configuration.
Lesson 4. Transfer everything to a cluster and save it all.
So, with sorted out. And what to do with the databases? With Kafka? With security issues?
If you read carefully, remember: I wrote that databases can be included in a chart package.
In Kubernetes, there is a separate API for launching databases and other stateful applications in a convenient form - StatefulSets.
The very essence of the existence of Kubernetes is to increase the reliability of the launch of containers. With it, and using the Velero tool (installed via the Helm Chart), you can create a backup of the entire Kubernetes cluster, along with attached disks, such as those created by StatefulSet, and restore all with a single command. In addition, it is easy to set up automatic backup on a schedule.
Backups, single-team recovery and the Kubernetes manager will help you deploy a completely new cluster and restore its backup using just two commands. A maximum of three if you want to create a new backup first.
Instead of thinking servers, you can operate with whole clusters.
Is the production annoying? Take it out of sight - at a distance of backup, restore and change DNS.
Lesson 5: VPN is too complicated, there are easier solutions
Have you ever enjoyed a VPN with pleasure?
No seriously. Is it even possible?
Google had such an attempt. But a couple of years ago, the company announced that it would not use a VPN. I think they also did not get pleasure.
Google switched to trustless networks. They do not have a master key, which is suitable for any network, but at the entrance to each service lies the key in the form of an SSO login screen. Want to go to the monitoring service? Log in using company login and password.
Only a small shift is needed: instead of using a VPN for authentication, use a proxy.
VPN also places a link to Kubernetes management on a private network, but SSO does not. But they have their own authorization mechanism. In the case of Google Cloud and AWS, this is Identity and Access Management (IAM) and the ability to whitelist IP addresses.
If it is possible to work with a less cumbersome architecture without much loss, why not? Be like Google: move from VPN to non-trusted systems.
Lesson 6. Systematize, automate and conquer
Ah, it seems to you that systematizing is a long time? Nonsense! In the future, it will save you hours, days, and even weeks. For the work!
Systematization is the only realistic way to recreate the infrastructure more than once.
If each setting item can be improved by changing and committing something in git, in fact, your entire technological organization is declarative. Any developer with access to git can easily maintain in working condition any system.
To do this, use a few small repositories. Monorepos make people cut corners and depend on a structure with artificial importance. Instead, it’s better to use a lot of small repositories that can be linked later.
Me and my friend Matt are creating a tool called meta , which will help to do this at the development stage. Helm does it at the production stage: for him everything is a dependency graph!
Conclusion
Do not pick off the egg shell in pieces. Beat and roll.
Source: https://habr.com/ru/post/457120/
All Articles