11 important things you need to know about DevOps - part two
(Continuation of the translation, the first part is here )
High deployment rates, usually associated with DevOps, often put tremendous pressure on QA and Infosec. Consider a case where developers do ten deployments per day, while security personnel require four months of application security testing. At first glance, it all looks like a total discrepancy between the pace of development and security testing.
An example of the risk associated with an insufficiently proven deployment process is the known Dropbox problem in 2011, when authentication was turned off for four hours, which allowed unauthorized users to access all stored data.
The good news for QA and Infosec is that if development teams are able to maintain high deployment rates, then they probably use continuous integration and delivery practices, which means they use continuous testing practices. In other words, whenever a code is sent to a repository, tests are automatically launched, and errors should be fixed immediately, as if the developer sent a code to the repository that is not compiled.
')
Through the use of functional, integration tests and tests for information security in the daily activities of developers, defects are detected and corrected faster than ever.
More and more Infosec tools, such as Gauntlet and Security Monkey , are helping to test security by embedding this step into the design and delivery processes.
The key problem is that static code analysis tools take too long to run to integrate them into continuous integration and testing, as it often takes several hours or even days to complete their work. In these cases, infosec should distinguish specific modules that contain key security-related functionality (for example, encryption, authentication modules, and so on). Whenever these modules are changed, a full launch retest is required, otherwise the deployment may continue.
And one last note: as we can see, DevOps tasks are often more related to problem detection and recovery, as opposed to standard functional testing. In other words, when developing a boxed product, when it is very difficult to roll out updates and patches, QA relies on functional testing before delivery. On the other hand, when software is delivered as a service and defects can be corrected very quickly, in this case QA can reduce its dependence on testing, and instead rely more on monitoring to detect defects in production, since patches can be quickly installed.
Rapid recovery from crashes can be made easier with the use of the “feature flag”, which enables and disables functionality directly in the code using configuration parameters, thus avoiding full branching.
And for QA, the transition to the detection and repair approach is quite applicable in cases where the worst thing that can happen is a loss of functionality or required performance. However, when there is a risk that an error will lead to a loss of confidentiality or integrity of the system and data, in this case you cannot rely only on detection and recovery - instead, a full test cycle is required before the code is deployed, because an error in the sale can lead to irreparable consequences.
Too often in software development projects, developers use all the time exclusively on writing code. This leaves not enough time to adequately solve administrative problems. And then they try to find the simplest way to identify and test everything that is associated with code, which includes databases, operating systems, networks, virtualization, and so on.
This is certainly one of the main reasons for the ongoing discord between the development and the IT department, which naturally does not lead to an increase in the efficiency of their work. The consequences of this are well known: the environments are not well defined and specified, there are no procedures for their deployment, incompatibility between the deployed code and the launch environment, and so on.
In the described pattern, we create environments in the early stages of development, and set the rule that the code and the environment should be tested together. When developers use Agile processes, we can do something very simple and elegant.
According to Agile, we must have a working, supplied code at the end of each sprint (usually once every two weeks). We’ll change the Agile rule so that instead of just having the viable code delivered at the end of each sprint, you will also have an environment into which it is deployed starting from the very first sprint (0 or 1).
Operations, instead of being responsible for creating the production environment specification, will instead set up an automated process for creating this environment itself. This mechanism will create an environment for combat launch, as well as for Dev and QA.
By making the environments (and the tools that create them) available at the beginning, perhaps even before the project starts, developers and QA will be able to run and test their code in a stable environment, with controlled deviation from the combat.
In addition, maintaining the minimum difference between different types of environments (for example, for developers, testing, integration testing, actual launch), we will find and eliminate problems of interaction between the code and the environment long before deployment.
Ideally, the deployment mechanism should be fully automated. Tools that can be used include scripts, Puppet, Chef, Solaris Jumpstart, Redhat Kickstart, Debian Preseed, etc.
One of my favorite quotes from Patrick Lightbody, the former CEO of BrowserMob, is: “We found that when we are developers at 2 am, we get phenomenal feedback. Defects are corrected faster than ever. "
This indicates a problem when the developers check their code on Friday at 5 pm, say goodbye to each other in the parking lot, and then go home, leaving the admins to clean up the whole weekend. Worse, the defects and known errors repeated in production, force administrators to constantly extinguish the fire, and not to look for its cause, because the developers are focused on creating new functionality.
An important element of the Second Path is the reduction and enhancement of feedback so that developers become closer to the customer.
Pay attention to symmetry: Pattern 1 talks about creating environments as early as possible and embedding ops into development, while Pattern # 2 talks about entering developers into ops.
See, we are building the development team into the information chain of administrators, for example, as technical support for level 3 or even making them fully responsible for the success of the delivery, including for rolling back to the previous version and fixing problems, including informing the customer.
The goal is not to replace admins with developers. Instead, we want to let the developers know what effect their careless work can have, and let them walk in ops shoes long enough to be motivated to quickly fix problems and achieve global goals.
Another constant problem that arises in DevOps is that the tasks between development and admins are not standardized enough. For example, when each deployment occurs differently and all environments differ from each other like snowflakes. When this happens, it means that no one tried to put the configuration process on the wheels.
In this pattern we will talk about reusable deployment procedures that can be used in different projects. There is a very elegant solution for this in the Agile methodologies, where the deployment task turns into a user story. For example, we would like to build a reusable user story for the IT department called Deployment in a High Availability Environment, which then determines exactly the steps needed to create the environment, and how long it will take, what resources are needed, etc.
This information can then be used by project managers to accurately integrate deployment tasks into the project plan. For example, we would be confident in the deployment schedule if we knew that the story “Deployment in a High Availability Environment” was completed fifteen times last year, taking an average of three days, plus or minus one day.
In addition, we will also be confident that the deployment tasks are now properly integrated into each project. By doing this, we benefit from the standardization of environments (for example, less obstruction in production, increased opportunities for the IT department to support, etc.).
Over the past two years, we have seen an unprecedented number of various options for solving this problem and countless suggestions for finding new ways of interaction between the development and administration departments.
We know that the current system is not working. We know there is a better way. We know that the solution will show the true potential of IT. One hundred years later, historians will look back on this decade and say: “This was the Cambrian explosion for IT, when we finally figured out how to organize the work of IT management to win.”
Over the past five years, we have seen unprecedented growth and convergence of innovative ideas and technologies. Some of them are positive, such as Agile, cloud computing, Big Data, DevOps movement, and Lean Startup. Some were negative, such as globalization and outsourcing, increased data leakage and security, and the largest economic downturn since the Great Depression.
We strive to positively influence the lives of 1 million people in IT over the next 5 years. To make this happen, we bring together leaders in all relevant areas that have common sense to help us achieve our goal and improve IT for future generations.
8. How are Infosec and QA integrated into the DevOps workflow?
High deployment rates, usually associated with DevOps, often put tremendous pressure on QA and Infosec. Consider a case where developers do ten deployments per day, while security personnel require four months of application security testing. At first glance, it all looks like a total discrepancy between the pace of development and security testing.
An example of the risk associated with an insufficiently proven deployment process is the known Dropbox problem in 2011, when authentication was turned off for four hours, which allowed unauthorized users to access all stored data.
The good news for QA and Infosec is that if development teams are able to maintain high deployment rates, then they probably use continuous integration and delivery practices, which means they use continuous testing practices. In other words, whenever a code is sent to a repository, tests are automatically launched, and errors should be fixed immediately, as if the developer sent a code to the repository that is not compiled.
')
Through the use of functional, integration tests and tests for information security in the daily activities of developers, defects are detected and corrected faster than ever.
More and more Infosec tools, such as Gauntlet and Security Monkey , are helping to test security by embedding this step into the design and delivery processes.
The key problem is that static code analysis tools take too long to run to integrate them into continuous integration and testing, as it often takes several hours or even days to complete their work. In these cases, infosec should distinguish specific modules that contain key security-related functionality (for example, encryption, authentication modules, and so on). Whenever these modules are changed, a full launch retest is required, otherwise the deployment may continue.
And one last note: as we can see, DevOps tasks are often more related to problem detection and recovery, as opposed to standard functional testing. In other words, when developing a boxed product, when it is very difficult to roll out updates and patches, QA relies on functional testing before delivery. On the other hand, when software is delivered as a service and defects can be corrected very quickly, in this case QA can reduce its dependence on testing, and instead rely more on monitoring to detect defects in production, since patches can be quickly installed.
Rapid recovery from crashes can be made easier with the use of the “feature flag”, which enables and disables functionality directly in the code using configuration parameters, thus avoiding full branching.
And for QA, the transition to the detection and repair approach is quite applicable in cases where the worst thing that can happen is a loss of functionality or required performance. However, when there is a risk that an error will lead to a loss of confidentiality or integrity of the system and data, in this case you cannot rely only on detection and recovery - instead, a full test cycle is required before the code is deployed, because an error in the sale can lead to irreparable consequences.
9. My favorite DevOps Pattern # 1:
Too often in software development projects, developers use all the time exclusively on writing code. This leaves not enough time to adequately solve administrative problems. And then they try to find the simplest way to identify and test everything that is associated with code, which includes databases, operating systems, networks, virtualization, and so on.
This is certainly one of the main reasons for the ongoing discord between the development and the IT department, which naturally does not lead to an increase in the efficiency of their work. The consequences of this are well known: the environments are not well defined and specified, there are no procedures for their deployment, incompatibility between the deployed code and the launch environment, and so on.
In the described pattern, we create environments in the early stages of development, and set the rule that the code and the environment should be tested together. When developers use Agile processes, we can do something very simple and elegant.
According to Agile, we must have a working, supplied code at the end of each sprint (usually once every two weeks). We’ll change the Agile rule so that instead of just having the viable code delivered at the end of each sprint, you will also have an environment into which it is deployed starting from the very first sprint (0 or 1).
Operations, instead of being responsible for creating the production environment specification, will instead set up an automated process for creating this environment itself. This mechanism will create an environment for combat launch, as well as for Dev and QA.
By making the environments (and the tools that create them) available at the beginning, perhaps even before the project starts, developers and QA will be able to run and test their code in a stable environment, with controlled deviation from the combat.
In addition, maintaining the minimum difference between different types of environments (for example, for developers, testing, integration testing, actual launch), we will find and eliminate problems of interaction between the code and the environment long before deployment.
Ideally, the deployment mechanism should be fully automated. Tools that can be used include scripts, Puppet, Chef, Solaris Jumpstart, Redhat Kickstart, Debian Preseed, etc.
10. My favorite DevOps Pattern # 2:
One of my favorite quotes from Patrick Lightbody, the former CEO of BrowserMob, is: “We found that when we are developers at 2 am, we get phenomenal feedback. Defects are corrected faster than ever. "
This indicates a problem when the developers check their code on Friday at 5 pm, say goodbye to each other in the parking lot, and then go home, leaving the admins to clean up the whole weekend. Worse, the defects and known errors repeated in production, force administrators to constantly extinguish the fire, and not to look for its cause, because the developers are focused on creating new functionality.
An important element of the Second Path is the reduction and enhancement of feedback so that developers become closer to the customer.
Pay attention to symmetry: Pattern 1 talks about creating environments as early as possible and embedding ops into development, while Pattern # 2 talks about entering developers into ops.
See, we are building the development team into the information chain of administrators, for example, as technical support for level 3 or even making them fully responsible for the success of the delivery, including for rolling back to the previous version and fixing problems, including informing the customer.
The goal is not to replace admins with developers. Instead, we want to let the developers know what effect their careless work can have, and let them walk in ops shoes long enough to be motivated to quickly fix problems and achieve global goals.
11. My favorite DevOps Pattern No. 3:
Another constant problem that arises in DevOps is that the tasks between development and admins are not standardized enough. For example, when each deployment occurs differently and all environments differ from each other like snowflakes. When this happens, it means that no one tried to put the configuration process on the wheels.
In this pattern we will talk about reusable deployment procedures that can be used in different projects. There is a very elegant solution for this in the Agile methodologies, where the deployment task turns into a user story. For example, we would like to build a reusable user story for the IT department called Deployment in a High Availability Environment, which then determines exactly the steps needed to create the environment, and how long it will take, what resources are needed, etc.
This information can then be used by project managers to accurately integrate deployment tasks into the project plan. For example, we would be confident in the deployment schedule if we knew that the story “Deployment in a High Availability Environment” was completed fifteen times last year, taking an average of three days, plus or minus one day.
In addition, we will also be confident that the deployment tasks are now properly integrated into each project. By doing this, we benefit from the standardization of environments (for example, less obstruction in production, increased opportunities for the IT department to support, etc.).
IT's future
Over the past two years, we have seen an unprecedented number of various options for solving this problem and countless suggestions for finding new ways of interaction between the development and administration departments.
We know that the current system is not working. We know there is a better way. We know that the solution will show the true potential of IT. One hundred years later, historians will look back on this decade and say: “This was the Cambrian explosion for IT, when we finally figured out how to organize the work of IT management to win.”
Over the past five years, we have seen unprecedented growth and convergence of innovative ideas and technologies. Some of them are positive, such as Agile, cloud computing, Big Data, DevOps movement, and Lean Startup. Some were negative, such as globalization and outsourcing, increased data leakage and security, and the largest economic downturn since the Great Depression.
We strive to positively influence the lives of 1 million people in IT over the next 5 years. To make this happen, we bring together leaders in all relevant areas that have common sense to help us achieve our goal and improve IT for future generations.
Source: https://habr.com/ru/post/167407/
All Articles