Patch management. Testing monthly software updates
The material in this article is based on the experience of installing over 5,000 updates for Microsoft and Adobe products.
Patch Management is the process of managing software updates (software), without which at least one modern company is unlikely to think about the security of its IT infrastructure.
Updates or patches are an additional software tool that is used to fix detected software defects or change its functionality.
There are 2 types of updates :
Among other things, the importance of updates and their timely installation to maintain an adequate level of information security is obvious. However, in the history there were notorious cases where the “patches” of vulnerabilities in one application caused critical malfunctions in other applications. Here are a few such examples:
Such unpleasant consequences of the application of updates led to business downtime and loss of money, and this risk has not gone away. However, trouble could have been avoided by thoroughly testing updates by the developer before deploying them.
')
Microsoft has repeatedly criticized itself for insufficient testing of monthly security updates. In response, representatives of the corporation made it clear that their plans were to switch to a new testing scheme by transferring the service of testing updates to “external” users, that is, not Microsoft employees.
And, indeed, if you look at this problem through the eyes of a development company, it becomes clear that testing updates on the developer’s side is a rather time-consuming process. Each update should be tested in conditions as close as possible to those that are in use at workstations and servers of product consumers. So, for example, the Adobe company spends up to 6000 man-hours per month for testing updates.
But life shows that testing by a developer will never be enough for 100% of users in the world. Accordingly, the risk that the next software update will stop the company's business processes remains. On the other hand, it is also impossible not to be updated, since you can be vulnerable to hackers who can harm the company.
As a way to significantly reduce these risks, large companies choose to install updates regularly with packages (releases) with mandatory testing of updates before deploying them throughout the company.
Update Management Methods
The update management method is a combination of an approach to testing updates and an approach to deploying releases with updates. We will tell about them further.
The two most common approaches to testing updates are:
Modern technologies make it possible to test updates using virtualization without the use of unnecessary pieces of equipment. The most frequently used and available products for creating virtual machines and networks: VMware or Oracle VirtualBox. The benefits of virtualization are that:
As a rule, virtualization technology is used for small networks with no more than 45-70 workstations.
Also, the change management process can be simplified using just a couple of corporate computers or so-called test clients. By installing the necessary updates on such clients and testing the system after installation, you can see the effects of the changes and verify the result of applying the updates, while having a minimal impact on the IT infrastructure. Testing using a complete test environment is applicable for large industrial networks and guarantees high purity testing. When testing in a full-fledged test environment, the same approaches to installing updates and update management tools are used as in an industrial environment. And now we turn to them and at the same time we will consider approaches to the deployment of releases with updates.
Update Management Tools
As a rule, all widely used software solutions for remote management of IT infrastructure provide the ability to manage updates, for example:
The leader in demand and use in large industrial networks is SCCM. Therefore, we give the details of the implementation of the management of updates on his example.
The principle of managing updates with Configuration Manager is quite simple. All managed SCCM systems are combined into sites. Sites contain:
Each of the site servers must have access to the Microsoft SQL Server database.
Windows Server Update Services (WSUS) is used as an update server . It synchronizes with the Microsoft website, downloading updates that can be distributed within the corporate LAN. This saves external company traffic and allows you to quickly install bug fixes and vulnerabilities in Windows operating systems at workstations, as well as centrally manage updates to servers and workstations.
Here it should be noted that the WSUS server is intended only for updates released by Microsoft.
However, there is another update server - System Center Updates Publisher (SCUP) from Microsoft, which allows you to manage third-party software updates and then import it to the WSUS server, and then deploy it to clients using SCCM.
Stages of the Patch Management process
The update management process consists of several stages:
Testing steps
As mentioned earlier, testing takes place on the basis of a test matrix. Below we present a description of the basic checks that are usually included in such matrices.
Testing the installation . Installing updates on a test client is initiated by SCCM. As a rule, after installing updates, the system requires a reboot. Accordingly, the first check: reboot the system, log in after rebooting under the test account, make sure there are no system error / warning messages.
The next step is to check the system variable PATH . Make sure that the PATH does not contain characters and characters that can lead to problems.
The third step is to check the system event log (Event Viewer). Information about the installation of updates is written to the log under ID 19. Checking ensures that the installation of all updates is successful and that there are no errors.
Next in line is checking office applications . As a rule, Office tests intersect all the test matrices of the Patch Management process.
Simple examples of Office checks:
This sample Office check is basic and generic. The matrix may include more complex detailed checks. For example, besides opening MS Word and saving the document, you need to check that the Spelling and Grammar checkbox is ticked in the Spelling and Grammar tab, etc.
One of the most common checks is also the verification of Adobe applications that are being updated. The minimum check for Adobe Flash Player is to check the version of this software on the site. If the update is for Adobe Reader, then the version is checked and the application is tested.
Check browser performance also goes to the list of basic. Patches should not make changes to the proxy and ActiveX settings, and this is also checked during testing.
Also, pay attention to the overall impact of updates on the system. At the stage of preparing a test client, which already contains previously tested updates, the tests described above are conducted, and the results of such testing are taken as the standard.
Any deviation of the system from the state prior to the newly installed patches is analyzed and, in some cases, it may be decided to exclude patches leading to undesirable changes in the system.
It is worth noting that to save useful time, some checks can be automated using VBScript, for example:
Summing up, we would like to emphasize once again that the streamlined update management process will make it possible to make sure in advance of the success of the updates and preserve the reliability and performance of the infrastructure with the constant installation of updates ensuring the necessary level of information security.
Posted by: Alisa_Khaliullina
Patch Management is the process of managing software updates (software), without which at least one modern company is unlikely to think about the security of its IT infrastructure.
Updates or patches are an additional software tool that is used to fix detected software defects or change its functionality.
There are 2 types of updates :
- for operating systems and server software, which are used to maintain an adequate level of security and eliminate security holes;
- for application software (for example, Microsoft Office, Adobe Acrobat, or client parts of business applications) that are needed to solve problems with frequently used or important libraries and other parts of the source code.
Among other things, the importance of updates and their timely installation to maintain an adequate level of information security is obvious. However, in the history there were notorious cases where the “patches” of vulnerabilities in one application caused critical malfunctions in other applications. Here are a few such examples:
- Microsoft's monthly update caused VMWare to fail (February 8, 2011);
- An update for the Internet Explorer browser, released by Microsoft, contained two vulnerabilities at once: a crash in the browser, as well as a buffer overflow that allows a hacker to launch a code on the vulnerable machine with the privileges of the browser user (August 8, 2006);
- Microsoft’s December updates were intended to correct a serious security breach in the use of the OpenType font. The update affected PowerPoint, Quark Xpress and Coreldraw users. The released updates did not allow programs to recognize OpenType font characters larger than 15 pixels (December 12, 2012);
- Installing Microsoft updates for kernel mode drivers led to a blue screen of death (BSOD) on users' computers in August 2014;
- The update for Office PowerPoint 2013 application made it impossible to launch the application (February 10, 2015).
Such unpleasant consequences of the application of updates led to business downtime and loss of money, and this risk has not gone away. However, trouble could have been avoided by thoroughly testing updates by the developer before deploying them.
')
Microsoft has repeatedly criticized itself for insufficient testing of monthly security updates. In response, representatives of the corporation made it clear that their plans were to switch to a new testing scheme by transferring the service of testing updates to “external” users, that is, not Microsoft employees.
And, indeed, if you look at this problem through the eyes of a development company, it becomes clear that testing updates on the developer’s side is a rather time-consuming process. Each update should be tested in conditions as close as possible to those that are in use at workstations and servers of product consumers. So, for example, the Adobe company spends up to 6000 man-hours per month for testing updates.
But life shows that testing by a developer will never be enough for 100% of users in the world. Accordingly, the risk that the next software update will stop the company's business processes remains. On the other hand, it is also impossible not to be updated, since you can be vulnerable to hackers who can harm the company.
As a way to significantly reduce these risks, large companies choose to install updates regularly with packages (releases) with mandatory testing of updates before deploying them throughout the company.
Update Management Methods
The update management method is a combination of an approach to testing updates and an approach to deploying releases with updates. We will tell about them further.
The two most common approaches to testing updates are:
- testing on local virtual machines;
- testing in a complete test environment.
Modern technologies make it possible to test updates using virtualization without the use of unnecessary pieces of equipment. The most frequently used and available products for creating virtual machines and networks: VMware or Oracle VirtualBox. The benefits of virtualization are that:
- the creation of the virtual machine itself is not time consuming;
- There may be several virtual machines on the same physical platform;
- each virtual machine has its own virtual hardware components: memory, processor, hard disk, network adapters;
- and most importantly, it is the ability to take a “snapshot” of the current state of the system and the contents of the disks with one click of the mouse, and then return to its original state within a very short time, which can be very useful provided that an update is detected, the installation of which causes critical system malfunctions .
As a rule, virtualization technology is used for small networks with no more than 45-70 workstations.
Also, the change management process can be simplified using just a couple of corporate computers or so-called test clients. By installing the necessary updates on such clients and testing the system after installation, you can see the effects of the changes and verify the result of applying the updates, while having a minimal impact on the IT infrastructure. Testing using a complete test environment is applicable for large industrial networks and guarantees high purity testing. When testing in a full-fledged test environment, the same approaches to installing updates and update management tools are used as in an industrial environment. And now we turn to them and at the same time we will consider approaches to the deployment of releases with updates.
Update Management Tools
As a rule, all widely used software solutions for remote management of IT infrastructure provide the ability to manage updates, for example:
- System Center Configuration Manager (SCCM) - Microsoft software;
- Unicenter Software Delivery - a software product of Computer Associates;
- OpenView is a software product from Hewlett-Packard.
The leader in demand and use in large industrial networks is SCCM. Therefore, we give the details of the implementation of the management of updates on his example.
The principle of managing updates with Configuration Manager is quite simple. All managed SCCM systems are combined into sites. Sites contain:
- site servers;
- Site systems that perform specific infrastructure management roles
- actually managed clients.
Each of the site servers must have access to the Microsoft SQL Server database.
Windows Server Update Services (WSUS) is used as an update server . It synchronizes with the Microsoft website, downloading updates that can be distributed within the corporate LAN. This saves external company traffic and allows you to quickly install bug fixes and vulnerabilities in Windows operating systems at workstations, as well as centrally manage updates to servers and workstations.
Here it should be noted that the WSUS server is intended only for updates released by Microsoft.
However, there is another update server - System Center Updates Publisher (SCUP) from Microsoft, which allows you to manage third-party software updates and then import it to the WSUS server, and then deploy it to clients using SCCM.
Stages of the Patch Management process
The update management process consists of several stages:
- Training test clients
An operating system image is rolled onto the update testing machine, which includes applications, as well as approved previously tested updates. Upon subsequent loading of the operating system, the data of the approved updates is automatically installed. - Creating update sheets
After the test environment has been prepared for installing new updates, the creation of update sheets begins, or, as the SCCM calls it, patch lists.
The patch list includes updates that came out this month and fit the definition of "required." The need for an update for the client is determined by SCCM itself. The logic is simple: if the Visio application is installed on the client and an update for Visio has been released this month, then a similar patch will be “required” for this client.
As for updates for operating systems and servers, the need here is determined depending on their bit depth and the Windows Service Pack.
After the patch list is created, a new package is created, which includes all selected updates, and the package is assigned to the collection to which the test clients are added. - Deployment in a test environment (LAB Deployment)
When adding a package to the collection, SCCM receives information that a new task is assigned to the customers of this collection, synchronizes with the SCCM client on the test machine, and deployment begins (deployment). The procedure takes only a few minutes, after which, the corresponding icon appears in the taskbar notification area. After double-clicking on the icon, Configuration Manager opens, where after selecting the updates, the installation begins. As a rule, installation ends with a request to reboot the system. After it, testing begins with the task: to find a patch that breaks the system / system components, or to make sure that it does not exist.
At its core, logging in after a reboot is already one of the check points. It is important that there are no warnings / errors at login. Further testing is carried out on a specific test matrix. The checks included in the test matrix depend mainly on the software of the tested IT infrastructure. More information about the tests will be described in the section “Test stages” below. - Deploying to pilot users (PRE Deployment stage)
At the PRE Deployment stage, a list of tested updates is prepared, which is sent via SCCM to the clients of the pilot users. The principle is the same: the update package is added to the collection that includes only pilot clients, SCCM receives information that a new task has been assigned to the clients of this collection and the deployment begins.
As a rule, as pilot clients, those users are selected who
well versed in the software for which they are responsible, and check whether the application works normally after installing updates. The number of users who participate in the pilot usually varies depending on the number of network workstations.
The PRE Deployment stage continues for 4 days, after which the pilot users give their feedback. If there are conflicts of software with installed updates, the latter are excluded from the list. Thus, the final list of updates that will be installed on all workstations in the industrial network is formed. - Deploy tested updates in a production information environment (PRO Deployment)
It occurs at the stage of PRO Deployment, with the help of SCCM, on the same principle as for test and pilot clients. In this case, the start time of the deployment and the end date are set.
Testing steps
As mentioned earlier, testing takes place on the basis of a test matrix. Below we present a description of the basic checks that are usually included in such matrices.
Testing the installation . Installing updates on a test client is initiated by SCCM. As a rule, after installing updates, the system requires a reboot. Accordingly, the first check: reboot the system, log in after rebooting under the test account, make sure there are no system error / warning messages.
The next step is to check the system variable PATH . Make sure that the PATH does not contain characters and characters that can lead to problems.
The third step is to check the system event log (Event Viewer). Information about the installation of updates is written to the log under ID 19. Checking ensures that the installation of all updates is successful and that there are no errors.
Next in line is checking office applications . As a rule, Office tests intersect all the test matrices of the Patch Management process.
Simple examples of Office checks:
- Launch Word, Excel, Power Point, Publisher, Access.
- Ensure that applications open without error.
- Make changes to the document, make sure that when you close it, you are asked to save and save the document by default to the Documents folder (if no other default storage is provided).
This sample Office check is basic and generic. The matrix may include more complex detailed checks. For example, besides opening MS Word and saving the document, you need to check that the Spelling and Grammar checkbox is ticked in the Spelling and Grammar tab, etc.
One of the most common checks is also the verification of Adobe applications that are being updated. The minimum check for Adobe Flash Player is to check the version of this software on the site. If the update is for Adobe Reader, then the version is checked and the application is tested.
Check browser performance also goes to the list of basic. Patches should not make changes to the proxy and ActiveX settings, and this is also checked during testing.
Also, pay attention to the overall impact of updates on the system. At the stage of preparing a test client, which already contains previously tested updates, the tests described above are conducted, and the results of such testing are taken as the standard.
Any deviation of the system from the state prior to the newly installed patches is analyzed and, in some cases, it may be decided to exclude patches leading to undesirable changes in the system.
It is worth noting that to save useful time, some checks can be automated using VBScript, for example:
- creating objects for office applications, opening program files, making changes, saving;
- launch browser and go to web pages;
- uploading Event Viewer logs to the Excel table.
Summing up, we would like to emphasize once again that the streamlined update management process will make it possible to make sure in advance of the success of the updates and preserve the reliability and performance of the infrastructure with the constant installation of updates ensuring the necessary level of information security.
Posted by: Alisa_Khaliullina
Source: https://habr.com/ru/post/251575/
All Articles