Testing installers. Theory

Testing software installation is aimed at verifying successful installation and configuration, as well as updating or uninstalling software.
Currently, the most common software installation with installers (special programs that themselves also require proper testing.

In the real world installers may not be. In this case, you will have to install the software yourself, using the documentation in the form of instructions or readme files, describing all the necessary steps and checks step by step.

In distributed systems, where the application is deployed on an already running environment, a simple set of instructions may be small. For this, often, a deployment plan is written (Deployment Plan), which includes not only the installation steps of the application, but also roll-back steps to the previous version, in case of failure. The installation plan itself must also go through a testing procedure to avoid problems during delivery to actual operation. This is especially true if the installation is performed on systems where every minute of downtime is a loss of reputation and a large amount of funds, for example: banks, financial companies or even banner networks. Therefore, testing installers can be called one of the most important tasks in software quality assurance.
')
It is such an integrated approach with the writing of plans, step-by-step verification of the installation and rollback of the installation, which can rightfully be called installation testing or Installation Testing.

Under the cut a lot of text that should be remembered when testing.

Features testing installers:

An installer is a “regular” program, the main functions of which are Installation (Installation), Update and Uninstall (Uninstall) software.
Proverbial wisdom is well known to everyone: “ They meet according to their clothes, but they escort according to their minds ”. The installation application is the very clothes that create the first impression of your product. That is why testing the installation is one of the most important tasks.

Being an ordinary program, the installer has a number of features, among which the following should be noted:
● Deep interaction with the operating system and dependence on it (file system, registry, services and libraries).
● Compatibility of both native and third-party libraries, components or drivers, with different platforms.
● Ease of use: intuitive interface, navigation, messages and prompts.
● Design and style of the installation application.
● Compatibility of user settings and documents in different versions of the application.
● And much more.

If these features do not charge you seriously for testing installation programs, then I want to give a small list of risks that will show the full significance of the correct work of the installers:
● risk of loss of user data.
● risk of operating system failure.
● risk of inoperability of the application.
● risk of incorrect application operation.

At the same time, as with any program, some functional requirements are imposed on the installer. Combining them with a list of features, we get a more complete picture showing the amount of upcoming work on testing. Further, based on the list of requirements, you will need to answer the questions: “What to test?”, And only then - “How to test?”.

With the modern abundance of personal computers, servers and operating systems, the need has arisen to install the same software on different platforms. To do this, installers need to understand what and where they install, depending on the environment.

What to test in installation programs?

Let us write in more detail, " What ?" You need to check to assess the correctness of the installer:
● Installation (Installation).
● Correctness of the file list in the installation package:
When selecting different types of installation or installation parameters, the list of files and the paths to them may also differ.
no extra files (project files that are not included in the installation package should not be on the user's disk).
● Registration of the application in the OS.
● Registration of extensions for working with files:
for new extensions.
for existing extensions.
● Access rights of the user who puts the application:
rights to work with the system registry.
Permissions to access files and folders, for example, % Windir% \ system32 .
● Correctness of the Installation Wizard.
● Installation of several applications in one go.
● Installing the same application in different working directories of the same workstation.
● Update.
● The correctness of the file list, as well as the absence of extra files:
checking the file list with different installation parameters.
no extra files.
● Backward compatibility of generated data:
preservation and correct operation of data created before the update.
the ability of the old versions of the application to work correctly with data created in the new versions.
● Update while the application is running.
● Aborting update.
● Uninstall (Uninstall).
● Correct removal of the application:
removal of installed libraries and service records from the system registry.
deletion of physical files of the application.
delete / restore previous file associations.
safety of files created while working with the application.
deletion while the application is running.
deletion with limited access to the application folder.
deletion by the user without appropriate rights.

How to test installations?

Retrieving the file list should be done before, and checking the files themselves - after installation.
The most correct thing is to try to get a list of files from the installer of the installation package. The phrase: “Take and make a list after installing the software, everything is right there” - this is a provocation and it is better not to give in to it.
If the program contains files signed by a certificate, for example from MS, then it is possible that temporary files that are no longer needed for the operation of the application (* .tmp, etc.) may appear. Please note that the same installation package can install different sets of files for different OS. Therefore, it is necessary to test for each OS separately.

Practically for any OS, registration of working libraries and service records is important. For example, in Windows, you will need to check the system registry for correct recording of new data and registration of a new / new application.
If when you open a document that your program is designed to work with, you will receive a dialog to select a program for opening files of this type or it will be opened by another application, then there will be an error registering the extension in the OS.

In an amicable way, the installation program should check the user account at the start and immediately report any problems with the rights. But the option is not excluded that additional restrictions will be placed on any service folders. You can try to simulate situations yourself when one of the folders is closed for writing, for example, " \ Program Files ", " \ Windows ", " % Windir% \ system32 ", and also to check how the application behaves when it is impossible to record any of files on the desired path. It is possible that without some files, the performance of the entire application will not be compromised. In this case, it is enough to indicate the problem with copying the file (s) to the log, and NOT to display an error message.

It is necessary to conduct a full Testing wizard of the application.
Quite often there are situations when the application in addition to itself puts another product. This can be either a separate third-party application, or a demonstration of its own product. During the installation process, it will usually not be another set of steps of the installation wizard, but just a small message that such a product is being put. When testing, you will need to pay special attention to the fact that we have a consistent installation of several products. If the first one requires a reboot after its installation, then the second one may be installed incorrectly, especially if drivers are installed into the system both there and there. Of interest are the errors that occur at the junction of the installation of two applications.
There are applications that can be installed in different working directories of the same workstation, and at the same time they will work independently from each other, without creating any conflict situations. But it's not always the case. There may be conflicts with access to shared resources on disk, in memory and / or in the system. All this should be tested thoroughly.

Update

Backward compatibility testing includes the following steps:
● After installing the update, all objects previously created by the application, such as documents, forms, saves (if it is a game), should open and work without errors. This behavior is called backward compatibility. User settings should remain the same, of course, if the updates did not affect their change.
● Single-type documents created in the new version should be correctly opened in older versions, of course, if the purpose of the update was not to change the format and structure of the files. If a new format has been implemented, then the new version should be able to save the document in the old format.
● In the event that the updated application is running, the user should receive a warning that the update is not possible while the application is running.

Note that the update process can be stopped at any time, while the original application must remain unchanged and in working condition. Suppose you have installed version 1 of the application, you are trying to update it to version 2, but during the installation process, change your mind and interrupt the installation. In this case, the installer should return all changes already made, clean the temporary files used for the update, and complete its work. At the same time, the application version 1 remains in working condition.

Uninstall (Uninstall)

It is considered a good practice to delete created during the installation process (registry entries in the registry, libraries in the % Windir% \ system32 system directories, files, etc.). Conventionally, the process of testing uninstallation can be divided into several parts:
● Ensuring that service records and links to libraries installed during the installation are deleted from the registry.
● Check the physical deletion of application files.
● Check that after deleting the application, the file extensions registered during installation are deleted, and the existing ones (registered before installation) are restored.
● Checking the integrity of data created during the work with the application. It is likely that they lie somewhere in the depths of the catalog of the program itself. These can be service scripts, save from games or other user-created data, the deletion of which will cause damage to the user. Just imagine that when you remove MS Office, all your documents will be deleted along with it. Therefore, such data cannot be deleted without user confirmation.
● In the event that the application to be deleted is running, the user should receive a warning that the deletion is not possible while the application is running.

It is worth checking the behavior of the installer if the program directory is closed for deletion of access rights. In this case, the deletion process should not be performed, and the user should receive the appropriate message.

Testing the installation wizard

In most cases, the installer is an application in the form of a wizard, which may have specific requirements, recommendations for testing which are discussed below:

Smart people wrote: "Wizards are evil." You can agree with this or not, but you still have to test them. The following test plan for installation wizard is proposed:
● Determine all paths from start to finish, and then prioritize each one. This will help us avoid unnecessary costs and effort when passing low-priority paths.
● Forget the GUI. Try to describe the test cases without binding to the interface elements. For example, a checkbox / radiobutton GUI control or a two-item menu is just a choice between true and false, it’s important what it ultimately affects.
● If, as a result of passing the wizard, it turns out some kind of file (file, describing properties in the form of a list: property = value), which is then passed on to the export procedure, in this case, you can divide the checks into two steps - the first, create (generate) such Check the files and verify that the export is working correctly. The second is to check that the correct files are obtained through the GUI.
● Do not forget to engage in such routine testing of wizards as walking back and forth through the pages:
without changing anything, all answers should be saved; changing something on the previous page, the next should be an adequate change or reset answers.
● Make sure that the wizard responds adequately to incorrect answers and does not allow to go further.
● The Cancel button (Close) should work always and on all pages of the wizard.
● Create a template result for each of the possible paths of the installation wizard (ideally, make several of them - for different input data). Then, if possible, automatically or manually compare the result with the template result.
● Select the options that do not affect any others, and which others do not affect. The operation of these options can be tested in isolation from others.

Cross-platform installer testing

I would like to single out a separate item cross-platform testing of installers, which must necessarily be carried out for all three functions - installation, update and removal:
● Correctness of the installer with different OS versions, Service Pack (s) and installed updates.
● Check files, drivers and libraries when installing under different operating systems.
● Verification of permissions to access files, folders and system records for different operating systems.
● Checking the permissions set on the application files (Permissions).

To simplify the testing procedure, it is recommended to create a table where the columns will be the required configurations, and the lines - test cases (test cases) or test functions. In the process of testing at the intersection of columns and lines, fill in the result, which will be able to visually show the progress of testing and compliance with the “cross-platform” requirements.

If the application should work in several operating systems, the installer should check the necessary set of functions in the course of its work. It is possible that some kind of update package needs to be installed somewhere or you just need to work with another library.
A case study: “The application must be closed for installation on Windows Vista. We start it, the program thinks and gives a message that they say this version will not work under this OS. That's right, click OK. After that, the computer thinks for a couple of seconds and reports the following: “The application was incorrectly completed. Let's try to run it in compatibility mode? “Agree and run. Automatically starts our application again, but this time it is put "with a bang". We do not know what and how Vista is doing there, but we had to redo the security code. ”

Checking the list of installed files is carried out by analogy, as for installing drivers and libraries, then we should note the special importance of this check. Not all drivers and libraries work equally well on different platforms.
A case study: “For some drivers, there is a dependency on the file system on which the OS is running. More precisely for some types of errors. There was a case when everything worked fine on NTFS, but not on FAT32. The reason was in the wrong entry when installing the driver in the registry. "
It is necessary to verify that the installer has access rights to files, folders and system records. This is especially important when installing under an OS of the UNIX family, with their severe restrictions on access to resources for different categories of users. When / after installation on unix systems, files must have appropriate permissions (Permissions). Those. if the file is intended to be launched, then it should be launched, if it is a configuration file, for example, then there should be permission for modification, etc.

---

The idea of ​​my friend dartos .