Risk based testing
To ensure the quality of the information product in medicine, insurance, banking and other industries, along with other testing methods, it is important to use risk-based testing. To check choose the most risky areas of software being created. This allows you to anticipate negative scenarios and successfully implement the customer's business goals.
In this article, we will describe how we at SimbirSoft worked with risks (using the example of the Internet acquiring system and other projects) and which risk assessment and risk management methods we use in customer projects.
To begin with, we will give an example from our practice in the development for the banking sector.
')
Customer proposal: focus on the web version of the bank for individuals.
The risk we identified: the possible loss of the audience due to the low demand of the web version of individuals, in contrast to the mobile client-bank.
Our arguments: statistics of user preferences based on reviews and audience distribution by mobile and web versions.
Conclusions: the mobile version is more convenient for individuals, as the phone is always at hand. Operations are performed quickly, the authorization system allows access to all convenient services. In this case, quick access to a limited set of the most popular features is important.
Legal entities are important completeness of the functions provided, the ability to upload, print and work with a large amount of information. For this, the web version is more convenient.
Our solution: focus on the mobile client bank for individuals.
At the beginning of the project it is important to choose the right testing strategy. Consider with examples why it is important and how to choose it.
Sooner or later, all companies are faced with the need to organize the testing process and come to understand that building its strategy is an important step in software development. Sometimes - through their own bitter experience. It is especially dangerous to underestimate the role of testing and selecting a strategy when developing large-scale projects. The testing process should be chosen for the business goals and specifics of the project, otherwise it will not lead to positive results in a month or a year.
For example, consider the testing of mobile and web applications for the bank. At the start of the project, we selected a strategy based on requirements with a low level of detail. We used checklists to reduce the time spent on testing and maintaining a test basis. With the growth of functionality, the addition of acquiring, sms-authentication and notification, more complex systems, checklists could no longer cope with their task. Over time, more and more QA specialists joined the team, it was necessary to transfer information and coordinate their actions. With the complication of the product, any change could affect the related functions, that is, the risk of regression increased. There was a need to automate regression tests, so we switched to test cases.
Conclusion: depending on the project, its specifics or development stage, the testing strategy changes.
The strategy must be selected under the project objectives in order to ensure the quality of the product being produced in the best way. She answers the questions “what”, “where” and “when” will be tested. At any point in time you know where you are and where you will come in the future - according to the strategy.
The business goal may be to ensure the security of the customer's data, as well as the software itself at the production stage. Security begins with the development process and continues through the testing phase.
For example, on one of the projects, we created a secure environment for development and testing, deployed an infrastructure that met all requirements and helped protect data. They requested certified tokens and personalized flash drives for each QA specialist to access the test infrastructure. So we provided the business goal of the project in software security and kept the confidentiality of customer and user data.
Due to the testing strategy, you can focus on the really important aspects for a particular project. It is logical that the release of a mobile game or a complex banking CRM system requires different approaches to testing.
We at SimbirSoft in our practice used the whole range of development methodologies, but flexible technologies always remain priorities for us. And even where for some reason there is no possibility to use them, the team adopts the best practices and applies them in their daily work. Testing becomes an integral part of the whole process and flows into the general workflow. In this case, it is responsible not only for the quality of the product, but even for the quality of the whole process of work.
What technologies we use:
The testing strategy fully reveals itself in projects with complex business logic. For example, software for information support of banks, building an Internet acquiring system, an automated marketplace. In such projects, it is important to apply a suitable testing strategy, since the price of some errors can lead to real losses and greatly affect the company's reputation for the worse.
Also to the main objectives of testing - the search for defects and checking the software for compliance with the requirements - may be added. For example, it is important for banks to quickly implement new requirements of the Central Bank. This means that timely testing with the required quality for critical tasks will be added to the main goal.
Recently in the project of the banking sector, we are faced with a change in federal legislation - an increase in the VAT rate from 18% to 20%. In order to adapt to changing the legislation, a great deal of preliminary work was done, since the change concerned not only the replacement of forms, interfaces, but also the alteration of the logic of several calculation formulas. It was necessary to ensure proper mapping on many platforms. Also in the function of deferred payment it was necessary to take into account the transition period with the rates of 18% and 20%.
Now let's talk in more detail about how we are building a strategy and why we often choose risk-based testing.
There are several testing strategies that are chosen for specific goals:
In the case of working with projects with complex business logic, it is necessary to define stringent requirements when designing systems on which testing is built on. A suitable tool is testing based on requirements.
One type of requirements-based strategy is risk-based testing. In this case, first of all, those parts of the system functionality that are most exposed to risks are tested. Risk is a possible negative consequence of system malfunction. The consequence is a risk in the presence of two components, such as opportunity and negativity.
Risks are of two types:
1. Product Risk
It can be controlled and unmanaged. In the example above, we were faced with manageable risk: rapid growth and complexity of functionality, and therefore the likelihood of regression increased. Here we solved the problem by having a clear test base and subsequent automation. The risk that we cannot influence is dependence on external systems and their failure in the integration process. Here we lay the events that will reduce their impact on our system. For example, the use of backup, exception handling, warning output for the user.
2. Project Risk
For example, on the project we were faced with the fact that the customer had not worked with a distributed team before, and therefore the workflow used did not meet the requirements and created additional communication problems: lack of understanding, duplication of tasks, execution of mutually exclusive tasks, and so on. We agreed on the restructuring and improvement of the process - we reworked the workflow, introduced all the team members, held rallies, presentations and retrospectives. As a result, the work went in the right direction.
The risk-based approach allows you to create a certain amount of risks, to test risks with high priority in a short period of time and to further provide the customer with metrics, how well they have been tested, showing the number of planned and completed cases and the number of defects.
The implementation of the risk-based approach on the project takes place in four stages:
Risk Identification - at this stage it is necessary to identify the risks and get their list.
Risk Assessment - here we analyze the list and classify it by priority.
Risk Mitigation - at this stage we determine how deeply we will test the risks.
Risk Management - here we decide how we will continue to work with them and pass them, to identify new risks.
Risks are identified and evaluated by a group of stakeholders during brainstorming sessions. The team should include business analysts or carriers of knowledge about the system from the customer, developers, manager or project manager, architects, QA-specialists. In order to identify and assess risks, we involve, among other things, information security specialists, employees who work directly with the current system, a business analyst who is immersed in processes.
Consider a risk-based approach on the example of testing the Internet acquiring system.
Highlight the following requirements:
D1.Providing 1000 simultaneous connections to the system per second.
D2. Security of transactions.
D3. Access to the transaction should only be available to the person making the transaction.
D4. Providing and maintaining standard SET (Secure Electronic Transaction).
As a product risk, we can highlight:
RP1. System crash with simultaneous connections.
RP2. Using SQL injection in the transaction process.
RP3. Access to someone else's transaction when changing parameters in the URL.
RP4. Data loss in the event of a connection failure with the bank at the time of the transaction.
RP5. The use of non-valid certificates while securing the SET (Secure Electronic Transaction) system.
As organizational risks:
RO1. The fall of the developed system due to the unavailability of external systems.
RO2. The presence of hard-to-reproduce cases that cannot be detected in the test environment.
Thus, each risk is logically derived from the requirements that are in the system, but not limited to them. Risks supplement the requirements and reveal additional cases that may arise during the work with the system.
Risks can be reduced or compensated depending on the project objectives and system requirements. The condition is accepted that the risk is covered by the test case at least once:
1. A set of test cases RP1-RP4 is prepared for each product risk with the condition that each requirement and each risk must be covered by at least one test case. Depending on the purpose of testing, the set of test cases is expanded to a certain level of detail.
2. For each organizational risk a list of measures is prepared that can reduce the impact of risk on the product being developed.
There are several risk assessment and risk management techniques: lightweight methods (PRAM, PRISMA) and more formal methods (FMEA, FTA).
With the FMEA model, the project team identifies all the processes, components, system modules in which a system failure or error may occur that could lead to a deterioration in the quality of the product. During the brainstorm, the system risks are determined by three indicators: severity, priority, probability. Then the Risk Priority Number for each risk is calculated and the depth of testing is set depending on the indicators.
With the FTA model, all the reasons that can lead to defects in the business processes of the system are determined step by step. The analysis runs from the highest to the lowest level. The difference between FMEA and FTA is that the first approach focuses on the functionality of the system, and the second - on business processes.
In addition to these formal and heavyweight approaches, there are more flexible and informal ones. For example, PRAM (Pragmatic Analysis and Risk Management) and PRISMA (Product Risk Management) methods. They successfully and easily combine strategies based on risk and requirements. Basically, these approaches use requirement specifications as input, but stakeholders can participate in the process.
Lightweight risk analysis methods are very similar to formal ones. They focus on technical or commercial risks, weighing the likelihood of risk occurring and the factors influencing this likelihood.
However, the only two factors used by these methods are the probability of risk and its effect. In addition, simplified qualitative judgments and scales are used to make decisions in these approaches.
Our teams use flexible, lightweight methods and adapt the PRAM and PRISMA approaches to fit their goals.
For example, in one of the projects we identify project and product risks that may work. To do this, analysts, developers and QA are involved in the analysis - one representative from each team.
Formed risk table with products. They determine the estimated probability of risk triggering and its possible impact on the system on a five-point scale. In table 1 - the strongest influence, 5 - the weakest. Also for probability 1 - a high probability, 5 - a small probability.
Next, for each of them, the risk coverage of the product is traced with test cases.
Select the following checks:
TC1. Load verification with more than 1000 system connections
TC2. Load check at 1000 system connections
TC3. Entering SQL injection at the time of the transaction
TC4. Entering SQL injection on successful transaction page, by substituting other data
TC5. Entering XSS (Cross-Site Scripting) scripts into available input fields when making a transaction
TC6. Imitation of breaking the connection with the Internet during the transaction
TC7. Exit transaction session at confirmation
TC8. Verify expired certificates when making a transaction
Thus, priority checks are TC2, TC4, TC5.
TC6 and TC8 have a high impact, but a lower probability, so the priority of checking them is lower, but checks are also required.
TC7 does not apply to any of the requirements, but extends the test to a negative scenario, which is possible with a stable system.
We also define actions for project risks, for which we assign additional measures and solutions.
At risk “RO2. The presence of hard-to-reproduce cases that cannot be detected in the test environment "can do the following:
It is important to have an action plan in case one of the product or project risks works. Sometimes setting up a project backup system can save. It is not always possible to reduce the risk to a minimum level, but it should be possible to reduce at least its effects. On this topic was our post "Christmas story" : how can work the risk, the probability of which tends to zero.
For example, we must completely eliminate data loss during a transaction, but consider all cases too laborious. Therefore, you need to have ways to handle such cases. One of the options for safety net is the development of more detailed logging. This will provide a permanent point of rollback to the previous action, if in the process of making the transaction something went wrong.
Risk-based testing allows you to cover the most risky areas with test cases, thereby reducing their impact and likelihood of triggering. This is the most winning strategy for systems with complex business logic and a high cost of error. The solution is suitable for the banking sector, insurance companies, complex internal CRM-systems of medical profile. With the risk-based approach, we also work with project risks, thereby improving the overall process of testing and project management.
In this article, we will describe how we at SimbirSoft worked with risks (using the example of the Internet acquiring system and other projects) and which risk assessment and risk management methods we use in customer projects.
Risks at the start of the project
To begin with, we will give an example from our practice in the development for the banking sector.
')
Customer proposal: focus on the web version of the bank for individuals.
The risk we identified: the possible loss of the audience due to the low demand of the web version of individuals, in contrast to the mobile client-bank.
Our arguments: statistics of user preferences based on reviews and audience distribution by mobile and web versions.
Conclusions: the mobile version is more convenient for individuals, as the phone is always at hand. Operations are performed quickly, the authorization system allows access to all convenient services. In this case, quick access to a limited set of the most popular features is important.
Legal entities are important completeness of the functions provided, the ability to upload, print and work with a large amount of information. For this, the web version is more convenient.
Our solution: focus on the mobile client bank for individuals.
At the beginning of the project it is important to choose the right testing strategy. Consider with examples why it is important and how to choose it.
Testing strategy must meet business objectives.
Sooner or later, all companies are faced with the need to organize the testing process and come to understand that building its strategy is an important step in software development. Sometimes - through their own bitter experience. It is especially dangerous to underestimate the role of testing and selecting a strategy when developing large-scale projects. The testing process should be chosen for the business goals and specifics of the project, otherwise it will not lead to positive results in a month or a year.
For example, consider the testing of mobile and web applications for the bank. At the start of the project, we selected a strategy based on requirements with a low level of detail. We used checklists to reduce the time spent on testing and maintaining a test basis. With the growth of functionality, the addition of acquiring, sms-authentication and notification, more complex systems, checklists could no longer cope with their task. Over time, more and more QA specialists joined the team, it was necessary to transfer information and coordinate their actions. With the complication of the product, any change could affect the related functions, that is, the risk of regression increased. There was a need to automate regression tests, so we switched to test cases.
Conclusion: depending on the project, its specifics or development stage, the testing strategy changes.
The strategy must be selected under the project objectives in order to ensure the quality of the product being produced in the best way. She answers the questions “what”, “where” and “when” will be tested. At any point in time you know where you are and where you will come in the future - according to the strategy.
The business goal may be to ensure the security of the customer's data, as well as the software itself at the production stage. Security begins with the development process and continues through the testing phase.
For example, on one of the projects, we created a secure environment for development and testing, deployed an infrastructure that met all requirements and helped protect data. They requested certified tokens and personalized flash drives for each QA specialist to access the test infrastructure. So we provided the business goal of the project in software security and kept the confidentiality of customer and user data.
Due to the testing strategy, you can focus on the really important aspects for a particular project. It is logical that the release of a mobile game or a complex banking CRM system requires different approaches to testing.
Testing strategy in the banking sector
We at SimbirSoft in our practice used the whole range of development methodologies, but flexible technologies always remain priorities for us. And even where for some reason there is no possibility to use them, the team adopts the best practices and applies them in their daily work. Testing becomes an integral part of the whole process and flows into the general workflow. In this case, it is responsible not only for the quality of the product, but even for the quality of the whole process of work.
What technologies we use:
- flexible planning and preparation of internal releases;
- user story preparation;
- holding status rallies;
- conducting retrospectives.
The testing strategy fully reveals itself in projects with complex business logic. For example, software for information support of banks, building an Internet acquiring system, an automated marketplace. In such projects, it is important to apply a suitable testing strategy, since the price of some errors can lead to real losses and greatly affect the company's reputation for the worse.
Also to the main objectives of testing - the search for defects and checking the software for compliance with the requirements - may be added. For example, it is important for banks to quickly implement new requirements of the Central Bank. This means that timely testing with the required quality for critical tasks will be added to the main goal.
Recently in the project of the banking sector, we are faced with a change in federal legislation - an increase in the VAT rate from 18% to 20%. In order to adapt to changing the legislation, a great deal of preliminary work was done, since the change concerned not only the replacement of forms, interfaces, but also the alteration of the logic of several calculation formulas. It was necessary to ensure proper mapping on many platforms. Also in the function of deferred payment it was necessary to take into account the transition period with the rates of 18% and 20%.
Now let's talk in more detail about how we are building a strategy and why we often choose risk-based testing.
Advantages of risk based testing
There are several testing strategies that are chosen for specific goals:
- based on requirements;
- methodical;
- reactive strategies;
- advisory strategies.
In the case of working with projects with complex business logic, it is necessary to define stringent requirements when designing systems on which testing is built on. A suitable tool is testing based on requirements.
One type of requirements-based strategy is risk-based testing. In this case, first of all, those parts of the system functionality that are most exposed to risks are tested. Risk is a possible negative consequence of system malfunction. The consequence is a risk in the presence of two components, such as opportunity and negativity.
Risks are of two types:
1. Product Risk
It can be controlled and unmanaged. In the example above, we were faced with manageable risk: rapid growth and complexity of functionality, and therefore the likelihood of regression increased. Here we solved the problem by having a clear test base and subsequent automation. The risk that we cannot influence is dependence on external systems and their failure in the integration process. Here we lay the events that will reduce their impact on our system. For example, the use of backup, exception handling, warning output for the user.
2. Project Risk
For example, on the project we were faced with the fact that the customer had not worked with a distributed team before, and therefore the workflow used did not meet the requirements and created additional communication problems: lack of understanding, duplication of tasks, execution of mutually exclusive tasks, and so on. We agreed on the restructuring and improvement of the process - we reworked the workflow, introduced all the team members, held rallies, presentations and retrospectives. As a result, the work went in the right direction.
The risk-based approach allows you to create a certain amount of risks, to test risks with high priority in a short period of time and to further provide the customer with metrics, how well they have been tested, showing the number of planned and completed cases and the number of defects.
The implementation of the risk-based approach on the project takes place in four stages:
Risk Identification - at this stage it is necessary to identify the risks and get their list.
Risk Assessment - here we analyze the list and classify it by priority.
Risk Mitigation - at this stage we determine how deeply we will test the risks.
Risk Management - here we decide how we will continue to work with them and pass them, to identify new risks.
Risks are identified and evaluated by a group of stakeholders during brainstorming sessions. The team should include business analysts or carriers of knowledge about the system from the customer, developers, manager or project manager, architects, QA-specialists. In order to identify and assess risks, we involve, among other things, information security specialists, employees who work directly with the current system, a business analyst who is immersed in processes.
Consider a risk-based approach on the example of testing the Internet acquiring system.
Work with risks (on the example of the Internet acquiring system)
Highlight the following requirements:
D1.Providing 1000 simultaneous connections to the system per second.
D2. Security of transactions.
D3. Access to the transaction should only be available to the person making the transaction.
D4. Providing and maintaining standard SET (Secure Electronic Transaction).
As a product risk, we can highlight:
RP1. System crash with simultaneous connections.
RP2. Using SQL injection in the transaction process.
RP3. Access to someone else's transaction when changing parameters in the URL.
RP4. Data loss in the event of a connection failure with the bank at the time of the transaction.
RP5. The use of non-valid certificates while securing the SET (Secure Electronic Transaction) system.
As organizational risks:
RO1. The fall of the developed system due to the unavailability of external systems.
RO2. The presence of hard-to-reproduce cases that cannot be detected in the test environment.
Thus, each risk is logically derived from the requirements that are in the system, but not limited to them. Risks supplement the requirements and reveal additional cases that may arise during the work with the system.
Risks can be reduced or compensated depending on the project objectives and system requirements. The condition is accepted that the risk is covered by the test case at least once:
1. A set of test cases RP1-RP4 is prepared for each product risk with the condition that each requirement and each risk must be covered by at least one test case. Depending on the purpose of testing, the set of test cases is expanded to a certain level of detail.
2. For each organizational risk a list of measures is prepared that can reduce the impact of risk on the product being developed.
Risk Assessment and Management Techniques
There are several risk assessment and risk management techniques: lightweight methods (PRAM, PRISMA) and more formal methods (FMEA, FTA).
With the FMEA model, the project team identifies all the processes, components, system modules in which a system failure or error may occur that could lead to a deterioration in the quality of the product. During the brainstorm, the system risks are determined by three indicators: severity, priority, probability. Then the Risk Priority Number for each risk is calculated and the depth of testing is set depending on the indicators.
With the FTA model, all the reasons that can lead to defects in the business processes of the system are determined step by step. The analysis runs from the highest to the lowest level. The difference between FMEA and FTA is that the first approach focuses on the functionality of the system, and the second - on business processes.
In addition to these formal and heavyweight approaches, there are more flexible and informal ones. For example, PRAM (Pragmatic Analysis and Risk Management) and PRISMA (Product Risk Management) methods. They successfully and easily combine strategies based on risk and requirements. Basically, these approaches use requirement specifications as input, but stakeholders can participate in the process.
Lightweight risk analysis methods are very similar to formal ones. They focus on technical or commercial risks, weighing the likelihood of risk occurring and the factors influencing this likelihood.
However, the only two factors used by these methods are the probability of risk and its effect. In addition, simplified qualitative judgments and scales are used to make decisions in these approaches.
Our teams use flexible, lightweight methods and adapt the PRAM and PRISMA approaches to fit their goals.
How we work with risk based testing
For example, in one of the projects we identify project and product risks that may work. To do this, analysts, developers and QA are involved in the analysis - one representative from each team.
Formed risk table with products. They determine the estimated probability of risk triggering and its possible impact on the system on a five-point scale. In table 1 - the strongest influence, 5 - the weakest. Also for probability 1 - a high probability, 5 - a small probability.
As we continue to work with product risks
Next, for each of them, the risk coverage of the product is traced with test cases.
Select the following checks:
TC1. Load verification with more than 1000 system connections
TC2. Load check at 1000 system connections
TC3. Entering SQL injection at the time of the transaction
TC4. Entering SQL injection on successful transaction page, by substituting other data
TC5. Entering XSS (Cross-Site Scripting) scripts into available input fields when making a transaction
TC6. Imitation of breaking the connection with the Internet during the transaction
TC7. Exit transaction session at confirmation
TC8. Verify expired certificates when making a transaction
Thus, priority checks are TC2, TC4, TC5.
TC6 and TC8 have a high impact, but a lower probability, so the priority of checking them is lower, but checks are also required.
TC7 does not apply to any of the requirements, but extends the test to a negative scenario, which is possible with a stable system.
As we continue to work with the risks of the project
We also define actions for project risks, for which we assign additional measures and solutions.
At risk “RO2. The presence of hard-to-reproduce cases that cannot be detected in the test environment "can do the following:
- prepare a “pre-production” stand, which is as close as possible to the actual application environment, in conjunction with all external systems;
- write end-to-end test scripts that pass through all adjacent systems and provide transaction verification;
- after all priority checks have been carried out, use error-guessing techniques according to the basic requirements of the system and scenarios of additional checks in the role of “system cracker”
Backup plan
It is important to have an action plan in case one of the product or project risks works. Sometimes setting up a project backup system can save. It is not always possible to reduce the risk to a minimum level, but it should be possible to reduce at least its effects. On this topic was our post "Christmas story" : how can work the risk, the probability of which tends to zero.
For example, we must completely eliminate data loss during a transaction, but consider all cases too laborious. Therefore, you need to have ways to handle such cases. One of the options for safety net is the development of more detailed logging. This will provide a permanent point of rollback to the previous action, if in the process of making the transaction something went wrong.
Conclusion
Risk-based testing allows you to cover the most risky areas with test cases, thereby reducing their impact and likelihood of triggering. This is the most winning strategy for systems with complex business logic and a high cost of error. The solution is suitable for the banking sector, insurance companies, complex internal CRM-systems of medical profile. With the risk-based approach, we also work with project risks, thereby improving the overall process of testing and project management.
Source: https://habr.com/ru/post/443672/
All Articles