Critical analysis of old and synthesis of new knowledge will make Testing software science
The main goal of this problem-setting article is to refer to the theoretical foundations of software testing, i.e. clear construction of original basic concepts or definitions and search for meanings. The main problem that had to be dealt with when writing was to slide out of the presentation of theoretical aspects to the plane of practical methodology.
Definition 1. Software (software) is a set of hardware, system services and ancillary services that constitute a configured environment (infrastructure) with which application programs integrate, in aggregate, implement target processes, endowed with specified characteristics, rules of conduct, configured to solve specific tasks in the respective modes of operation, including interaction with the User.
Note_1. The specific software (created by the Developer for the Customer) is inextricably linked: a package of working documentation and, possibly, a resource supporting this software in working condition. In a common infrastructure, you can select the landscape in which this software is deployed and operates.
')
Definition_2. Testing is a way to collect the measured characteristics of an Object, including information about its ability to change its state depending on the initial conditions and effects.
Definition_3. The object of testing is a whole or a certain subset of its components, up to logically or physically isolated primitives (decomposition elements, see Note_2), to which a given subset of requirements is associated.
Definition_4. A requirement is a part or combination of static and dynamic attributes and properties, which are summarized in the criteria for assessing the ability of an Object to perform a specific purpose in appropriate operating conditions.
Note_2. Architecturally, software is divided into functional modules, which form individual subsystems that are combined into systems and software systems that are physically (or virtually) running on distributed equipment. Usually, decomposition takes place here, when a set of tasks is solved to achieve a specific goal (s), the result of each is obtained on the basis of a specific algorithm in which you can select branches for which you need to execute commands, control the flow of execution and initiate execution of operations which, according to access, are able to process / modify entities. To the mentioned goals we will add the provision of flexible, fault-tolerant, testable construction with minimization of changes when changing / adding requirements.
Definition_5. Testability is the presence in the Object (see Note_3) of such important properties as controllability, observability and coverage.
Definition_6. Manageability is a property of an object, when it can be transferred from a given initial state to a desired final state, which is determined by the output interface, through an input interface.
Definition_7. Observability is the property of an Object, when by its obtained final state at its output and known effects that were previously submitted to the input, you can determine its previous state, i.e. You can judge the causal relationships of the events between these states.
Definition_8. Coverage is the property of an Object to be a verified finite set of Equivalent tests.
Definition_9. The equivalent test is a way to verify an Object, when from a set of all input actions for a specific initial state of an Object, you can select a subset, with each input action from which the same final state will be obtained. In this subset, the only option is fixed that defines one exponential check.
Note_3. The object of software testing, for example, may be documentation with requirements, text with the output of final formulas or algorithms of an applied solution, text of program code. If the object of software testing is a workflow with the context of its execution, then this object must be testable.
Definition_10. A defect is a fact of deviation of a specific characteristic / behavior of the developed software from the requirements, their descriptions in the working documentation; objective incompatibility of the solution with the subject area; deviation from existing local Standards and Laws; non-compliance with corporate software development requirements; subjective remarks concerning negative precedents (see Note4).
Note_4. It is almost impossible to describe in detail in the working documentation everything that software should and should not do. Therefore, at different stages of the life cycle there are observations / situations that are not clear how to interpret. When such a thing is detected for the first time - it is an Incident (incident), and after when a decision is made concurrently whether it is a defect or not, then this “something” becomes a Precedent (precedent) for identifying repeated such cases. So the practice of precedents is formed, covering the “holes” of documentation.
Note_5. A feature is a revealed undocumented manifestation in software operation, which, due to the practice of precedents, is not a Defect. By its nature, it is more often a useless artifact that does not irritate the User, which does not make sense to offer for use.
Definition_11. The task of software testing is an artifact (a regulation element) containing: test objectives; the necessary stages of the decision and the allocated time and resources; a brief description of the test landscape and data access to its components (with reference to the Task for its preparation for the Administrator); description of the object of testing with the requirements to be checked (with links to the relevant sections of the working documentation and Tasks for the Developers, the fulfillment of which will be checked); conditions of the unsatisfactory state of the Test Object, under which the solution of the problem is interrupted; the required format for reporting on the results of the work performed, including the necessary metrics and statistics that need to be collected / provided.
Definition_12. Software testing is the process of solving the assigned Task for software testing. Its beginning is the moment of receiving the task for testing. The end of testing is determined by the timing of the task (or conditions). This process consists of the following stages: examination (see Note_6); planning (see Note_7); tests (including registration of Defects, Problems, suggestions for Improvements, see Definition_21..23); formalizing the results (drawing up a report and highlighting a subcoat for the regression, and for acceptance tests in the future).
Note_6. Testing is the stage of examination of the task and in-depth immersion in its context, resulting in the formation of a full range of what needs to be checked, an understanding of the methodology (what / where / when / how to do and observe) that are ranked by importance, risk and interdependence , an idea of ​​a suitable tool with which you can test or measure specific characteristics or collect statistics. Note: Importance and Risks (see Definition_19, 20) are attributes whose ratings should be close to objective, they are agreed with the competent specialists (or provided by them).
Note_7. Planning in testing is a stage of development of strategy and tactics of carrying out tests. As a result, test methods and technologies are formed, and on their basis, a coordinated program indicating the roles and actions of test participants (or connected automation tools) at the corresponding event-time intervals for preparing / conducting a test and recording its results. Provides optimal coverage with tests of the Test Object, which must be implemented with available resources within the specified time frame, with minimization of uncontrollable factors.
Note_8. In parallel with the stages of the survey, planning, and also for the purpose of replacing them, research testing can be performed by a specialist with a high level of competence of the researcher, with the ability to quickly build an analytical base, not limited to working documentation; deepen knowledge of software-related technologies and domain; creating or owning supporting tools; in one language interacting with Business, Developer, System Administrator.
Definition_13. Research testing is a creative process consisting in generating creative or most significant, from the point of view of a problem to be solved, questions about cause-effect relationships in software operation and in implementing search for answers to them in experiments aimed, on the one hand, at identifying unobvious logical deadlocks and conditions under which software operation problems occur or security is violated, on the other hand, to accumulate and synthesize knowledge about the Object of testing and improve the technologies used s, allowing it to go more in-depth study.
Note_9. Defects are an important characteristic for analyzing both the state of the software and the level of development at which work was done on its creation. An explicit defect is one that is registered but not fixed (otherwise Corrected). Note, it can remain never eliminated or be in principle not eliminated. A latent defect is one that was not detected during the tests (possibly due to time and resource constraints or the insufficient level of development at which the testing was performed) and may manifest itself during direct operation.
Definition_14. Error (error) - incorrectness of thoughts, actions, orders of a specific specialist in the framework of the statement / execution of the task, the result of which will not allow to solve this problem correctly (to achieve the goal).
Note_10. The error can be attributed not only to the Software Provider, but also to the Customer or the User.
Definition_15. A bug is a fact of unintentionally incorrectly executed coding of the program, enclosed in its text, causing (after assembling and deploying the program in the working landscape) deviation from the requirements for software operation.
Note_11. Paradoxical and very undesirable situation, when two Bugs in the program can compensate each other without being identified.
Note_12. A tester using the “White Box” technique finds Bugs, while testing with a “Black Box” detects Defects.
Definition_16. Problem (problem) - a situation where at the current stage a specialist does not have the ability to effectively or, in principle, perform his part of the work due to the fact that the previous (or earlier in the software life cycle along the processing chain) made an Error.
Note_13. The bug can be borrowed from third-party auxiliary frameworks or be the result of an Error made directly by the Developer, or a result of Problems previously created by other specialists.
Definition_17. Accident (accident) is a fact of hardware failure or serious violations of the system or application software (for example, the license or certificate has expired) on the Server side or network / connection or on the User’s side, i.e. situation in which the software is not objectively workable.
Definition_18. Failure is a fact of global (failure - on the Server side) or local (fault - for an individual User) temporary lack of capacity / resources, i.e. the situation in which the normal operation of the software is interrupted, then it self-restores, probably with the assistance of a support resource.
Note_14. In the event of a Crash or Global Crash, indeed, the software may not be objectively operational. But! The superposition principle allows you to select modules that can be backed up or use special fault tolerance technologies.
Note_15. Different prerequisites give rise to Errors, which cause Problems and Bugs, which, together with additional (un) predictable factors, are the causes of Defects (explicit and implicit) that determine the state of the software, analysis of which allows to predict where and to what extent the software is still insufficiently developed and tested. . Based on the statement that the ideal software does not exist, the danger is obvious - to give it into operation in such a state that instead of achieving the objectives set by the Customer, it will cause damage to it (the Users). An assessment of the state of the software can be generated on the basis of the analysis of the reports of the corresponding test tasks. This reveals the importance of how fully and clearly it is necessary to set / formulate these tasks and ensure their correct execution, preparing the necessary information base for the chosen methodology of the Theory of Decision Making under Risk and Uncertainty.
Definition_19. Risk (risk) is a quantitative assessment of the consequences of choosing an alternative solution in the face of uncertainty.
Definition_20. The importance of an entity is a quantitative assessment of its significance or the degree to which other entities depend on it in the context of the specifics of its implementation or the circumstances of its operation.
Definition_21. Bug description (bug-report) is a registered message, the main content of which is information about the essence of the deviation from the standard observed by the Test object (expected implementation) and a clear description of the conditions and actions for its re-observation, if possible, supplemented by Bug localization in the program.
Definition_22. Problem Description (issue) is a registered message for the escalation of identified Incidents and Errors that impede the performance of effective work related to software development and testing, as well as suggestions for solving them.
Definition_23. Description Enhancement is a registered message containing a proposal for the need to modify / supplement the software (and, accordingly, the requirements in the working documentation) for its convenient and efficient operation / administration.
Note_16. Details of the format and content for the entities in Definitions_21..23 are regulated separately.
I admit, I do not like to argue, but, if in essence, I am ready. Obviously, here I drew attention only to those concepts that I have a special view on or often notice their incorrect use. I will be especially grateful to readers who do not like the article for their opinions and arguments. Again, in this article I tried to find points of scientific growth in the field of software testing, its philosophy and aesthetics.
Definition 1. Software (software) is a set of hardware, system services and ancillary services that constitute a configured environment (infrastructure) with which application programs integrate, in aggregate, implement target processes, endowed with specified characteristics, rules of conduct, configured to solve specific tasks in the respective modes of operation, including interaction with the User.
Note_1. The specific software (created by the Developer for the Customer) is inextricably linked: a package of working documentation and, possibly, a resource supporting this software in working condition. In a common infrastructure, you can select the landscape in which this software is deployed and operates.
')
Definition_2. Testing is a way to collect the measured characteristics of an Object, including information about its ability to change its state depending on the initial conditions and effects.
Definition_3. The object of testing is a whole or a certain subset of its components, up to logically or physically isolated primitives (decomposition elements, see Note_2), to which a given subset of requirements is associated.
Definition_4. A requirement is a part or combination of static and dynamic attributes and properties, which are summarized in the criteria for assessing the ability of an Object to perform a specific purpose in appropriate operating conditions.
Note_2. Architecturally, software is divided into functional modules, which form individual subsystems that are combined into systems and software systems that are physically (or virtually) running on distributed equipment. Usually, decomposition takes place here, when a set of tasks is solved to achieve a specific goal (s), the result of each is obtained on the basis of a specific algorithm in which you can select branches for which you need to execute commands, control the flow of execution and initiate execution of operations which, according to access, are able to process / modify entities. To the mentioned goals we will add the provision of flexible, fault-tolerant, testable construction with minimization of changes when changing / adding requirements.
Definition_5. Testability is the presence in the Object (see Note_3) of such important properties as controllability, observability and coverage.
Definition_6. Manageability is a property of an object, when it can be transferred from a given initial state to a desired final state, which is determined by the output interface, through an input interface.
Definition_7. Observability is the property of an Object, when by its obtained final state at its output and known effects that were previously submitted to the input, you can determine its previous state, i.e. You can judge the causal relationships of the events between these states.
Definition_8. Coverage is the property of an Object to be a verified finite set of Equivalent tests.
Definition_9. The equivalent test is a way to verify an Object, when from a set of all input actions for a specific initial state of an Object, you can select a subset, with each input action from which the same final state will be obtained. In this subset, the only option is fixed that defines one exponential check.
Note_3. The object of software testing, for example, may be documentation with requirements, text with the output of final formulas or algorithms of an applied solution, text of program code. If the object of software testing is a workflow with the context of its execution, then this object must be testable.
Definition_10. A defect is a fact of deviation of a specific characteristic / behavior of the developed software from the requirements, their descriptions in the working documentation; objective incompatibility of the solution with the subject area; deviation from existing local Standards and Laws; non-compliance with corporate software development requirements; subjective remarks concerning negative precedents (see Note4).
Note_4. It is almost impossible to describe in detail in the working documentation everything that software should and should not do. Therefore, at different stages of the life cycle there are observations / situations that are not clear how to interpret. When such a thing is detected for the first time - it is an Incident (incident), and after when a decision is made concurrently whether it is a defect or not, then this “something” becomes a Precedent (precedent) for identifying repeated such cases. So the practice of precedents is formed, covering the “holes” of documentation.
Note_5. A feature is a revealed undocumented manifestation in software operation, which, due to the practice of precedents, is not a Defect. By its nature, it is more often a useless artifact that does not irritate the User, which does not make sense to offer for use.
Definition_11. The task of software testing is an artifact (a regulation element) containing: test objectives; the necessary stages of the decision and the allocated time and resources; a brief description of the test landscape and data access to its components (with reference to the Task for its preparation for the Administrator); description of the object of testing with the requirements to be checked (with links to the relevant sections of the working documentation and Tasks for the Developers, the fulfillment of which will be checked); conditions of the unsatisfactory state of the Test Object, under which the solution of the problem is interrupted; the required format for reporting on the results of the work performed, including the necessary metrics and statistics that need to be collected / provided.
Definition_12. Software testing is the process of solving the assigned Task for software testing. Its beginning is the moment of receiving the task for testing. The end of testing is determined by the timing of the task (or conditions). This process consists of the following stages: examination (see Note_6); planning (see Note_7); tests (including registration of Defects, Problems, suggestions for Improvements, see Definition_21..23); formalizing the results (drawing up a report and highlighting a subcoat for the regression, and for acceptance tests in the future).
Note_6. Testing is the stage of examination of the task and in-depth immersion in its context, resulting in the formation of a full range of what needs to be checked, an understanding of the methodology (what / where / when / how to do and observe) that are ranked by importance, risk and interdependence , an idea of ​​a suitable tool with which you can test or measure specific characteristics or collect statistics. Note: Importance and Risks (see Definition_19, 20) are attributes whose ratings should be close to objective, they are agreed with the competent specialists (or provided by them).
Note_7. Planning in testing is a stage of development of strategy and tactics of carrying out tests. As a result, test methods and technologies are formed, and on their basis, a coordinated program indicating the roles and actions of test participants (or connected automation tools) at the corresponding event-time intervals for preparing / conducting a test and recording its results. Provides optimal coverage with tests of the Test Object, which must be implemented with available resources within the specified time frame, with minimization of uncontrollable factors.
Note_8. In parallel with the stages of the survey, planning, and also for the purpose of replacing them, research testing can be performed by a specialist with a high level of competence of the researcher, with the ability to quickly build an analytical base, not limited to working documentation; deepen knowledge of software-related technologies and domain; creating or owning supporting tools; in one language interacting with Business, Developer, System Administrator.
Definition_13. Research testing is a creative process consisting in generating creative or most significant, from the point of view of a problem to be solved, questions about cause-effect relationships in software operation and in implementing search for answers to them in experiments aimed, on the one hand, at identifying unobvious logical deadlocks and conditions under which software operation problems occur or security is violated, on the other hand, to accumulate and synthesize knowledge about the Object of testing and improve the technologies used s, allowing it to go more in-depth study.
Note_9. Defects are an important characteristic for analyzing both the state of the software and the level of development at which work was done on its creation. An explicit defect is one that is registered but not fixed (otherwise Corrected). Note, it can remain never eliminated or be in principle not eliminated. A latent defect is one that was not detected during the tests (possibly due to time and resource constraints or the insufficient level of development at which the testing was performed) and may manifest itself during direct operation.
Definition_14. Error (error) - incorrectness of thoughts, actions, orders of a specific specialist in the framework of the statement / execution of the task, the result of which will not allow to solve this problem correctly (to achieve the goal).
Note_10. The error can be attributed not only to the Software Provider, but also to the Customer or the User.
Definition_15. A bug is a fact of unintentionally incorrectly executed coding of the program, enclosed in its text, causing (after assembling and deploying the program in the working landscape) deviation from the requirements for software operation.
Note_11. Paradoxical and very undesirable situation, when two Bugs in the program can compensate each other without being identified.
Note_12. A tester using the “White Box” technique finds Bugs, while testing with a “Black Box” detects Defects.
Definition_16. Problem (problem) - a situation where at the current stage a specialist does not have the ability to effectively or, in principle, perform his part of the work due to the fact that the previous (or earlier in the software life cycle along the processing chain) made an Error.
Note_13. The bug can be borrowed from third-party auxiliary frameworks or be the result of an Error made directly by the Developer, or a result of Problems previously created by other specialists.
Definition_17. Accident (accident) is a fact of hardware failure or serious violations of the system or application software (for example, the license or certificate has expired) on the Server side or network / connection or on the User’s side, i.e. situation in which the software is not objectively workable.
Definition_18. Failure is a fact of global (failure - on the Server side) or local (fault - for an individual User) temporary lack of capacity / resources, i.e. the situation in which the normal operation of the software is interrupted, then it self-restores, probably with the assistance of a support resource.
Note_14. In the event of a Crash or Global Crash, indeed, the software may not be objectively operational. But! The superposition principle allows you to select modules that can be backed up or use special fault tolerance technologies.
Note_15. Different prerequisites give rise to Errors, which cause Problems and Bugs, which, together with additional (un) predictable factors, are the causes of Defects (explicit and implicit) that determine the state of the software, analysis of which allows to predict where and to what extent the software is still insufficiently developed and tested. . Based on the statement that the ideal software does not exist, the danger is obvious - to give it into operation in such a state that instead of achieving the objectives set by the Customer, it will cause damage to it (the Users). An assessment of the state of the software can be generated on the basis of the analysis of the reports of the corresponding test tasks. This reveals the importance of how fully and clearly it is necessary to set / formulate these tasks and ensure their correct execution, preparing the necessary information base for the chosen methodology of the Theory of Decision Making under Risk and Uncertainty.
Definition_19. Risk (risk) is a quantitative assessment of the consequences of choosing an alternative solution in the face of uncertainty.
Definition_20. The importance of an entity is a quantitative assessment of its significance or the degree to which other entities depend on it in the context of the specifics of its implementation or the circumstances of its operation.
Definition_21. Bug description (bug-report) is a registered message, the main content of which is information about the essence of the deviation from the standard observed by the Test object (expected implementation) and a clear description of the conditions and actions for its re-observation, if possible, supplemented by Bug localization in the program.
Definition_22. Problem Description (issue) is a registered message for the escalation of identified Incidents and Errors that impede the performance of effective work related to software development and testing, as well as suggestions for solving them.
Definition_23. Description Enhancement is a registered message containing a proposal for the need to modify / supplement the software (and, accordingly, the requirements in the working documentation) for its convenient and efficient operation / administration.
Note_16. Details of the format and content for the entities in Definitions_21..23 are regulated separately.
I admit, I do not like to argue, but, if in essence, I am ready. Obviously, here I drew attention only to those concepts that I have a special view on or often notice their incorrect use. I will be especially grateful to readers who do not like the article for their opinions and arguments. Again, in this article I tried to find points of scientific growth in the field of software testing, its philosophy and aesthetics.
Source: https://habr.com/ru/post/337344/
All Articles