Future testing
Continuing to translate a series of notes by James Whittaker entitled "The Future of Testing". This series in the original was published at the end of 2008, and in it James made a number of predictions as to what the work of testers will look like in the future, in 10-20 years. His predictions are largely based on the ideas that developed and continue to grow at Microsoft, where James was working at the time.
In the translation, we have collected all the notes in the series in one article, consisting of eight parts (the last four parts are presented in this post):
What can a computer program look like? Wouldn't it be helpful to have a visualization of a program that could be used during development and testing? At a glance it would be clear to us which parts of the product were still left unfinished. Dependencies, interfaces, and data would be easier to understand, and hopefully it would be easier to test them. At the very least, we could observe during development how the product grows and develops, and during testing to look at how input data is processed and how interaction with the environment takes place.
Other engineering disciplines have such visualizations. Take those who make cars. All people involved in the assembly process can see the car. They can see that the bumpers or the steering wheel are not installed on the car. They can observe the assembly along the entire length of the conveyor, from the empty body to the full-featured product, ready to be shipped to the dealer. How much is left to complete? Well, twenty meters to the end of the line!
')
The fact that all participants in the car assembly process have a common vision of the product is very useful. They use common terms that everyone can understand, since every part, every connection, every interface is where it should be, and then when it should be there.
Unfortunately, this is not our world. We are constantly tormented by the questions “how much is left to complete?”, “What tasks have not been accomplished?”. This is a problem that 21st century testers will solve.
Architects and developers are already engaged in it. Visual Studio contains a large number of diagrams and visualizations - from sequence diagrams to dependency graphs. Testers also solve this problem. Inside the walls of the “empire”, there are already visualization systems for viewing code changes on the Xbox (objects whose code has changed, are highlighted in green when rendering, and after testing the backlight disappears) to determine insufficiently thoroughly tested complex sections of Windows code (heat maps showing the coverage ratio code and complexity of the code performed in three-dimensional space, can detect problem areas). These visualizations look impressive, they are beautiful and allow testers to determine at a glance what they need testing.
We need more similar visualizations, but this task needs to be approached carefully. We cannot simply accept diagrams provided by UML and modeling groups. These visualizations are designed to solve other problems that may or may not coincide with those encountered in ours. Many of the existing visualizations are designed to help architects or developers, but their needs are different from ours. We need to think about it from the standpoint of testers. We need visualizations that associate requirements with code, tests with interfaces, changes in code with a user interface, code coverage with controls. Isn't it great when you launch a test application and see that the controls “glow” with an intensity that reflects the test coverage or the number of tests in which they were involved? Or would it be nice to see an animated graph of network usage or real-time interaction with the database? Why can't we see network traffic or SQL queries right away? We do not see much of what is hidden inside our application, and it's time to pull it to the surface and use it to improve the quality of the program.
This task can be solved in the near future, and many smart people are working on it.
This is testing programs in bright colors.
A couple of months ago, I attended a lecture given by one of the Leading Microsoft Specialists (however, it may have been an Outstanding Engineer, I don’t understand very well the difference between them). Like all our Leading Specialists, he was incredibly clever and when he presented projects to develop several new products that he did with his colleagues, something came down to me.
Apparently, I could not keep it in myself, and I had an expression on my face resembling that which sometimes happens with people with kidney stones. The Leading Specialist noticed this (just like the girl who was sitting next to me, but I don’t want to talk about it) and after the lecture approached me. This is how our conversation began:
- James - (well, he knew my name!) - I see that you have any doubts about my project or product, about which I told. I would like to hear your opinion.
- No, no problems with your product or with the project. My doubts concern you .
- Sorry?
“People like you scare me,” I said. “You spend all your time on dreams of new opportunities and ways of using, you design interfaces and protocols.” You are an important person, and people listen to you and do what you dream. But you do all this without having a clue about testing .
And at that moment he tried to do the right thing - turn to testers for help. He invited me to participate in the evaluation of product design. That was exactly what was expected of him.
But this is absolutely not what is needed.
Attracting a tester to design a product is better than not doing it at all. But not much better. Testers will focus on testability issues. Developers on the possibility of implementation. And who take into account both aspects? Who will decide who is right and when it is necessary to compromise? No one. Attracting a tester to the design provides only a slight improvement; attracting designers (and all other participants) to testing is the future.
Seriously, why do people who create software have so little testing experience? And why didn't we try to change that? Maybe we, testers, are too entrenched in their role and are now jealously guarding the keys to our intellectual realm? Is testing so full of secrets and riddles that developers can not comprehend it? Or maybe the developers are accustomed to dumping this “not very interesting” part of the work for us and now take it for granted?
Including testers in the development team does not help. Involving them in the early stages too. We have projects with a one-to-one ratio of developers and testers, but the products produced still do not look quite reliable. There are projects where the “worst” ratio of the number of developers to the number of testers who produce products is clearly better. I think in the future we will see that the separation of roles does not work. This division, on the contrary, contributes to the fact that testing is applied too late, and it is impossible to use its full potential.
The current culture of testing and separation of roles is flawed, and the right way is to combine roles. Quality should be the work of everyone. Think of it in Tolkien's terms: one role to rule over all!
Imagine a world in which every participant in the development process has knowledge of testing. The architect knows how to test, the designer knows this, the developer also does not lag behind, and they all apply this knowledge in everything they do, constantly and consistently. This does not eliminate the need for the existence of dedicated testers - someone has to present an independent opinion, it will simply lead to better testing. If every decision made at any of the development stages is subjected to the correct checks, then the final testing will be able to achieve the level of thoroughness that we can only dream of right now. If each project participant understands testing, imagine what several dedicated testers can accomplish!
For this utopia to become a reality, huge changes in the testing culture will be required. Testing should be included in the programs of all educational institutions graduating programmers. As the developer grows, the training in testing should not stop, but, on the contrary, should become more powerful and sophisticated. We need to reach a level where everyone interested in the project understands testing, and they have no choice but to apply the principles of testing in everything they do . One day, tools will appear to support this. One day we will be at the turn, when it’s just not possible to write a program that cannot be tested, and not because some cool tester achieved this, but because each project participant worked on it.
Testing is too important a role to “lag behind” the development process. Design decisions that are important from the point of view of testing are made at an early stage, and this is when testing should begin. You can not also give testing in the hands of the only selected role responsible for quality. On the contrary, we need such cultural changes that will make the quality of each work, and the principles of testing will penetrate into everything we do.
In the modern world, testers mostly occupy the last line of defense, but this is often not recognized when evaluating work and while distributing bonuses. If we find a serious bug - well, so we should have found it, so it was expected. If we miss a bug, they ask us questions. Testers are often referred to as such - ignored if you do, and scolded, if not.
But everything changes, and this too will change soon, because it must change. My friend Roger Sherman (Microsoft’s first corporate testing director) describes this change by comparing testing with a caterpillar that becomes a butterfly. According to Roger, the butterfly that should come out of testing is design.
And I can not disagree with him. If testing and testing techniques are applied at earlier stages of the development process, the work of testers will be more similar to designing programs than to verification. We will focus more on developing a quality strategy for all software artifacts, and not just for executable code. We will spend more time identifying the need for testing than doing test cases. We will monitor and evaluate automation, not create and debug it. We will spend more time analyzing the state of ready-made tests than creating new ones. We will become designers, and our work will rise to a higher level of abstraction and shift closer to the beginning of the life cycle.
At Microsoft, this role is played by test architects, and I think most of the testing work is moving in this direction. If you have read the previous six parts, then you understand what changes are required so that this design-oriented role can come out on top.
It looks like a bright future, but there is definitely a fly in the ointment. The source of the problem is in the types of errors and testing methods that we successfully use now. It would not be an exaggeration to say that we are better looking for structural errors (crashes, hangups and defects associated with the program itself, with its internal structure, rather than functionality) than with business logic errors. But in the future, which I am talking about, there will be many technological solutions to eliminate structural errors. This will free testers to work with business logic errors, and this is such a category of problems with which, I think, in general, our entire industry does not know how to cope in an organized or systematic way.
Finding business logic errors means that we need to understand the business logic itself. Understanding business logic means closer interaction with customers and competitors, it means immersion in the area of ​​knowledge in which our software works, it means not only engaging in the early stages of the development cycle, but also participating in work on prototypes, requirements, usability and so on, as we have never done before.
Working in the initial stages of the development cycle is very difficult, and testers do not have relevant experience. In order to start successfully, we must be ready to meet these new problems, we must be ready to master new ways of thinking, new approaches to users and to quality.
The work at the beginning of the pipeline is definitely different from what we are doing now, and this is the place where many, many testers will find their place as the future becomes present.
This is the final part of my predictions about the future of testing. Hope you liked the previous ones. And finally, I saved one of my possibly most contradictory predictions: in the future we will supply test code with software products and will be able to execute it remotely. I already anticipate hacker smiles and resentment of privacy advocates, but I will come back to this a little later.
I was in the Windows group when Vista was released, and I remember how I showed it to my eight-year-old son one evening at home. He plays a lot (and it works, if you can believe it) on the computer, and he really liked the Aero interface, the cute gadgets on the sidebar, and the speed with which his favorite games worked (at the time it was Line Rider and Zoo Tycoon ), really impressed him. I also thought: “What a pity that he does not maintain a sectoral blog,” but I digress from the topic.
So, at the end he asked me a question that awe every tester: “Dad, what did you do here?”
I fell silent, which is rather unusual for me, and muttered something unintelligible. How do you explain to an eight-year-old child that you worked on something for several months (I just started working at Microsoft and joined the work on Vista at the very end), but didn’t actually do anything? I tried to get by with standard answers to this terrible question (exclamation marks are required, they help me convince myself that there is some truth in my words):
“I worked to make it better!”
“That it works the way it works ... thanks to me!”
“If not for us, testers, it would be a threat to society!”
The last of these statements I particularly like. However, they all sound equally unconvincing. How can it be that I have been working on the product for so long, and I can’t demonstrate anything as a contribution, except for the absence of some errors.
It seems to me that at that very moment I had this idea: the test code should come with the program, it should continue to exist after the release and do its work in the absence of testers. This is not a pathetic attempt to give me and my colleagues the opportunity to point out something for bragging, but a way to provide ongoing testing and diagnostics. Let's face it, at the time of release we can never say that testing has been fully completed, so why should we stop?
We already do some of this. The Watson technology (you probably know the “send / not send error report” message in Windows applications) included in Windows allows us to collect error data as soon as they occur during operation. The next logical step should be the ability to do something with the information received.
Watson currently collects error data and captures relevant debug information. Then some poor fellow developer should get through all this data and figure out how to fix it with Windows updates. In 2004 it was a revolutionary idea, and it is still relevant. In a few years, this will probably be considered an outdated approach.
What if that same poor fellow can run additional tests and take full advantage of the test infrastructure that existed before the product was released? What if he can install fixes and run regression testing in the environment in which the failure occurred? What if he can install updates into the operational environment and ask the application to perform regression testing himself?
He will not be a poor fellow, that's for sure. But since testing and test artifacts are lost after final assembly and product delivery, this is now impossible.
To achieve such capabilities, the application must keep the pre-test data and always have it with you. And this means that the possibility of self-testing will be one of the main features of the software of the future. Our job will be to find a way to incorporate our “testing magic” into the product itself. And our reward will be the joy of how the eyes of our children shine when they see that the most first-class feature of the product is the one we designed!
Oh yeah, about hackers and privacy advocates: do not be afraid! Hugh Thompson and I had long warned about the inclusion of a test code in the delivery (see chapter Attack No. 10 in the book How to Crack Software Protection). Since we know how to hack, we have the chance to do it right.
Conclusion
Colleagues, even if you disagree with the ideas expressed by James - think about what will happen to testing not in a year, not in two, but in half a century. You need to start building this future now. And we will build it with you, so it will be the way we make it.
Those who want to get acquainted with a more detailed version of the predictions of James Whittaker can watch and listen to the recording of his speech at the GTAC conference , as well as the webinar recording organized by uTest .
In the translation, we have collected all the notes in the series in one article, consisting of eight parts (the last four parts are presented in this post):
- Testosourcing
- Virtualization
- Information
- Move testing to top
- Visualization
- Culture testing
- Testers in the role of designers
- Testing after release
5. Visualization
What can a computer program look like? Wouldn't it be helpful to have a visualization of a program that could be used during development and testing? At a glance it would be clear to us which parts of the product were still left unfinished. Dependencies, interfaces, and data would be easier to understand, and hopefully it would be easier to test them. At the very least, we could observe during development how the product grows and develops, and during testing to look at how input data is processed and how interaction with the environment takes place.
Other engineering disciplines have such visualizations. Take those who make cars. All people involved in the assembly process can see the car. They can see that the bumpers or the steering wheel are not installed on the car. They can observe the assembly along the entire length of the conveyor, from the empty body to the full-featured product, ready to be shipped to the dealer. How much is left to complete? Well, twenty meters to the end of the line!
')
The fact that all participants in the car assembly process have a common vision of the product is very useful. They use common terms that everyone can understand, since every part, every connection, every interface is where it should be, and then when it should be there.
Unfortunately, this is not our world. We are constantly tormented by the questions “how much is left to complete?”, “What tasks have not been accomplished?”. This is a problem that 21st century testers will solve.
Architects and developers are already engaged in it. Visual Studio contains a large number of diagrams and visualizations - from sequence diagrams to dependency graphs. Testers also solve this problem. Inside the walls of the “empire”, there are already visualization systems for viewing code changes on the Xbox (objects whose code has changed, are highlighted in green when rendering, and after testing the backlight disappears) to determine insufficiently thoroughly tested complex sections of Windows code (heat maps showing the coverage ratio code and complexity of the code performed in three-dimensional space, can detect problem areas). These visualizations look impressive, they are beautiful and allow testers to determine at a glance what they need testing.
We need more similar visualizations, but this task needs to be approached carefully. We cannot simply accept diagrams provided by UML and modeling groups. These visualizations are designed to solve other problems that may or may not coincide with those encountered in ours. Many of the existing visualizations are designed to help architects or developers, but their needs are different from ours. We need to think about it from the standpoint of testers. We need visualizations that associate requirements with code, tests with interfaces, changes in code with a user interface, code coverage with controls. Isn't it great when you launch a test application and see that the controls “glow” with an intensity that reflects the test coverage or the number of tests in which they were involved? Or would it be nice to see an animated graph of network usage or real-time interaction with the database? Why can't we see network traffic or SQL queries right away? We do not see much of what is hidden inside our application, and it's time to pull it to the surface and use it to improve the quality of the program.
This task can be solved in the near future, and many smart people are working on it.
This is testing programs in bright colors.
6. Culture testing
A couple of months ago, I attended a lecture given by one of the Leading Microsoft Specialists (however, it may have been an Outstanding Engineer, I don’t understand very well the difference between them). Like all our Leading Specialists, he was incredibly clever and when he presented projects to develop several new products that he did with his colleagues, something came down to me.
Apparently, I could not keep it in myself, and I had an expression on my face resembling that which sometimes happens with people with kidney stones. The Leading Specialist noticed this (just like the girl who was sitting next to me, but I don’t want to talk about it) and after the lecture approached me. This is how our conversation began:
- James - (well, he knew my name!) - I see that you have any doubts about my project or product, about which I told. I would like to hear your opinion.
- No, no problems with your product or with the project. My doubts concern you .
- Sorry?
“People like you scare me,” I said. “You spend all your time on dreams of new opportunities and ways of using, you design interfaces and protocols.” You are an important person, and people listen to you and do what you dream. But you do all this without having a clue about testing .
And at that moment he tried to do the right thing - turn to testers for help. He invited me to participate in the evaluation of product design. That was exactly what was expected of him.
But this is absolutely not what is needed.
Attracting a tester to design a product is better than not doing it at all. But not much better. Testers will focus on testability issues. Developers on the possibility of implementation. And who take into account both aspects? Who will decide who is right and when it is necessary to compromise? No one. Attracting a tester to the design provides only a slight improvement; attracting designers (and all other participants) to testing is the future.
Seriously, why do people who create software have so little testing experience? And why didn't we try to change that? Maybe we, testers, are too entrenched in their role and are now jealously guarding the keys to our intellectual realm? Is testing so full of secrets and riddles that developers can not comprehend it? Or maybe the developers are accustomed to dumping this “not very interesting” part of the work for us and now take it for granted?
Including testers in the development team does not help. Involving them in the early stages too. We have projects with a one-to-one ratio of developers and testers, but the products produced still do not look quite reliable. There are projects where the “worst” ratio of the number of developers to the number of testers who produce products is clearly better. I think in the future we will see that the separation of roles does not work. This division, on the contrary, contributes to the fact that testing is applied too late, and it is impossible to use its full potential.
The current culture of testing and separation of roles is flawed, and the right way is to combine roles. Quality should be the work of everyone. Think of it in Tolkien's terms: one role to rule over all!
Imagine a world in which every participant in the development process has knowledge of testing. The architect knows how to test, the designer knows this, the developer also does not lag behind, and they all apply this knowledge in everything they do, constantly and consistently. This does not eliminate the need for the existence of dedicated testers - someone has to present an independent opinion, it will simply lead to better testing. If every decision made at any of the development stages is subjected to the correct checks, then the final testing will be able to achieve the level of thoroughness that we can only dream of right now. If each project participant understands testing, imagine what several dedicated testers can accomplish!
For this utopia to become a reality, huge changes in the testing culture will be required. Testing should be included in the programs of all educational institutions graduating programmers. As the developer grows, the training in testing should not stop, but, on the contrary, should become more powerful and sophisticated. We need to reach a level where everyone interested in the project understands testing, and they have no choice but to apply the principles of testing in everything they do . One day, tools will appear to support this. One day we will be at the turn, when it’s just not possible to write a program that cannot be tested, and not because some cool tester achieved this, but because each project participant worked on it.
Testing is too important a role to “lag behind” the development process. Design decisions that are important from the point of view of testing are made at an early stage, and this is when testing should begin. You can not also give testing in the hands of the only selected role responsible for quality. On the contrary, we need such cultural changes that will make the quality of each work, and the principles of testing will penetrate into everything we do.
7. Testers in the role of designers
In the modern world, testers mostly occupy the last line of defense, but this is often not recognized when evaluating work and while distributing bonuses. If we find a serious bug - well, so we should have found it, so it was expected. If we miss a bug, they ask us questions. Testers are often referred to as such - ignored if you do, and scolded, if not.
But everything changes, and this too will change soon, because it must change. My friend Roger Sherman (Microsoft’s first corporate testing director) describes this change by comparing testing with a caterpillar that becomes a butterfly. According to Roger, the butterfly that should come out of testing is design.
And I can not disagree with him. If testing and testing techniques are applied at earlier stages of the development process, the work of testers will be more similar to designing programs than to verification. We will focus more on developing a quality strategy for all software artifacts, and not just for executable code. We will spend more time identifying the need for testing than doing test cases. We will monitor and evaluate automation, not create and debug it. We will spend more time analyzing the state of ready-made tests than creating new ones. We will become designers, and our work will rise to a higher level of abstraction and shift closer to the beginning of the life cycle.
At Microsoft, this role is played by test architects, and I think most of the testing work is moving in this direction. If you have read the previous six parts, then you understand what changes are required so that this design-oriented role can come out on top.
It looks like a bright future, but there is definitely a fly in the ointment. The source of the problem is in the types of errors and testing methods that we successfully use now. It would not be an exaggeration to say that we are better looking for structural errors (crashes, hangups and defects associated with the program itself, with its internal structure, rather than functionality) than with business logic errors. But in the future, which I am talking about, there will be many technological solutions to eliminate structural errors. This will free testers to work with business logic errors, and this is such a category of problems with which, I think, in general, our entire industry does not know how to cope in an organized or systematic way.
Finding business logic errors means that we need to understand the business logic itself. Understanding business logic means closer interaction with customers and competitors, it means immersion in the area of ​​knowledge in which our software works, it means not only engaging in the early stages of the development cycle, but also participating in work on prototypes, requirements, usability and so on, as we have never done before.
Working in the initial stages of the development cycle is very difficult, and testers do not have relevant experience. In order to start successfully, we must be ready to meet these new problems, we must be ready to master new ways of thinking, new approaches to users and to quality.
The work at the beginning of the pipeline is definitely different from what we are doing now, and this is the place where many, many testers will find their place as the future becomes present.
8. Testing after release
This is the final part of my predictions about the future of testing. Hope you liked the previous ones. And finally, I saved one of my possibly most contradictory predictions: in the future we will supply test code with software products and will be able to execute it remotely. I already anticipate hacker smiles and resentment of privacy advocates, but I will come back to this a little later.
I was in the Windows group when Vista was released, and I remember how I showed it to my eight-year-old son one evening at home. He plays a lot (and it works, if you can believe it) on the computer, and he really liked the Aero interface, the cute gadgets on the sidebar, and the speed with which his favorite games worked (at the time it was Line Rider and Zoo Tycoon ), really impressed him. I also thought: “What a pity that he does not maintain a sectoral blog,” but I digress from the topic.
So, at the end he asked me a question that awe every tester: “Dad, what did you do here?”
I fell silent, which is rather unusual for me, and muttered something unintelligible. How do you explain to an eight-year-old child that you worked on something for several months (I just started working at Microsoft and joined the work on Vista at the very end), but didn’t actually do anything? I tried to get by with standard answers to this terrible question (exclamation marks are required, they help me convince myself that there is some truth in my words):
“I worked to make it better!”
“That it works the way it works ... thanks to me!”
“If not for us, testers, it would be a threat to society!”
The last of these statements I particularly like. However, they all sound equally unconvincing. How can it be that I have been working on the product for so long, and I can’t demonstrate anything as a contribution, except for the absence of some errors.
It seems to me that at that very moment I had this idea: the test code should come with the program, it should continue to exist after the release and do its work in the absence of testers. This is not a pathetic attempt to give me and my colleagues the opportunity to point out something for bragging, but a way to provide ongoing testing and diagnostics. Let's face it, at the time of release we can never say that testing has been fully completed, so why should we stop?
We already do some of this. The Watson technology (you probably know the “send / not send error report” message in Windows applications) included in Windows allows us to collect error data as soon as they occur during operation. The next logical step should be the ability to do something with the information received.
Watson currently collects error data and captures relevant debug information. Then some poor fellow developer should get through all this data and figure out how to fix it with Windows updates. In 2004 it was a revolutionary idea, and it is still relevant. In a few years, this will probably be considered an outdated approach.
What if that same poor fellow can run additional tests and take full advantage of the test infrastructure that existed before the product was released? What if he can install fixes and run regression testing in the environment in which the failure occurred? What if he can install updates into the operational environment and ask the application to perform regression testing himself?
He will not be a poor fellow, that's for sure. But since testing and test artifacts are lost after final assembly and product delivery, this is now impossible.
To achieve such capabilities, the application must keep the pre-test data and always have it with you. And this means that the possibility of self-testing will be one of the main features of the software of the future. Our job will be to find a way to incorporate our “testing magic” into the product itself. And our reward will be the joy of how the eyes of our children shine when they see that the most first-class feature of the product is the one we designed!
Oh yeah, about hackers and privacy advocates: do not be afraid! Hugh Thompson and I had long warned about the inclusion of a test code in the delivery (see chapter Attack No. 10 in the book How to Crack Software Protection). Since we know how to hack, we have the chance to do it right.
Conclusion
Colleagues, even if you disagree with the ideas expressed by James - think about what will happen to testing not in a year, not in two, but in half a century. You need to start building this future now. And we will build it with you, so it will be the way we make it.
Those who want to get acquainted with a more detailed version of the predictions of James Whittaker can watch and listen to the recording of his speech at the GTAC conference , as well as the webinar recording organized by uTest .
Source: https://habr.com/ru/post/83274/
All Articles