Learning from mistakes in the organization of quality control
Hi, Habr! My name is Ilya Kudinov, and I work as a QA-engineer at Badoo. Three years ago, I began to attend various IT conferences and talk about the processes and technologies used by us in quality control. And of course, after each report I talked with the audience, wondered how they worked. In this case, I was always motivated by reviews like, “We used to work like this, but after listening to your report, we saw how you can do better,” and even better when people don’t copy our techniques, but come up with something themselves, sometimes more interesting options. I have accumulated many such stories, and I want to share with you some of them (all names are fictitious, any coincidence with real people is an accident). Maybe some of this will help you see the direction of development of your own project - and this will be the biggest reward for me! Of course, I will be glad after that to listen to your stories - in the comments or personal messages.
First of all, let's make two points.
First: the tips from this article will not be suitable for all types and scales of development (for some, I am sure they will even be very harmful), therefore for representatives of the developers of aircraft software (as well as for those testing gurus to whom my advice seems obvious) This story will be just an interesting read. I primarily focus on fast-growing products with a tight schedule of releases. The most interesting will be the representatives of those companies where the QA-department exists only in the bud or does not exist at all.
Second: no, “sad” stories are not made up. And no, they may not be as terrible in action as I describe them, but I will always try to argue my opinion.
Visualization of thoughts and ideas will help these three comrades, get acquainted:
So let's start by identifying who we are, actually. QA-engineers? Testers? Testers?
A tester is a device used to measure electrical characteristics of a circuit. Do not call us so, we are offended.
Tester - a specialist involved in testing. Testing is an important and irreplaceable process when developing anything, but it includes only the process of placing the system in various situations in order to study its behavior and verify the tasks set for testing.
QA engineer - specialist in quality control. This concept includes both testing and, for example, follow-up monitoring, development and support of various autotests and other means of automation and process optimization, release engineering.
')
However, in order to simplify speech, the “tester” will also come down, but only if you say this with due respect.
I proudly say that I am a QA engineer. My work (and the work of my colleagues both in the company and in the field in general) is aimed at ensuring the highest attainable product quality - that in which all other parties to the project are interested in. That is why I am very sorry that in many companies my colleagues are often viewed as purely support personnel, whose role in the project is the smallest (cleaners at least dirty coffee mugs are removed by the developers).
In fact, the role of a tester in a project is ultimately just as important as the role of a developer. Bjarne Straustrup (dat. Bjarne Stroustrup) in his book “C ++ Programming Language” wrote: “A program that has not been tested does not work.” Why do you need programmers whose products will never work? The tester is not a developer slave, generously issuing tasks like “check while I lay out on the prod”. On the contrary, the developer and tester jointly achieve the goal - to release the product by the appointed date in the highest possible quality. This is what the quality control department is created for. But ... how to organize it?
QA-department
So, the company " Foliant " decided to engage in software development. She approached this case thoroughly, even ventured to create a QA-department! How many people need to recruit there, if we already have 12 developers? The classical approach suggests that the approximate labor ratio should be as follows: 1 QA-engineer for 3-4 developers . No sooner said than done! We hire three engineers and consider ourselves to be great fellows.
The project is starting to develop; more and more new demands are being made on the QA department. We have already matured enough to produce releases not just by exposing a gzip archive to production, but by a well-constructed and well-established deployment process. Who will do this? Give it to our specialist in the quality control department!
Functionality is becoming more and more - testing regression is more difficult. Does one of our testers have development experience? Fine! Let him now engaged in the development of autotests! Well we came up, right?
What did we get as a result of such a wonderful plan? Only one of the QA engineers is constantly engaged in testing the tasks of our twelve developers. I wonder why the testing queue began to grow and our well-organized releases are regularly delayed?
Lesson : when counting the number of specialists for the QA department, you need to take into account all the areas of activity that they will be engaged in besides direct testing, and hire new staff if necessary, and not distribute new responsibilities among existing ones. Isn't that common sense? Practice shows that not always.
See, another small company - " CHEBETALKA " ! And she has very big and serious plans. They start small: four developers, a couple of managers and one (very proud) tester. Time passes, the business goes uphill, the number of orders for development is increasing and increasing. Happy and satisfied with the result of the project managers announce the expansion of the state. To begin with, we will hire another product manager, let him generate more explosive ideas. Requests for new features are becoming more and more? Urgently recruiting new developers!
Wow, more users? More profit? Guys, we are going the right way - we are hiring more developers, let our project be filled with features! How is all this great! But what is this? Why are there more bugs? Why are users dissatisfied? We hired more people, why are we lagging behind schedules? Oh, our unfortunate tester, we completely forgot about him!
Lesson : the quality control department should be developed in parallel with the development department. Perhaps even ahead of schedule. It is difficult to deal with competitors and deliver innovative functionality to users if you cannot ensure the project quality.
But the company " TINDUKS " all processes have long been set. She has a rather big development department and a quite appropriate testing department. Developers and QA-engineers are sitting in different rooms, separated by a large hall, and unkindly look at each other through the cracks of ajar doors.
Having completed the task, the developer, with all the anger that is available when you click on the Assign button in Jira, throws the puzzle for testing. The tester looks incredulously at the puzzle, opens it with disgust, and after a few seconds with enviable envy returns the task for revision with an explanation: “You have a typographical error in the comments!” It would seem that with such zeal for work the quality should be at a height. Perhaps it is there. But we will never know, because with this approach, the task will get to production approximately ... never.
Lesson : Testing and development departments are not enemies of each other. They should not sit on opposite sides of the fortified areas, trying to dump the work on each other as soon as possible. As I have already said, their joint goal is the release of the task within the set time frame, and their duty is to unite efforts to achieve this goal. This does not mean that you need to turn a blind eye to mistakes: just after detecting a minor defect, it is not necessary to immediately send the task for revision, you can quite write it aside, check as much as possible, and send it with a complete list of errors to save mountains time at constantly recurring stages of development and testing.
It is worth noting here that in some cases a similar approach is still appropriate. In the "industrial" development (for example, with the support of software used on airplanes) quality and compliance with the processes can be set higher than speed, and rightly so. Probably, I would not want to fly on airplanes that are tested and developed by employees of small “ultramodern and fast-developing” startups.
QA process
And now the QA-department is organized. What will he do? That's right, quality control. But how will this process be arranged?
Any QA processes are always built around a balance between quality and speed. Absolute quality is unattainable (if you argue with me - I hope you develop airplanes) , it is also impossible to test tasks instantly. Where to find this balance? Here, each team can come up with their own decision. For some, this is Continious Integration , someone prefers strictly regulated and planned releases - and you can’t just spy on the processes set by your neighbors in the garage.
How are QA processes integrated into project development? Let's divide them into three phases: product design, development and testing. Two young startups approached this issue in completely different ways: BODUNA closely associated QA engineers with each of them, and POCHTI.RU left them only testing. Which of them is right? As they say, the truth can be somewhere in the middle.
What do we get with each new stage of the QA process? Quality. What do we lose while doing this? Speed. What are the quality control steps that can be sacrificed?
Testing? NOT.
Quality control during development? It sounds important. No, of course it is not necessary for the tester to sit at the shoulder of the developer and painfully pinch him every time he forgets to put a semicolon. There are many other ways to help developers meet a certain level of quality. Successfully configured version control system, various hooks and scripts to check the correctness of the code, writing unit tests (here, however, QA can only act as a guard with a whip and a carrot), a convenient code review system is all in the area of ​​quality control .
Product control design? Well ... In the "industrial" development there is a whole direction of quality control - testing and analysis of TK. This helps to avoid logical and architectural errors in the initial stages of development of each project - and it drastically removes the possibility of starting development.
Probably, in a young startup, these are unnecessary precautions. The high integration of various departments with each other will allow all project participants (both developers and testers) to see the details of the project, determine the roughness and convey to the management useful (and not so) ideas. In addition, in our world of dynamic projects and Agile-methodologies, TK often changes already during development (and even after release, eventually), so it makes sense to allow product managers to express themselves uncontrollably and only then make suggestions and corrections .
Lesson : quality control at each stage of project development has a positive effect on quality and catastrophically lowers the speed. It is necessary to build processes so that they meet the requirements for your project, and not to copy thoughtlessly other practices and articles from books (and from Habr, yes) .
See, the developer of the company " BUBL " has given his task for testing and is absolutely sure that he will see it again only in case of a return for revision. And what, only QA-engineers are involved in the testing phase? Not necessarily. Of course, if any ambiguity is found, you can immediately return the puzzle, but this may well be a common misunderstanding, and the task will hang in some intermediate statuses. Therefore (and not only therefore, of course), communication and interaction in the testing process is priceless. In case of detection of strange and incomprehensible moments, you can always sit down with the developer and try to figure it out. On the one hand, if this behavior is not an error, then you can understand the algorithm and avoid all delays with the transition status. And if this turns out to be an unexpected error, then such a study can help the developer to solve the problem on the move, and not get to the reopened task after a few days and try to figure out from scratch what was happening there.
Lesson : you should not return the puzzle at the first incomprehensible moment. Debugging with the developer is not only a useful and efficient process, but also often instructive (both participants can go deeper into what is happening) and fun activities.
In the company " NALINII " well-designed development process. Developers have a complete set of tools to optimize development processes and simplify the distribution of work. They use project management systems and version control systems: no one interferes with each other and all their work is always in integrity and security.
But their quality control department is not so rosy for a long time. Historically, releases for testing were sent to them as archives attached to mail. Well, someone came up with something like that, and everyone got used to it. They write their documentation on countless pieces of paper scattered throughout the department (some very clever engineers have gotten a real Google Doc collaboration, but they are very afraid that they will be caught sooner or later and forced to rewrite everything to pieces of paper). It is a pity that no one can take the initiative and try to combine the used technical means!
After all, it would be possible to use common repositories, common knowledge bases, integrate bug-tracking systems, build applications into each other and make everything so beautiful, automated and complete ... But no, unfortunately, the developers would have to break the usual flow and even (!) develop and customize something new. But no, letters are somehow more convenient ...
Lesson : you should try to integrate the processes and tools of the development and testing departments as much as possible. This will help both departments and the project as a whole. Sometimes it requires small sacrifices from everyone involved, but ultimately it can achieve much greater success.
Testers of the company “ Krupnozhestvo ” always work on strict test cases. Any freedom of action is forbidden - and God forbid someone will allow himself to miss at least one point in the test plan! The missing checkmark in the checklist is tantamount to a crime and is punishable by reading "War and Peace" out loud in front of the entire department. Individual engineers spent the whole day engaged exclusively in supporting this documentation, and each tester was required to make up cases for each of the functionalities affected by them. Of course, the quality was on top! At first ... It turned out that the production began to appear bugs. Production bugs, Carl! When checking the checklists, it turned out that this functionality was checked, and the responsible tester with the most innocent look actively nods his head: checked, checked! And then it turned out that some components of the product had not been tested for years simply because they had not found a place in the test documents. The head of the department was forced to write a report and an explanatory note regarding what is happening in four volumes with a table of contents.
It would probably be better if testers thought with their heads when working and did not follow strict instructions. Would have a common source of information, but not in the form of "What to do", but in the form of "How it works." Of course, this may adversely affect the speed of testing, because each time it will be necessary to investigate a feature or (God forbid!) Communicate with other testers and (!) Developers to understand what to check here.
Lesson : general documentation that allows you to determine what should be tested in a particular task is good. Detailed instructions and cases ... probably not very. (aircraft developers - forget what you read now. PLEASE!)
PRINTEL appreciates the division of labor. Each tester is tied to a particular functional and component - and he knows perfectly well what it makes sense to test in it. The quality and speed at the height, the company goes to success. And then one of the QA engineers wins the lottery and goes to live in the Canaries. The rest looked at each other and tried to figure out what he left behind him. Nobody understood anything, and tested somehow until a new engineer was found to replace the lucky one. Everyone sighed with relief ... Until it became clear that he had left in his dreams, he was a doctor and all his records were kept in a slightly incomprehensible handwriting. The development of the component was almost entirely up until the newcomer could not get used to it from scratch.
Lesson : it makes sense to have the practice of sharing knowledge within the QA department. You can periodically exchange tasks in order not to get used to one component (one of the bad consequences of such a "specialization" - "zamurlivayuscheysya" eye), you can hold internal seminars and lectures in which experts will tell about innovations in their area of ​​responsibility, interesting cases and technologies: This will lead not only to the interchangeability of specialists, but to the professional growth of each of them.
The KRABYRADA QA-engineer has finally finished work on a complex system task for which he spent more than one working day. He followed her in on the production, sighed freely and went to drink beer with friends. Is he doing the right thing? I do not think so.
Even if his company has a serious monitoring department that watches tens and hundreds of graphs and metrics around-the-clock with red eyes, it may take a long time to locate a sudden or out-of-time error or activity drop. It would be great if someone could help them or even just not allow them to find the reasons for this mess ...
But the same engineer today could drink only one pint of Guinness less, but he could save a lot of man hours (and, possibly, company money) if he spent some time studying the production behavior after the release. Examine the error logs, check the trends of those graphs that describe the state of the components affected in the task. Yes, sometimes bugs get to production. This may be due to the human factor, differences in the environment, or even just one of those millions of user stories that real users generate.
Lesson : QA-process does not stop immediately after the release of the production task. It always makes sense to follow the behavior of the new functional, and it is extremely important to have the tools that allow it.
Tester of the company " YABLOCHKO " for the first time in his life missed an annoying bug for production. Probably, he could have discovered it if he had spent a couple more hours on this, but this had already happened. Bug fixed, but he managed to affect users. The company is very dissatisfied with the tester. Developer features displeased tester. He receives a penalty and loses the prize, so that it will not save himself and later did not spare himself when testing. Has the company acted correctly? I think not.
QA-processes do not cancel the possibility of defects - they are aimed at reducing the probability of their occurrence. And all the people involved in the project participate in these processes. And the responsibility for defects equally falls on everyone: the leader who has not motivated enough and trained his employees; on the developer who made the mistake; on the tester who missed this error. Even for those engineers who covered this functionality with autotests - after all, their tests could catch this bug, but they could not! Should they be punished? Perhaps, but only with systemic punctures. It would be much more useful to conduct an in-depth study of the causes of the incident, to organize educational seminars and all possible events in order to reduce the likelihood of such defects in the future.
The lesson is that all parties involved in the project are to blame for the occurrence of defects; you should not blame only the tester for its occurrence.
Automation
The company " TSONNI " decided to start the development of automated tests. One of the testers has programming skills, and this task was given to him. He wrote tests, was pleased with his work and at some point covered almost the entire product with tests. Everyone was happy until they noticed that the tester who had gone to another project was not looking for a replacement.
The event turned into a trend, and the QA department began to melt. When the frightened engineers came running to the management, they answered with a smile: “We now have such wonderful auto-tests, why do we need boring out-of-date manual testers?” There was nothing to explain to them, and everything remained in their places. Things went well until more and more errors appeared on the production, and all of them were in the newest, not yet covered with tests functionality. The management realized its mistake, but it was already too late ... * loud dramatic music *
Lesson : Autotests are only an auxiliary means of quality control and in no way replace manual testing.
But in the company " SHANTSUNG " did not experience such a problem. Their test automation department was completely separated from the manual testing department. The “automators” even felt themselves to be representatives of some higher caste and condescendingly watched those whom they called “handbrake”. And this separation seemed to be convenient for everyone until problems began to arise. “Automators” have become so deeply immersed in the support of hundreds of their tests that they have lost any opportunity to develop something new (and comply with the KPIs, of course), and the “handbrake” had no idea what their colleagues were doing and stopped relying on tests, manually checking everything that was completely covered by tests (but nobody knew about it!), and this significantly reduced the benefit of the whole event.
Lesson : it is much better when all representatives of the QA-department work with auto-tests. Let not all of them be able to write a test from scratch, but make it so that every engineer can evaluate the status of tests at the moment, understand the reason for their failures, fix a simple error or cover a new simple case with a test.
Instead of an afterword
I hope that some of the things I wrote about verbally will be useful to you. Someone will help to find the roughness in the processes, others will tell a way out of the already studied problem. Even if not, maybe you even laughed at what I told you. Or at least over my naivety. The main thing is to make the world better even in something, even a little bit. True?
Source: https://habr.com/ru/post/301764/
All Articles