Quality guaranteed: What the game tester really looks like (Part 2)

The first part is here .

The old Westwood college ad became a bit of a joke in the world of video games. Two guys, comfortably sitting on the couch, are killed in the trash controllers, enthusiastically playing on the Sony PlayStation. A girl comes in and says, “Hey, guys, have you finished testing this game? I have one more here. ” "We just passed the third level, the schedule should be a little pull up," - says one of the guys. Then, turning to his friend, he smiles, as if he had just won the lottery: "I can not believe that we play games, and we are still paid for it." “I know,” answers the second. “And my mother used to say that my passion for video games would not lead to anything good.”
')
This is exactly how people have been imagining the lives of those who test computer games for a long time - not as a job for 5–9 hours a day, but as a dream of all teenagers. Who would not want to sit on a comfortable sofa and play games all day with a few interruptions to “pull up” the graphics in the third level?

Dining Disaster

Rob Hodgson, a video game tester, once checked out the early build of the multiplayer Fallen Earth game at lunchtime when game servers suddenly fell. Soon after, they failed again. The puzzled employee tried to reproduce the bug, but his attempts were not crowned with success - neither he nor any other team member could understand what caused this critical error.

“It all seemed very strange,” Rob told me. - We asked one programmer to check the logs during server hangs, but after some time he only shrugged, being unable to determine the problem. It all seemed as if someone had reached the limits of the game world, and our servers simply choked and died in an attempt to track the player. However, none of the QA-team did not test the limits of the world, and the designers went for lunch! ”

After some thought, Hodgson and other members of the team began to question other employees, and eventually they identified the source of the problem, putting the servers on both blades. “It turns out that the lunch break designers had a direct connection to this problem. - says Hodgson. - The fact is that one of the guys working in the design department put his horse on the “autopilot” and left for a snack. Sometimes she just found a tree in her path and stopped. But there were also days when she went into the vast expanses outside the gaming world, as a result of which the servers simply boiled hard. ”

***

Starting from Assassin's Creed Unity and ending with the Batman: Arkham Knight version for PC, it seems that modern games come with much more problems than before. Of course, it's easy to put all the blame on the testing department. And sometimes this is confirmed - in particular, several testers with extensive experience in the QA-sector told me that they had the opportunity to work with indifferent employees who carelessly treat their duties and perform them quite differently. Two "experienced" testers reported that they had repeatedly seen their colleagues smoking weed at the lunch break almost every day, one of my interlocutors noted that some even wore dark glasses in unsuccessful attempts to hide their hobbies (apparently, dilated pupils - translator comment).

At the same time, many testers say that they manage to find an overwhelming number (if not all) of the bugs with which modern games are crammed. The problem, in their opinion, is that no one corrects mistakes.

Most testers work using a priority setting system, where each bug is given priority on criticality, the importance of eliminating it. The bugs with the highest priorities include errors that cause a critical crash in the game (crash or freeze). All other glitches are ranked in different categories in accordance with the level of importance - it all depends on the position of the tester. As a rule, producers and programmers find time to eliminate critical errors, because with them the game will be difficult even to pass standard certification. But minor errors and “medium gravity” bugs are attached to the game, like a bath sheet, being victims of tight deadlines for the completion of the project and programmers who can do as much as they have been allotted for this time - and no more.

“We usually conclude that the risk involved in fixing a bug, as well as the time spent on solving a problem, is not worth it - especially when we find a mistake, doing something quite unusual in the game world or deliberately“ breaking ”the game "- says the tester, who worked on the popular role-playing action movie The Elder Scrolls V: Skyrim. The latter is widely known to gamers, being a project with an unprecedented scale and at the same time an unusually buggy game.

“Part of the bugs is marked as“ will not be fixed ”and“ fixed after release ”. The latter means: “It would be good to eliminate it, but right now you can do without it. Now, if people write about this bug on forums and other online resources, then maybe we will change its status. ” I think now in the gaming industry we are working with a much larger margin in time than, say, 20 years ago. Because now the developer / publisher can simply say: “Well, we took into account this annoying bug, but it can be fixed with the help of day-one-patch (the so-called“ first day patch ”, which is released on the day of the official release of the game - Translator).



Games are getting bigger and bigger, evolving both in terms of graphics and in terms of gameplay and game mechanics. As a result of this evolution, more and more bugs are generated - including those that are not at all easy to catch or fix. Game development studios cannot change the game release schedule simply because the product is filled with glitches. The release date of the product, dedicated to some holiday, will not change, no matter how raw the game may be - unless the publisher himself agrees to postpone the release date, and that will cost him dearly. Therefore, as several testers told me, during their work, they focus their efforts on catching precisely those bugs that will not allow them to pass a mandatory certification for release on the console. Typically, this behavior involves ignoring some of the other, often more significant problems in the game.

According to one specialist who tested the product for compliance with certain requirements for one major publisher, developers and publishers have independent QA teams. “We found a bug and sent as much information to developers as it allowed us to reproduce it. The development team determined whether this was a bug and then either canceled it or tried to fix it. We received an updated build with the specified number of corrected errors, then carefully checked whether the problem was really solved, and then proceeded to the next bug. From the player’s point of view, such an algorithm was a complete disappointment. The fact is that the bugs that spoil the game to the user are usually marked as “incorrect”, because they did not affect the release of the game. Developers could return to questions about eliminating such bugs later - if a large number of people expressed their dissatisfaction with one or another mistake that was noticed in the game.

Other testers say that because of tight deadlines, they had to perform only the necessary minimum of actions and find only those gaps that the game could be certified to eliminate, after which they could start printing discs and promise to release the “first day patch”.

“If there were deadlines for us, and there were still a lot of glitches in multiplayer, then we focused on testing the single player mode and other offline components of the product,” one of the testers wrote to me via email. “Console manufacturers can easily reject the game if it has serious problems with multiplayer, but they also allow us to send the game to print, provided that the company provides the first day patch to fix problems A, B and / or C. This policy is used because neither the developer nor the console maker is able to guarantee the receipt of the expected "patch" that fixes problems in the offline components of the game, because such modes do not require an Internet connection (a gamer can play in one night mode, without connecting to the network and the mother of the developers, not knowing about the patch output - note translator).. Therefore, we worked on eliminating those bugs that were related to offline content, polishing only this component of the game to successfully pass the certification procedure and realizing that there are mistakes in the multiplayer. And by the time of the release of the game, we tried to prepare a patch that should eliminate known bugs. Unfortunately, this is a common practice that has often been used in the context of unrealistic project completion times and the resulting excessive workload. Usually, it is precisely because of this behavior (and not the low qualifications of the development team and testers) that the game has many bugs. ”

Sometimes bugs are intentionally left in the game for one reason or another, incomprehensible to outsiders. According to one tester who worked for the American division of the legendary gaming company Nintendo, even before the international release of the game, they managed to catch the well-known bug related to Sky Drop in Pokémon Black and White, but by this time the title was released in Japan, so the company instead of fixing the bug, I decided to keep it in the game to preserve parity. At the same time, the tester notes that, in addition to a strange policy, Nintendo has created an internal standard for carefully recording each bug, according to which testers who perform submissions are asked to attach a corresponding video that vividly shows the “nature” of the error found: “I think this It is excellent in strengthening their reputation as a company that makes quality games. ”

Explaining another specialist who is testing a product for compliance with certain requirements about the increasing complexity of testing will surely infuriate gamers even more than publisher policies. “It's all about the DLC,” says the tester.

“Imagine that we have a game with 40 DLC (elements of downloadable content). Even if we imagine that some or even all of these 40 packages simply open the hidden content originally placed on the disk, we still need to test every possible combination of these 40 parts:

• in all game modes;
• using various physical media for storing information (say, 20 DLCs are located on the hard disk, 15 are on a USB drive, 5 are on memory cards, etc.);
• comparing packages installed by the game owner with other DLCs activated by his rivals in multiplayer;
• verifying the proper behavior of the game in the event that some idiot has all the DLC stored on an external drive, and he suddenly decides to turn off this device;
• with each update of the game (1.01, 1.02, etc.);
• with various save files;
• etc. etc.

As a result, this can lead (and leads) to the moment when the task to test all possible combinations and scenarios becomes physically impossible for a person. The difficulty here increases exponentially with the addition of new variables. ”

As a result, the current testing situation can be described in two words: everything is difficult. It is especially difficult for testers to work with multiplayer games: even hundreds of testers sometimes will not be able to reproduce such bugs that can “catch” hundreds of thousands of people scorching for newcomers on game servers. And even though some testers say that now developers are working more closely and fruitfully with the QA-team than in the past, no one can change unrealistic deadlines.

“The deadlines for completing projects now set are absurd,” says one former tester. - Apparently, most people just do not understand that even for a game that is under development, say, 3 years, only half of this period can be called the time of full production. Only 9 months the game is being tested, of which only 3 months is being tested under conditions of the maximum load of the QA-team. At this time, we may be completely blocked access to the game, and by the time we can provide information about the product, the game is already receiving a beta version. Here I am exaggerating a little, if we talk about big well-known titles, but for most of the projects I worked with, this is a very real situation. When the QA department is finally allowed to speak, it turns out that there is simply no time for the proper response and appropriate actions. ”

Farewell no offense

Says one former tester:

“On one of the projects I had the opportunity to work in a room with many other testers - a kind of big“ pen ”. Most of them carried out their work on the basis of a temporary agreement. I sat next to people who had been testing one game for years and were damn exhausted by this job, but continued to work because they had no other choice, they still hoped. As I recall, only two or three employees from hundreds of testers got a permanent job. We regularly received indulgent letters of this kind: “Congratulations! We just got 8 hours of vacation! ” There was a feeling as if we were small children, and the teachers draw an asterisk for diligent behavior on our palm. Personally, I did not need it at all. Better put your praise on the payroll.

Meanwhile, in the corporate parking lot, every couple of months, new, bright cars appeared, as if descended from the covers of expensive magazines. The cost of each of these wheelbarrows was probably higher than the total amount that four or five testers from our team earned in a year. The company now and then organized parties for its regular employees. When you work hard and in spite of this you cannot afford new clothes, and such things happen very close, it is very depressing and depressing. As soon as the release of the game took place, a wave of layoffs swept through the testing department. I think about 100 people lost their jobs, but I do not know the exact numbers. I only know that the QA-department was practically erased from the structure of the company. A few hours after the guards took us out of the company, a party about the long-awaited release of the product broke out.

A lot of bad things happened in my life at that time. But I met many good people who knew how hard it was for us, temporary workers, - for this reason, good people left the company themselves. The policy of the company disappointed everyone with whom I happened to work there. I was transferred between departments, in the end, before leaving, I got into a small team with a large load. But I learned a lot and met really good people. When I told my colleagues about leaving, nobody was offended by me. On the contrary, the team members almost openly approved my step, as if I told them that I was quitting drinking or something like that. ”

Is it possible to build a successful long-term career in QA? Or will this area remain forever a means to achieve something more, a kind of transit point for moving to more interesting sectors of game development?

During the preparation of this article, I talked about 60 testers, and only four of them said they were considering a long-term career in QA. Many used this area to “jump over” to the position of producer or designer in gaming companies, while others worked as testers for several months or years, after which they gave up and transferred to other, more profitable working areas. Those who feel strong enough to stay and succeed in testing can try themselves in the role of a manager in the QA department - that specialization where the ability to manage other people is much more important than helping to create video games. For some, this, however, does not bring the expected satisfaction, especially given the fact that the stereotype about young, immature, careless testers in the work is often true.

“Immediately after starting work as a manager, managing a team, you get an experience for which no one can be prepared in advance,” says an experienced tester. - Testers come to work in pajamas, arrange each other sexual harassment, steal things ... Even such an elementary thing as ordering food can be a real challenge for a manager. QA is a work area, clearly biased towards young “specialists”. The team, which consists entirely of 18-19-year-olds, is quite common, and often this is their first serious job. Managing such a team is a real challenge for a manager who has to deal with young, fragile personalities and raging hormones. ”

Several experienced game developers told me about employees who are valued higher than the head of the QA department. They are those who know how to write a report on the caught bug, how to concentrate on working with critical errors instead of searching for small glitches, how to work with other people to achieve maximum productivity and avoid dumb stuff. Such people are rarely found in the field of QA - the combination of low wages and poor attitude towards employees discourages many testers from working in this area long before they get proper experience and learn useful things.

Another former tester named James, who asked not to write his last name, said the following: “Reflecting on my retirement, I asked my manager, assistant to the producer of the game project, to give me a few minutes of my time to ask for advice. I wanted to know how to move from the position of a temporary tester to a permanent position in the field of QA - for example, in development or another sector. He honestly advised me to distance myself from testing in general: "The faster you can get out of this area, the better for you." Eight months later, I left QA, moreover, I left the gaming industry altogether. Now I am a commercial software developer. I admit, in my life there is now less fun, but stability has increased, and the salary has increased significantly. ”

Based on real events

The film “ Boy for three ” (Grandma's Boys), which was released in 2006 and the image of actor Alan Covert (Allan Covert), who played the tester, became something of a joke in the QA world. People constantly ask real testers: which of the films shown in the film is true? Usually the answer is “not much.”

One of the episodes of the film, in which the narcissist game designer JP looks at an early version of the game and sees how the main character falls off his head, is often referred to as moments indicating a discrepancy between reality (waste 1:31):



Stupid right Bugs do not really appear like this. With the exception of special, isolated cases ...

“Once I had the opportunity to work on one game, sharpened under the Kinect touch controller, which included various obstacle competitions, taught as a TV show,” says the tester, who worked with a major publisher. - Before the beginning of each round, short introductory videos were played with the announcement of the participants and introductory comments. Once, and only once, the commentator's head suddenly moved from the usual position on the neck to an empty space near the character's body, and then just fell down, disappearing off the screen. Nobody subsequently could reproduce this bug, and we tried, believe me.

It looked surreal, especially after I had already become disgusted with that famous film because of all the inaccuracies and the simulated simplification (if you wish, of the vulgarization) of my work (“Does this guy just fell off his head? games! "). And a similar situation happened in a real video game I was working on! I will never forget this moment. ”

***

The fantasies of gamers from Westwood College advertising are far from real game testing. In fact, we are dealing with hard work and low wages. As a result, many testers find themselves in an unenviable position, doing outsourcing or performing contract work for a game developer who does not even know who does the testing for them. One QA specialist wrote to me that once he received an email from the HR manager of the company where he worked: “Employees were invited to write their thoughts and wishes on a festive corporate evening. When I responded to the letter by proposing a couple of my ideas, it turned out that the employees working under the temporary agreement were not even invited to the party. ”

Even those who find testing worthwhile, satisfying, say that the situation could be better - much better. Large-scale changes in the industry could improve the working atmosphere and increase the productivity of testers who would work productively in conjunction with game developers to eliminate the maximum number of bugs.

"Testing video games is the work of unsung and unqualified heroes," was reported in an analytical articlefrom the very well known company ST Labs. - For some companies, a typical testing strategy is to give the game to a large number of testers before the release and hope that the experts will find serious problems. The desire to understand the specifics of the game and modern technology, building a systematic methodology, rigorous testing - all this is still perceived by most development companies as a completely new approach. ”

This article was written by Duri Price and Ilya Pearlman back in 1997. After 18 years and 4 generations of game consoles, you can confirm that nothing has changed. Good game testers are still unsung and unqualified heroes who do everything in their power to eliminate all the bugs in the game. Despite the absurd deadlines and completely motivated colleagues, they do their job, which they always underestimate - at best.

However, there is another view on this situation.

Here's how one tester voiced it with a lot of experience: “You’ll deal with crappy games, crappy people and a crappy producer (or project manager) who thinks that he’s working with“ all these crap testers ”and cannot hold a candle . You will work for many hours, you will be underestimated, and every now and then you will notice that the most unusual bug with which you have been busy for almost a whole working day has been annulled with the note “Known. Uncritical for release. " But at the same time you will see your name in the final credits of the game, you will understand that you have contributed to a really cool project ... and you will take up work on another cool game. "

Do you work as a tester? Do you agree with the opinion of the author of the original article? Please tell us about your work, its pros and cons, as you see it.

PS I apologize to the Megamind community for such a protracted translation of the second part. I sincerely hope that you enjoyed the proposed reading.