Developer Errors in MMO Testing
This article is a look at MMO testing from the “other side of the barricades” from a gamer's point of view. I want to talk about how to make the alpha test, PTA and MBT more efficient. And why beta testers stop writing bug reports and go “into opposition” to the project.
During my life I participated in the PTA and MBT for about 30 games. And the most important conclusion that I made for myself is that game developers do not know how to manage those human resources in the form of several thousand gamers eager to cry out all the functionality that gamers are dumped on their heads. And that is why this material was written. From the gamer to the developer sitting on the wrong side of the screen. From the tester to companies that do not understand how to properly organize the testing process through the efforts of the players.
The article is divided into four sections and written in conversational style. I sort through the mistakes of the developers on the alpha test, the PTA / OBT, and when recruiting testers after the release. Those who want specifics without long preludes and reflections can immediately flip through to the last paragraph of each section and conclusions.
In recent years, it has become a sign of good tone to invite players to the alpha test: World of Tanks (2009), WarThunder (2011), Lost Sector (2013), Armored Warfare (2014), etc. I argue that this is a bad idea and AT should be conducted by professional QA engineers, not gamers.
')
Why? Because the main consequence of having players on AT is the reputational loss for your company . Obviously, this thesis may cause confusion. Explanation below:
1. Players come before others to your project not to test it on bugs and bring you all the errors on a platter, but to get your portion of entertainment and pleasure first. Undoubtedly, if they find some critical bug that prevents them from playing, then they will report about it. But in general, gamers want to play (and play again!), And the next glitches of the next test build will in fact only cause irritation, not understanding. And since people remember the negative much more easily than the positive, then by the release this part of the players will remember, not the new features that you have entered, but all your mistakes. And it will be the first army of critics of your online game.
2. In practice, it looks like this (on the example of the players' reaction to the next patch):
newbie : Hooray, new patch! New opportunities!!! This is wonderful!!! I will like the developers on the forum.
beta-tester : Oh, yes ... New patch - new glitches. Until I play. I'll wait until rolled back, correct and bring to mind. My nerves are more precious to me. In the meantime, play a game of competitors.
alpha-tester : Well, if again not wrought out to the whole world. Remember how the server crashed when 0.4.5 rolled in? I’ll go, I’ll write to LiveJournal that we’ve been screwed up again and we’ve done everything wrong, although we’ve told them that we need to rebalance this card.
Of course, this is a hypothetical dialogue, but I think the essence is clear. I could bring the real dialogues of players who came at different times in WoT, but this is contrary to the rules of this resource. In particular, those items that concern insults and mats. Because I can’t remember without thinking any ZBT-person who would censor about the developers.
Yes, alpha testers and PTAs are far fewer than other categories of players. But it is they who most often occupy leadership positions in all clans / guilds / corporations (underline the appropriate). At least at first. And accordingly, as the most experienced players, they also form an opinion about the project as a whole within the framework of their clan.
Of course, if your project “shoots”, then over time, the first clans will “dissolve” and, in the general information noise, the voices of the ever-dissatisfied first testers will not be heard. But, as they say, the sediment will remain ...
A small clarification-summary : no, I do not claim that the testers are evil. I say that the players who came to AT / PTA / MBT and QA engineers have completely different goals and, accordingly, a different reaction to the problems in the game. Therefore, the idea of inviting players to AT when the game is a bundle of bugs is a very bad idea.
In theory (in my understanding), the main goal of the PTA and MBT is to test the efforts of gamers to dynamically update the game functionality for errors prior to release. And from here we get the second goal - testing the reaction of the target audience to the concept of the game and various features that developers introduce (or plan to introduce). And as a side effect - minimizing the costs of internal testing by employees of the company-developer.
In practice , any PTA and MBT turns primarily into a rather dubious info-noise generator (as an example - google shkoloty shkoloty, which, having first heard of the existence of Armored Warfare, lifted the game to the skies, and now pours mud on it, although the game is even on release did not come out). And in terms of efficiency of testing, the results are also very controversial. To be precise in the formulations - the developers are doing everything to discourage the tester players from the desire to do something. And this catastrophic inability to work with an audience of several thousand people who can really give high-quality feedback, made me write down below typical mistakes in the PTA / MBT organization and methods of their treatment (from the player’s point of view).
So, let's go:
In general, in my personal conviction, the majority of the PTA / MBT are held under the motto “Test It, I don’t know What”. So, suppose we have some kind of hyperactive grouping of tester players who sit in the game day and night, find bugs and write messages on the forum (for example, for military-historical games - these are “riveters” who defend the historical performance characteristics of their favorite ).
And here the main problem lies in the fact that neither the developers nor the community managers are not able to direct the energy of this target group in a constructive way to the benefit of the project. From the word “absolutely”. Seriously. Where are the recommended targets for testing? Where is the rough plan of what the developers would like to see tested from the players? Not. None of the above is available in practice. Therefore, everyone is testing himself, who is in that much - one makes a video on YouTube with the penetrability of models, and the second is sawing on the forum signs with information on the penetrability of guns of different calibers from different sources.
The special charm of the situation lies in the fact that a “historical consultant” often hangs in company vacancies, but at the same time, the developers send almost obscenities to the game forum of professional historians with historia.ru or tsushima.ru, pointing to inconsistencies. that "in our 100% realistic game, historicism is sacrificed for balance." About the wiki, which make schoolchildren based on GoogleTransfer articles, generally keep quiet.
Separately freezes, when the developers in the patch-laptop on the test do not write changes - from the series "in order to optimize the balance, the characteristics of some machines were changed." Well, and how, without knowing at all what has changed, can we test innovations? Or conduct regression testing? Us with hot tongs every time to pull information out of you, gentlemen developers? I would like to emphasize - we are talking about the PTA / MBT, when absolutely all the performance characteristics in the balancing process (and we, the testers, understand this perfectly), and not about the post-release, when the nerf or up some equipment can cause a wide negative resonance of millions of players.
And now let's imagine that despite all of the above, a certain Vasya Pupkin (who does not even suspect that he is actually engaged in black-box testing) intuitively guessed what he needed to test and wrote his first bug report on the forum!
And in response:
- did not receive any response at all;
- the topic was closed and transferred to the archive without explanation;
- I received the answer “the bug is known, we are working on it” even without elementary gratitude;
- and so on according to the speaker ...
What will be Vassenka's reaction? - Right! Vassenka will not write a single bug report anymore.
Gamers (even testers) are, first of all, consumers. Never forget about it. They are not diplomats or corporate spokesmen. So this is quite a normal working situation, when together with the bug reports developers are poured a tub of cold water. Sometimes even with slop (deservedly or not - this is another question). What is it for me? And besides, about a third of my friends on the PTA / OBT stopped writing bug reports, receiving a sharp rebuke.
As part of a gaming forum at the PTA / MBT stage, I usually observe two options for responding to valid criticism of players, supported by bug reports:
- punitive-repressive, up to the bans (a moderator came with a banhammer);
- accusatory (the developer came and started scolding the players that they are ungrateful bastards - the developer didn’t sleep here at night, my wife hadn’t had sex for a week, and the players all poked their bugs in their head, they would not calm down).
As a result, some more testers become hostile to your project and quit testing it. This is at best. And at worst - runs to complain about gaming portals and goes to competitors.
And then came the long-awaited MBT. Our hero Vasya Pupkin, who has 7 bug reports, and Dima Kozlov, who registered on the last day of the PTA (real-story) and did nothing, are sitting in the Tim-spike. And the reward of both for participation in the PTA is the same - a certain tanchik, for example. What do you think, Vassenka after such a universal injustice, at least one bug-report will write? You think correctly - no. And here is another group of qualified testers sunk into oblivion.
In general, it seems to me that the conclusions are already obvious. But when you explain it with examples in a personal development, you come across a blank wall of incomprehension (although outside of the game maiden this is most often a completely adequate person, according to FB and VK).
1. If you want players to really test something on a beta test - treat them like your full-time QA: give them a test plan and create an infrastructure on the forum to digest the results of these tests. Explain what, when, how and why to test. Ideally, with mailing to everyone. In short, raise the standards for beta testing! Do you need students for extras or testers !?
2. Provide your beta testers with feedback and, most importantly, with timely (!) Communication. Adequate feedback to bug reports is a guarantee of the long-term cooperation between beta testers and the DG company.
3. Learn to respond adequately to the criticism of players and admit their mistakes. An offended player is not only the line “banned” on your forum, but also 500 terabytes of your games distributed on torrents (true-story). And not a single licensed copy of your game in the gift box on the shelf of his clan members.
4. Remember once and for all - it is impossible to reward all participants of the PTA / MBT equally. The reward must be proportional to the contribution of the player to the project! Otherwise, those who actually tested, you completely discourage the desire to do anything until the end of the life of your project.
After the release, nothing changes: players want new content, and developed - their own penny for the content of two mistresses and a small house on the islands. So patches and add-ons come out like hotcakes from under the baker’s hand, and bugs continue to multiply like rabbits. The only difference is that super testers are replacing beta testers. And here begins the next act of Marlezonsky ballet of our drama of the Absurd.
If someone does not understand - I'm on the criteria for recruiting in super testers. Remembering WoT, we were very amused by the picture when the requests of our tactical officers, who work in IT in real life and have a winrate of 53-55%, were rejected, and our acquaintance’s son with a stat of 43% was taken. And all because, unlike us, he could play 12 hours a day. By the way, in those days when I had already deleted my account in the tanks a year later, he had not yet found his first bug.
And this problem is not only tanks. For some reason, they prefer to recruit shkolota, ready to play around the clock, for a super-test. I have nothing against Central Asia - for someone we have to fill in experience and frags. But offer them to test? People, are you serious? What kind of tester will be more useful - from a teenager who will play 16 hours in the style of “pysch-pysch ololo - I'm a UFO driver!” Or from an hour of play an adult who comes to the test to do business?
On this, I have almost everything and, as the classics said, “Scheherazade is silent, moving from the allowed speeches to unauthorized thoughts.”
In fact, I have only one clever idea - gamification of game testing. If you want players to test your games, make them enjoy it! Do you rank top players? - Add to the site a rating of top testers. Do you issue medals and achivki for any sneeze? - Make separate unique achievements for all who find bugs and help you make the project better. Do you have x2 experience on weekends for zadrotov? - Enter increased rates for testers. From this all will benefit.
Ps . I do not claim that I "discovered the Truth." I just chewed on quite obvious things, understandable to anyone who has ever been involved in organizational work. But since the game developers from game to game, from release to release, diligently continue to jump on a rake called “how to force gamers to stop writing bug reports and sabotage the testing of the game”, then someone had to write this post.
Foreword
During my life I participated in the PTA and MBT for about 30 games. And the most important conclusion that I made for myself is that game developers do not know how to manage those human resources in the form of several thousand gamers eager to cry out all the functionality that gamers are dumped on their heads. And that is why this material was written. From the gamer to the developer sitting on the wrong side of the screen. From the tester to companies that do not understand how to properly organize the testing process through the efforts of the players.
The article is divided into four sections and written in conversational style. I sort through the mistakes of the developers on the alpha test, the PTA / OBT, and when recruiting testers after the release. Those who want specifics without long preludes and reflections can immediately flip through to the last paragraph of each section and conclusions.
I. Alpha testing
In recent years, it has become a sign of good tone to invite players to the alpha test: World of Tanks (2009), WarThunder (2011), Lost Sector (2013), Armored Warfare (2014), etc. I argue that this is a bad idea and AT should be conducted by professional QA engineers, not gamers.
')
Why? Because the main consequence of having players on AT is the reputational loss for your company . Obviously, this thesis may cause confusion. Explanation below:
1. Players come before others to your project not to test it on bugs and bring you all the errors on a platter, but to get your portion of entertainment and pleasure first. Undoubtedly, if they find some critical bug that prevents them from playing, then they will report about it. But in general, gamers want to play (and play again!), And the next glitches of the next test build will in fact only cause irritation, not understanding. And since people remember the negative much more easily than the positive, then by the release this part of the players will remember, not the new features that you have entered, but all your mistakes. And it will be the first army of critics of your online game.
2. In practice, it looks like this (on the example of the players' reaction to the next patch):
newbie : Hooray, new patch! New opportunities!!! This is wonderful!!! I will like the developers on the forum.
beta-tester : Oh, yes ... New patch - new glitches. Until I play. I'll wait until rolled back, correct and bring to mind. My nerves are more precious to me. In the meantime, play a game of competitors.
alpha-tester : Well, if again not wrought out to the whole world. Remember how the server crashed when 0.4.5 rolled in? I’ll go, I’ll write to LiveJournal that we’ve been screwed up again and we’ve done everything wrong, although we’ve told them that we need to rebalance this card.
Of course, this is a hypothetical dialogue, but I think the essence is clear. I could bring the real dialogues of players who came at different times in WoT, but this is contrary to the rules of this resource. In particular, those items that concern insults and mats. Because I can’t remember without thinking any ZBT-person who would censor about the developers.
Yes, alpha testers and PTAs are far fewer than other categories of players. But it is they who most often occupy leadership positions in all clans / guilds / corporations (underline the appropriate). At least at first. And accordingly, as the most experienced players, they also form an opinion about the project as a whole within the framework of their clan.
Of course, if your project “shoots”, then over time, the first clans will “dissolve” and, in the general information noise, the voices of the ever-dissatisfied first testers will not be heard. But, as they say, the sediment will remain ...
A small clarification-summary : no, I do not claim that the testers are evil. I say that the players who came to AT / PTA / MBT and QA engineers have completely different goals and, accordingly, a different reaction to the problems in the game. Therefore, the idea of inviting players to AT when the game is a bundle of bugs is a very bad idea.
Ii. PTA & MBT
In theory (in my understanding), the main goal of the PTA and MBT is to test the efforts of gamers to dynamically update the game functionality for errors prior to release. And from here we get the second goal - testing the reaction of the target audience to the concept of the game and various features that developers introduce (or plan to introduce). And as a side effect - minimizing the costs of internal testing by employees of the company-developer.
In practice , any PTA and MBT turns primarily into a rather dubious info-noise generator (as an example - google shkoloty shkoloty, which, having first heard of the existence of Armored Warfare, lifted the game to the skies, and now pours mud on it, although the game is even on release did not come out). And in terms of efficiency of testing, the results are also very controversial. To be precise in the formulations - the developers are doing everything to discourage the tester players from the desire to do something. And this catastrophic inability to work with an audience of several thousand people who can really give high-quality feedback, made me write down below typical mistakes in the PTA / MBT organization and methods of their treatment (from the player’s point of view).
So, let's go:
1. Lack of information on tests
In general, in my personal conviction, the majority of the PTA / MBT are held under the motto “Test It, I don’t know What”. So, suppose we have some kind of hyperactive grouping of tester players who sit in the game day and night, find bugs and write messages on the forum (for example, for military-historical games - these are “riveters” who defend the historical performance characteristics of their favorite ).
And here the main problem lies in the fact that neither the developers nor the community managers are not able to direct the energy of this target group in a constructive way to the benefit of the project. From the word “absolutely”. Seriously. Where are the recommended targets for testing? Where is the rough plan of what the developers would like to see tested from the players? Not. None of the above is available in practice. Therefore, everyone is testing himself, who is in that much - one makes a video on YouTube with the penetrability of models, and the second is sawing on the forum signs with information on the penetrability of guns of different calibers from different sources.
The special charm of the situation lies in the fact that a “historical consultant” often hangs in company vacancies, but at the same time, the developers send almost obscenities to the game forum of professional historians with historia.ru or tsushima.ru, pointing to inconsistencies. that "in our 100% realistic game, historicism is sacrificed for balance." About the wiki, which make schoolchildren based on GoogleTransfer articles, generally keep quiet.
Separately freezes, when the developers in the patch-laptop on the test do not write changes - from the series "in order to optimize the balance, the characteristics of some machines were changed." Well, and how, without knowing at all what has changed, can we test innovations? Or conduct regression testing? Us with hot tongs every time to pull information out of you, gentlemen developers? I would like to emphasize - we are talking about the PTA / MBT, when absolutely all the performance characteristics in the balancing process (and we, the testers, understand this perfectly), and not about the post-release, when the nerf or up some equipment can cause a wide negative resonance of millions of players.
2. Lack of adequate feedback
And now let's imagine that despite all of the above, a certain Vasya Pupkin (who does not even suspect that he is actually engaged in black-box testing) intuitively guessed what he needed to test and wrote his first bug report on the forum!
And in response:
- did not receive any response at all;
- the topic was closed and transferred to the archive without explanation;
- I received the answer “the bug is known, we are working on it” even without elementary gratitude;
- and so on according to the speaker ...
What will be Vassenka's reaction? - Right! Vassenka will not write a single bug report anymore.
3. Negative reaction
Gamers (even testers) are, first of all, consumers. Never forget about it. They are not diplomats or corporate spokesmen. So this is quite a normal working situation, when together with the bug reports developers are poured a tub of cold water. Sometimes even with slop (deservedly or not - this is another question). What is it for me? And besides, about a third of my friends on the PTA / OBT stopped writing bug reports, receiving a sharp rebuke.
As part of a gaming forum at the PTA / MBT stage, I usually observe two options for responding to valid criticism of players, supported by bug reports:
- punitive-repressive, up to the bans (a moderator came with a banhammer);
- accusatory (the developer came and started scolding the players that they are ungrateful bastards - the developer didn’t sleep here at night, my wife hadn’t had sex for a week, and the players all poked their bugs in their head, they would not calm down).
As a result, some more testers become hostile to your project and quit testing it. This is at best. And at worst - runs to complain about gaming portals and goes to competitors.
4. Doubtful reward
And then came the long-awaited MBT. Our hero Vasya Pupkin, who has 7 bug reports, and Dima Kozlov, who registered on the last day of the PTA (real-story) and did nothing, are sitting in the Tim-spike. And the reward of both for participation in the PTA is the same - a certain tanchik, for example. What do you think, Vassenka after such a universal injustice, at least one bug-report will write? You think correctly - no. And here is another group of qualified testers sunk into oblivion.
In general, it seems to me that the conclusions are already obvious. But when you explain it with examples in a personal development, you come across a blank wall of incomprehension (although outside of the game maiden this is most often a completely adequate person, according to FB and VK).
So, brief conclusions
1. If you want players to really test something on a beta test - treat them like your full-time QA: give them a test plan and create an infrastructure on the forum to digest the results of these tests. Explain what, when, how and why to test. Ideally, with mailing to everyone. In short, raise the standards for beta testing! Do you need students for extras or testers !?
2. Provide your beta testers with feedback and, most importantly, with timely (!) Communication. Adequate feedback to bug reports is a guarantee of the long-term cooperation between beta testers and the DG company.
3. Learn to respond adequately to the criticism of players and admit their mistakes. An offended player is not only the line “banned” on your forum, but also 500 terabytes of your games distributed on torrents (true-story). And not a single licensed copy of your game in the gift box on the shelf of his clan members.
4. Remember once and for all - it is impossible to reward all participants of the PTA / MBT equally. The reward must be proportional to the contribution of the player to the project! Otherwise, those who actually tested, you completely discourage the desire to do anything until the end of the life of your project.
Iii. Post release
After the release, nothing changes: players want new content, and developed - their own penny for the content of two mistresses and a small house on the islands. So patches and add-ons come out like hotcakes from under the baker’s hand, and bugs continue to multiply like rabbits. The only difference is that super testers are replacing beta testers. And here begins the next act of Marlezonsky ballet of our drama of the Absurd.
If someone does not understand - I'm on the criteria for recruiting in super testers. Remembering WoT, we were very amused by the picture when the requests of our tactical officers, who work in IT in real life and have a winrate of 53-55%, were rejected, and our acquaintance’s son with a stat of 43% was taken. And all because, unlike us, he could play 12 hours a day. By the way, in those days when I had already deleted my account in the tanks a year later, he had not yet found his first bug.
And this problem is not only tanks. For some reason, they prefer to recruit shkolota, ready to play around the clock, for a super-test. I have nothing against Central Asia - for someone we have to fill in experience and frags. But offer them to test? People, are you serious? What kind of tester will be more useful - from a teenager who will play 16 hours in the style of “pysch-pysch ololo - I'm a UFO driver!” Or from an hour of play an adult who comes to the test to do business?
On this, I have almost everything and, as the classics said, “Scheherazade is silent, moving from the allowed speeches to unauthorized thoughts.”
Iv. Clever thoughts
In fact, I have only one clever idea - gamification of game testing. If you want players to test your games, make them enjoy it! Do you rank top players? - Add to the site a rating of top testers. Do you issue medals and achivki for any sneeze? - Make separate unique achievements for all who find bugs and help you make the project better. Do you have x2 experience on weekends for zadrotov? - Enter increased rates for testers. From this all will benefit.
Ps . I do not claim that I "discovered the Truth." I just chewed on quite obvious things, understandable to anyone who has ever been involved in organizational work. But since the game developers from game to game, from release to release, diligently continue to jump on a rake called “how to force gamers to stop writing bug reports and sabotage the testing of the game”, then someone had to write this post.
Source: https://habr.com/ru/post/259607/
All Articles