About testing one game with pictures

I have many friends working in IT, and almost all of them are sure that I and my department are playing at work. For reference, I have been in charge of Testing at Innova since November 2009.

The world of gamedev for the inhabitants of the world of the development of "human", as I call it, software, is something incomprehensible and fabulous. Test gnomes, start a bug on an elf, simulate a test environment for a castle siege. I told on SQA days 7, what is the specifics of testing games (I have a slidecast in my blog).
')
We recently launched the Atlantic. This is a game from the Korean developer NDoors, which can be characterized in two words: MMORPG and step by step.

Everything, more about the game itself will be nothing. And it will be about her testing.

Yes, we did not develop this game, but only localized it, released it and support it, but there were a lot of testing too.

What to test in a localized game?

As I mentioned in my report, testing the game is divided into testing functionality, content and gameplay. Since the game, which is localized, is already running in other countries (at least in Korea, and in the case of the Atlantic - both in Europe and in the USA), we consider the gameplay to be the least risky area. Because, firstly, this part is run-in on production in other countries, that is, the number of errors here is the least, and secondly, these errors in the gameplay are more efficiently left to the players at beta testers.

Therefore, we test content and functionality.

Why functionality, the question arises. After all, the functionality is also tested in other countries. Of course, it is tested by the developer before the release of the original game, and at the level of code, and at the level of user testing. But, in the game, a huge number of combinations, and the same functionality in each entity is combined with model cards (or interface dialog), animation, voice acting and text. The probability of an error is great in our client assembly.
We combined content testing with functionality testing.

Yes, I must separately say that, of course, to test content at level 100, we didn’t have to swing for weeks, and to get the right quest to look for NPC ( Non Player Character ) in the woods. Having an account with a certain property in the database, you get access to the administration tool that allows you to do almost everything with existing game content: navigate to the coordinates, call on NPC, get the necessary items, weapons and clothes, raise and lower levels, learn skills in batches, etc. d.

Back to testing

A typical checklist for checking looked like this (using the example of a game item):

- the item is invoked
- there is a model
- the display name corresponds to the callee
- the name does not contain errors
- the name is placed when showing
- the name corresponds to the model
- the item appears in inventory
- information about the subject corresponds to the name
- information does not contain errors (grammar)
- information is displayed correctly (completely, placed in the interface)
- information corresponds to the work of the subject (if it says +10 to protection - really gives +10 to protection)
- correct message when it is impossible to use the item
- the item is discarded by the character
- correct message when discarding an item
- other related messages

It all looks like this in Excel:



With such checklists we covered all the game content. As you can see, without losing sight of and functionality.
After passing such a checklist is painted in red, green and yellow colors and looks much more beautiful.

Few numbers

The version of the Atlantic that we started testing contained about 2000 NPCs and monsters, almost 7000 items, 500 locations, about 800 skills, many interfaces and a lot of quests. Three of them mastered this volume in 2 months (with a break for the New Year holidays).

During this time, 455 localization errors and 101 game functionality errors were found and fixed. In fact, I have these metrics more complex: both by criticality, and by rediscovery, and by the time of correction, and by the initial incorrect identification of the owner of the error. But I think this is not as interesting as the examples of the errors found.

I am a poor peasant woman, letting sleep, my run for a long, long time

The simplest example of a localization error is the wrong gender. “When I was in Rome,” the faceless resource file says to the translator. “When I was in Rome,” the letters from the screen tell us in a bass. And in fact this is a fragile young lady.



A small lyrical digression about the process of translating the texts of the game. It occurs in two stages: first, just a translation, and then editing. In these activities are involved different people with different specializations. But those and others initially work only with texts. So the replica “What are you saying, did the prompts translate?” - this is actually not a minus for the translators, but a minus for testing, which perhaps could not have happened at all.
Neither the translator nor the editor is unaware that the word “Mammoth” can be translated differently than “Mammoth”, but the tester’s eyes clearly see a bug:



Or another example: just seeing the picture and understanding the purpose of the object, you can determine that the scroll is incorrectly called a stack, and the glue is not a medicine:



Or, for example, quests that meant entering the correct answer in a text field.

The answer taken by the program, for some reason, was not moved to the resource files, but was hidden in the code, so the quest where you had to answer the question: “ In which city did you meet the Japanese policeman? "Accepted at first the answer" Tokyo ", which is completely unobvious for the Russian-speaking user.

Translate the picture!

Many texts turned out to be pictures. We translated, requested images, our designers reworked them, the developers inserted into the client, we tested.
Naturally, the cycle of correcting this kind of error is much longer than just an error in the text.

Not always, by the way, corrections were dedicated to really mistakes in the literal sense of the word. For example, the message YOUR MOVE during the transition course during the battle after discussion transformed into YOUR MOVE that by that time, with the light hand of our designers, had become the slogan of the game.



Fixed button length - game tester headache

Separately, all interfaces were run, the correct placement of the inscriptions on the interface elements, the correspondence of the inscription to the meaning of the element, the same name of the element in different places were checked here.
This is how the game information book looked like in the first version:



It should be noted that after such testing a lot of abbreviations appeared in the game, because the sizes of the interface elements are rigidly fixed, and the meaning is to be preserved.

There were unsolved cases. For example, for the button, the meaning of which is “Collect everything with one action, which has fallen from all the killed enemies after the battle”, we were allocated only 7 characters. Even the shortest version that came to mind "Take Everything" contained as many as 9 characters. The players also could not come up with a suitable option. An attempt to call this action "Marauder" also did not lead to anything. As a result, this button is called “Search All”, which frankly does not fit in the area designated for the inscription, but decided to sacrifice beauty for the sake of clarity.



I do not know the rules lane
enos text you in
general about what?

Initially, all the texts in the game when you go to a new line just tore out the word. In Korean, words are transferred in syllables (hieroglyphs), without a hyphen, and in any place. We were surprised to learn that in Europe and the USA they play with such transfers. We have achieved partial corrections, and now in the game hyphenation in the texts of the dialogues with the NPC are implemented by words. But, as we see, not everywhere.

It was	It became

By the way, the texts in the chat rooms and systemic dialogues of the Atlantic are still being torn on the letter at the end of the line, so they could not fix it.

Quotes us to nothing

Or such a case: we have a list of ~ 500 books of the type Book “Deadly Shot”, after reading which, the character learns the skill. In the team calling the object in the game when testing we enter the full name - we get the object without quotes in the name.



We go to the editors, those in horror! It turns out that files without quotes came from the translators and the guys manually added quotes to all the names in order to comply with the rules of the Russian language, and indeed, it was better perceived by the eye. And so, in the game, no quotes. We check in the system from where the developers take the files to build the build - there are quotes. In the game - no quotes. We found out that at the level of the game, all quotes are cut out, because their presence leads to problems.

Resource files for wimps!

I must say that the Atlantic has a very big disadvantage for us as localizers. How do you imagine the software to be localized? Well, obviously: all texts are rendered into resource files, all common elements are rendered into variables, all references to common elements refer to these variables. There were problems with this.

Most often, every mention of a term contained precisely its name, and not a reference to a variable at all. And one and the same concept could have been quite differently translated (the translator is not one):



Or so:



Localization is not only a translation

Especially for the release of the game in Russia, the developers created the locations of Moscow and St. Petersburg. They can easily recognize the familiar architectural structures.
With local NPCs it’s harder, sometimes it’s hard to guess without signatures :)



In order to identify Kurbsky , you will have to wait until he stops doing facepalm:



We test not only texts

Plus, of course, the Smoke test of each client assembly (even if it includes only text corrections). This Smoke test contains a check of the main functionality of the game and takes 1-2 hours of work by one person + half an hour of group play.

People often ask me why you are testing the game itself, if it has been working in other countries for a long time. The answer is: do not think that in other countries it is without bugs. There are a lot of them, believe me. Therefore, our testing is not superfluous.

And even this is not the most important. The most important thing is build bugs. There is no guarantee that in the next build they will not forget to put a folder with NPC textures in Moscow, and the players will not wander around the deserted city in search of at least someone who would give them a quest.
There is no guarantee that the new build will be installed correctly. Because the slightest error in encryption, cheksummy did not converge, and the server does not see some of its components.

Or even such a thing:
In the game information book, all NPC descriptions have moved down one position. Suddenly. You look at a peasant, and the information you see about the Traveler - the next on the list of NPC. By the way, this bug could not be fixed for a very long time. If I'm not mistaken, he lasted more than a month.



User assistance

Testing any game is divided into 3 parts: internal testing, closed beta testing and open beta testing. Testing a localized game is no exception.

We also added pre-PTA, calling it alpha-testing. Experienced Atlantic players who played on European servers and would be happy to switch to Russians after they were invited were invited to it.
These guys helped us a lot to hone the quality of translations. Only they could tell that the correct, from the point of view of the Russian language, text actually contains a semantic error:



As a result, for the period of alpha testing (2 weeks), 95 localization bugs and 31 bugs of the game itself were found and fixed.

How to teach the user to correctly describe the bug

And it should be noted that we have chosen a forum for communicating with beta testers. Separate closed branch with topics broken by builds or types of errors and hard formatted error posts. And not lost. Firstly, the forum is a clear and convenient tool for users. When people who are not obliged to do this help you in work, it is important to create an environment as close as possible to their usual one. Secondly, the forum has saved us from the need to disassemble duplicates, as it would have had, if we had chosen the tracking system.

Beginning with alpha testing and before the release itself, there was only one tester on the project who was engaged in playing (if needed), designing, assigning and tracking found bugs from beta testers, as well as Smoke and regression testing of bug fixes.

Closed beta testing

After the end of the internal testing phase, we distributed the accounts to the registered, opened a public server and started the PTA.

In the PTA version opened + 20 levels of content, so most of the errors fell on this, untested area. However, for the project it is wrong to delay the PTA for the sake of rolling out the users of the fully "licked" version. Firstly, they want to be quick, secondly, they want to help! Here it is important to choose good enough, suitable for the audience of this game.
Judging by the reviews of our beta testers, we have met their expectations with fairly high quality.

PTA is a stage in which there are still many bugs. Just by virtue of the law of large numbers. If 1000 players are combing the same thing, then the probability that they will find a bug missed by one person is very high.

It is at the PTA stage that it becomes possible to identify server configuration bugs, for example, that a tournament starts every day at 6:00 pm and ends after 5 minutes if 10 participants do not have enough (each number is a parameter), or that the hunting limit is removed at 6 am only for characters that meet certain conditions.

Localization bugs that our beta testers found during the PTA mainly related to quest texts. Naturally, the tester, reading the hundredth quest for the day, doesn’t see with his eye those little things that the beta tester player, seeking to receive an in-game horse after release for active participation in testing, seeks out.

There were pearls. Naturally, we cannot limit either the age level, or the educational level, or the level of enthusiasm of the players, so many “bugs” from the forum never moved into the bug tracking system.

A soft sign is missing, the user tells us - well, a person with a high syllable is not familiar:



But the syllable of gamers is a separate language. What are the bug reports of the type:

- “ even though I am a noob, I don’t know how to put images, but there is a hole in the Sea Palace, where the first bosses, admirals follow them, and 1 of them will go to the place where the hole is located”
- "I did not take the mail tried in different permissions, but still"

That is, my guys had to not only translate bugs for developers from Russian to English, but also from gaming to Russian bugs for editors.

Or another phenomenal report:

- “ I don’t remember where and where I ran - I noticed that the shadow of the tree moves as I approach ” - a report without a video.

During the PTA period, 93 localization bugs and 22 game bugs were found and fixed.

Open beta testing

There was a short break between the PTA and MBT, as they were preparing a new server, already designed for other volumes of players.

MBT is a stage when everyone who wants can already play, but the game still has the right to contain bugs. And the game at this time is absolutely free. If she has a business model of a monthly fee, then people do not pay a subscription, if she has a free-to-play model, and she earns from selling in-game items, then during this period these items cannot be bought in her. The game has not yet built a store and it is not connected to the billing.

Of course, it is more difficult to cope with so many fasting on the forum. We transferred user reports to the usual ticket system, and the forum was already followed by moderators and community managers.

Here, of course, the main focus on multiplay. Clans, guilds, tournaments can not be tested by several people. The difficult gameplay of the social direction is really more effective to test with the help of people who understand this.

Total for the MBT period, 36 localization bugs and 9 game bugs were found.

The best bug of the Atlantic during its testing

“I left the spell on the night of the AFC fishing. There was disconnection due to the change of the IP address by the provider. In the morning I could not log in, I threw out the login window when I tried to connect to the server.
People in the game said that my spell is online and continues to fish . ”

We really could not figure out what the problem was for a long time, the player so hung up several times. Already thought to put a monument there, or ask the Koreans to do a quest with the NPC Rybak ghost with the player's name.

Testing money

While on the server, which in the future was going to become a production, beta testers were starting to build their empires, a production version was being tested on the test server. The one that was already intended to make a profit. Then a game store connected to the billing system appeared, then the content of this store appeared - items on which players spend their real money. There are weapons chests, teleportation scrolls, and horses speeding up movement.

The financial part of the game is also not the first time was ready for adulthood. More than a dozen mistakes had to be fixed by the developers. Moreover, almost every mistake involved the joint work of our and Korean developers. After all, the integration bug is always common.

Happy Birthday!

And here is the release. First one server, and after 2 weeks they opened the second one.
After the release, testers no longer communicate with the players, because they are no longer beta testers, but users. They communicate with the support service. It sends us user requests related to errors in the game, and then the usual work with the bug goes on.

A little later, work will begin on the selection and organization of a group of beta testers who will help us test updates, but this later.

Conclusion

This is the first project of my team in Innova, and I can say that we tested well.

This is my first ever game project to be tested. Very interesting experience.

I didn’t tell here about testing the site, testing the installers, testing the autoupdate system, testing the integration with the Frost game protection, testing the client’s downloading from file transfer, testing the integration of the game with billing, configuration testing, testing the referral system, and building the process of working with localizers and with Korean developers. These are all the usual things for you. About them it would be interesting to listen to the players. And I tell you things that are commonplace for them.

Of course, we did not solve all the problems. Here are some of those that remain:

- A large number of files in the client do not contribute to a quick download: 4 gig client contains more than 30,000 files
- the transfer of words in chat rooms and system messages is still performed by a simple word break
- some pop-up messages during the battle have not been translated: Bonus, Skill Up, Exp +, etc.
- do not fit the three-line description of the missions, despite the presence of space in the interface:



Some game bugs promise to be fixed in the next version, which we expect at the end of the year. Not everything is done as quickly as correcting typos in the quest text.

Thanks to Sasha and Zhenya, who served as a thread for my entry into gamedev, thanks to Max and Stepa, who appeared at the time of maximum tension, thanks to Misha, who brought the project to release. You guys are great!