Testing games in Innov: a story about the work of the department
As a preface, I will say that I came to Innova a little over a year ago, my task was “to do testing in the company”. My testing department consists of two groups: the web application testing group and the gaming application testing group. Such a division has developed because these areas have different tasks and different requirements for employees.
Next story will be about the direction of game testing. This is a story about my guys, about our processes, about our organization of work. I invite you on a word trip.
External environment
')
I'll start with a description of the external system. We have 11 game projects. They are of different sizes, they are developed by different companies (most often Korean), they have different processes, different life cycles, different frequency of updates, different quality and different initial quality requirements.
At first glance, it seems that Innova deals only with localization, and it would seem, what is there to test? They give us the finished product, test only the localization. But these different initial requirements for the quality of products play a role. Different development companies consider it normal to release a product on their (Korean) market with a certain number of known bugs. And this amount is different for everyone, as you understand.
We are trying to keep this number to a minimum in all our games. Because as soon as we release the game in Russia, it becomes “ours”. We think so.
Testing process
However, we are not testing the whole game. That would be wrong. In the ideal case, we really need to test only the localization and build. To this we add another thorough testing of the new functionality and beta testing by users.
That is, the test plan for the update looks like this:
- localization testing:
- - check lists, deadlines
- testing of new functionality:
- - checklists for checking, priorities, deadlines
- assembly testing:
- - Smoke tests, timing
- beta testing:
- - task for players, terms
I talked about the full cycle of testing localized games on the example of the Atlantic.
The process of interaction with developers depends on the project and on the company-developer. Bug-reports can be made out in BTS, on our or on their side, can be collected in Excel or Google-docs. The testers interact with the developers to fix them, but in some places all communication goes through the project manager. We adapt to the project, but make changes to the processes for their optimization.
Test environment
In addition to the combat servers (of which different games from one to thirteen), there is a Public Test Server system (PTS) and an internal test server (QA-stand). On the TCP can enter any player if desired. There are all the beta testing. In order to get there, you do not need any special difficulties, you do not even need a separate account, the one that you play on combat servers will do.
At the QA-stand, internal testing is carried out, before giving to the players on the PTS.
Not any game update goes all the way QA-booth -> PTS -> combat servers. Some updates can not be put on the TCP without hitting the combat system. These immediately go into battle after internal testing. Some, on the contrary, cannot be properly tested on the internal stand, and they immediately go outside to our beta testers.
The quality of the combat product
And yet, you have to release products with known bugs. And - it happens that with critical. For example, the latest major update of the legendary Lineage II High Five Part 3 project installed on the servers on December 28 brought a serious bug: randomly the game client crashed with a critical error on the teleport users.
This may depend on the fact that, for example, this bug manifests itself with the load on the servers. And then he pops up only in combat operation. We do not recreate the load in our testing cycle, because it is a development company. But, not always effective. Since updates are often launched at the same time as Korea and often before Europe, we run the risk of launching with unknown bugs.
This may depend, for example, on the scheduled launch of the update, which all players are waiting for.
This may depend on indefinite deadlines for solving the problem from the developers.
I’m sure that if they were able to arrange a survey among the players affected by the bug with the teleport in the High Five, they would agree to play without updating to this day, they would still have chosen the pre-New Year update.
Of course, known errors are reported to players. And the status of their decision is constantly updated. And, of course, they are unhappy with the answer “Sent to the company-developer, they are working on it”!
Team organization
Almost every project has a test manager. This is one of my guys who best understands this project, which plans to test the project, participates in the planning of installing updates. He knows the weak points in his project, and he knows what is most important for his audience.
His tasks include:
- planning the testing of the project (or its updates)
- organization of project testing (or its updates)
- - internal testing
- - external testing (using beta testers)
- informing all interested parties about the status of testing and the state of the product
The test manager interacts with the work:
- with the project manager (plan updates together, approve the launch)
- project engineer (information on installing patches, fix assembly errors)
- developers (reports bugs found, test results)
- colleagues-game testers (sets tasks, collects results)
- project editor (localization error message, their correction)
- community project manager (known errors, working with beta testers)
- user support team (sets tasks, collects results, collects information about errors found by users)
“There are plants, some managers in the country, ” you will say :) Of course, a test manager is not a position, but a role. My employee enters this role then, as there is testing activity on his project. The rest of the time - he is a tester. He is "the hands of the other head." One person will never cope in time with testing a large update of a big game. Therefore, when he needs, the whole team is at his service. At the moment of the activity of his project - he is in charge, he sets tasks and collects the result.
In addition, if a massive attack is required (for example, to check the client’s build on all project servers), the user support team comes to the rescue: they know the game content and are ready to help. Here the tester again plays the role of the task designer.
Planning team work for the day occur on the morning stand-ups. The main issues that the team discusses are:
- “I will do it today”
- “Today I need 4 people to test this.”
- “I am not loaded today, who needs help?”
Owner's approach
In our company there is a concept “task driver”. It means “to be its owner”, to be the most interested in its decision. It is clear that the tester can not do much on their own:
- it cannot directly affect the early fix of the bug by developers
- it cannot directly affect installation acceleration
- he cannot control the engineer
etc.
But he is a driver for his tasks. He first of all needs to get a patch with a fix as soon as possible. This is he first needed release notes. That he needs to install an update on a test server on Friday, so that at the weekend beta testers can help us. And it is he who strives to solve these questions from the project manager, from the engineer, from me, in the end.
A person who, in case of difficulty, simply writes a letter, folds his hands and waits - it does not suit us.
My testers are their project managers. Project Testing Projects. Process owners
From the information that I drew from the guys who come for interviews to the position of a game tester, this approach to work, to put it mildly, is very rare in companies localizing games. Most often, testing departments are made up of performers who are given clear objectives and require clear results. All decisions are made by the head, all plans are written and approved by the head.
I consider it wrong to do it for everyone, so I teach my guys to do it. It develops them, and it gives me the freedom as a leader to engage in more strategic tasks.
Side-effect
True, this approach also has a side effect. The guys see all sides of the project, communicate with all project participants, with almost all departments in the company. They are seen, watched. And they grow.
Some grow to other departments. Two of my game testers went to junior project managers. At the same time, one of them is on the project on which he worked, and the second one was bitten by the head of a neighboring project. Right now, he (my former tester) in Seoul is developing communication with a software company. Another one goes to the analytics department of gaming products, just now I am looking for a replacement for him. I am happy for them. So I work well.
And no less, I am happy for those for whom testing is the industry where they want to develop and benefit. They grow as testers, as test managers, and are not going to sharpen their skis.
At company level
Not all projects are tested by the testing department. In Innovov, our manager is fully responsible for his project. Including for its quality, and for its testing. I, as the head of the service department, am the owner of my direction, and in order to remain competitive, I have to provide the projects with a testing service with such quality that the leaders ordered it from me. That is, if suddenly my testers start to mess up, then the project manager has the right to refuse the services of my department and arrange the testing for themselves. Any methods.
Or he may not order it for other reasons. For example, in several small projects, testing is carried out by the project team itself. Because the update testing cycle is very small (a few hours), there is little content, and within the team it is easier to allocate time “Right now, everyone sat down and ran,” than to put a task in my department.
In another project, the manager does not give testing to us, because this activity is interesting to his employees, and they cope with it, at the same time studying new content, the knowledge of which they need to work.
In such projects we act as carriers of expertise: we can prompt, help write tests, suggest how to arrange artifacts correctly.
It's hard, but interesting. They scold us, and users tell us thanks. We fight with the system and are friends with the developers.
Thank you guys!
Next story will be about the direction of game testing. This is a story about my guys, about our processes, about our organization of work. I invite you on a word trip.
External environment
')
I'll start with a description of the external system. We have 11 game projects. They are of different sizes, they are developed by different companies (most often Korean), they have different processes, different life cycles, different frequency of updates, different quality and different initial quality requirements.
At first glance, it seems that Innova deals only with localization, and it would seem, what is there to test? They give us the finished product, test only the localization. But these different initial requirements for the quality of products play a role. Different development companies consider it normal to release a product on their (Korean) market with a certain number of known bugs. And this amount is different for everyone, as you understand.
We are trying to keep this number to a minimum in all our games. Because as soon as we release the game in Russia, it becomes “ours”. We think so.
Testing process
However, we are not testing the whole game. That would be wrong. In the ideal case, we really need to test only the localization and build. To this we add another thorough testing of the new functionality and beta testing by users.
That is, the test plan for the update looks like this:
- localization testing:
- - check lists, deadlines
- testing of new functionality:
- - checklists for checking, priorities, deadlines
- assembly testing:
- - Smoke tests, timing
- beta testing:
- - task for players, terms
I talked about the full cycle of testing localized games on the example of the Atlantic.
The process of interaction with developers depends on the project and on the company-developer. Bug-reports can be made out in BTS, on our or on their side, can be collected in Excel or Google-docs. The testers interact with the developers to fix them, but in some places all communication goes through the project manager. We adapt to the project, but make changes to the processes for their optimization.
Test environment
In addition to the combat servers (of which different games from one to thirteen), there is a Public Test Server system (PTS) and an internal test server (QA-stand). On the TCP can enter any player if desired. There are all the beta testing. In order to get there, you do not need any special difficulties, you do not even need a separate account, the one that you play on combat servers will do.
At the QA-stand, internal testing is carried out, before giving to the players on the PTS.
Not any game update goes all the way QA-booth -> PTS -> combat servers. Some updates can not be put on the TCP without hitting the combat system. These immediately go into battle after internal testing. Some, on the contrary, cannot be properly tested on the internal stand, and they immediately go outside to our beta testers.
The quality of the combat product
And yet, you have to release products with known bugs. And - it happens that with critical. For example, the latest major update of the legendary Lineage II High Five Part 3 project installed on the servers on December 28 brought a serious bug: randomly the game client crashed with a critical error on the teleport users.
This may depend on the fact that, for example, this bug manifests itself with the load on the servers. And then he pops up only in combat operation. We do not recreate the load in our testing cycle, because it is a development company. But, not always effective. Since updates are often launched at the same time as Korea and often before Europe, we run the risk of launching with unknown bugs.
This may depend, for example, on the scheduled launch of the update, which all players are waiting for.
This may depend on indefinite deadlines for solving the problem from the developers.
I’m sure that if they were able to arrange a survey among the players affected by the bug with the teleport in the High Five, they would agree to play without updating to this day, they would still have chosen the pre-New Year update.
Of course, known errors are reported to players. And the status of their decision is constantly updated. And, of course, they are unhappy with the answer “Sent to the company-developer, they are working on it”!
Team organization
Almost every project has a test manager. This is one of my guys who best understands this project, which plans to test the project, participates in the planning of installing updates. He knows the weak points in his project, and he knows what is most important for his audience.
His tasks include:
- planning the testing of the project (or its updates)
- organization of project testing (or its updates)
- - internal testing
- - external testing (using beta testers)
- informing all interested parties about the status of testing and the state of the product
The test manager interacts with the work:
- with the project manager (plan updates together, approve the launch)
- project engineer (information on installing patches, fix assembly errors)
- developers (reports bugs found, test results)
- colleagues-game testers (sets tasks, collects results)
- project editor (localization error message, their correction)
- community project manager (known errors, working with beta testers)
- user support team (sets tasks, collects results, collects information about errors found by users)
“There are plants, some managers in the country, ” you will say :) Of course, a test manager is not a position, but a role. My employee enters this role then, as there is testing activity on his project. The rest of the time - he is a tester. He is "the hands of the other head." One person will never cope in time with testing a large update of a big game. Therefore, when he needs, the whole team is at his service. At the moment of the activity of his project - he is in charge, he sets tasks and collects the result.
In addition, if a massive attack is required (for example, to check the client’s build on all project servers), the user support team comes to the rescue: they know the game content and are ready to help. Here the tester again plays the role of the task designer.
Planning team work for the day occur on the morning stand-ups. The main issues that the team discusses are:
- “I will do it today”
- “Today I need 4 people to test this.”
- “I am not loaded today, who needs help?”
Owner's approach
In our company there is a concept “task driver”. It means “to be its owner”, to be the most interested in its decision. It is clear that the tester can not do much on their own:
- it cannot directly affect the early fix of the bug by developers
- it cannot directly affect installation acceleration
- he cannot control the engineer
etc.
But he is a driver for his tasks. He first of all needs to get a patch with a fix as soon as possible. This is he first needed release notes. That he needs to install an update on a test server on Friday, so that at the weekend beta testers can help us. And it is he who strives to solve these questions from the project manager, from the engineer, from me, in the end.
A person who, in case of difficulty, simply writes a letter, folds his hands and waits - it does not suit us.
My testers are their project managers. Project Testing Projects. Process owners
From the information that I drew from the guys who come for interviews to the position of a game tester, this approach to work, to put it mildly, is very rare in companies localizing games. Most often, testing departments are made up of performers who are given clear objectives and require clear results. All decisions are made by the head, all plans are written and approved by the head.
I consider it wrong to do it for everyone, so I teach my guys to do it. It develops them, and it gives me the freedom as a leader to engage in more strategic tasks.
Side-effect
True, this approach also has a side effect. The guys see all sides of the project, communicate with all project participants, with almost all departments in the company. They are seen, watched. And they grow.
Some grow to other departments. Two of my game testers went to junior project managers. At the same time, one of them is on the project on which he worked, and the second one was bitten by the head of a neighboring project. Right now, he (my former tester) in Seoul is developing communication with a software company. Another one goes to the analytics department of gaming products, just now I am looking for a replacement for him. I am happy for them. So I work well.
And no less, I am happy for those for whom testing is the industry where they want to develop and benefit. They grow as testers, as test managers, and are not going to sharpen their skis.
At company level
Not all projects are tested by the testing department. In Innovov, our manager is fully responsible for his project. Including for its quality, and for its testing. I, as the head of the service department, am the owner of my direction, and in order to remain competitive, I have to provide the projects with a testing service with such quality that the leaders ordered it from me. That is, if suddenly my testers start to mess up, then the project manager has the right to refuse the services of my department and arrange the testing for themselves. Any methods.
Or he may not order it for other reasons. For example, in several small projects, testing is carried out by the project team itself. Because the update testing cycle is very small (a few hours), there is little content, and within the team it is easier to allocate time “Right now, everyone sat down and ran,” than to put a task in my department.
In another project, the manager does not give testing to us, because this activity is interesting to his employees, and they cope with it, at the same time studying new content, the knowledge of which they need to work.
In such projects we act as carriers of expertise: we can prompt, help write tests, suggest how to arrange artifacts correctly.
It's hard, but interesting. They scold us, and users tell us thanks. We fight with the system and are friends with the developers.
Thank you guys!
Source: https://habr.com/ru/post/114140/
All Articles