Layfkhaki manual testing on mobile phones from 2GIS - Report from the conference SQA Days 15
The article was prepared based on the report of Yulia Gorlova at the SQA Days-15 conference.
Presentation: www.slideshare.net/VLDCORP/ss-33705537
')
The task is to support as many different configurations as possible: in testing several platforms, for each platform several versions of the operating system, for each platform several screen sizes and resolutions. There are a lot of devices, and testing is only manual.
In this article I will talk about several techniques that allow you to transparently and simply solve this problem.
Baseline : manual testing only, without automation.
Problem: we want to qualitatively test the product, while meeting the deadlines.
Why is this a problem? 2GIS is a detailed and up-to-date directory of organizations with a detailed city map.
First, there is a lot of functionality in the mobile application. Secondly, a large number of users (8.5 million). Thirdly, we support the main four platforms (iOS, Android, Blackberry, MeeGo). The latter two are partially supported (this is porting versions from Android). Basically, the article will deal with iOS and Android.
We strive to provide maximum comfort to users, we try to ensure that everything works on all versions of operating systems, of course, with certain reasonable limitations.
The problem is aggravated by a small number of testers: two specialists are engaged in testing external products. Only manual testing is applied. So historically, because the application was originally created for Symbian, then expanded on WinCE, and later added Android. After that, they refused WinCE and Symbian, and added iOS. We have abandoned active support for WinCE and Symbia, but as applications they exist. All applications have common internal parts; there is a huge complex and ornate architecture, which when they wrote, did not lay it under automatic testing. Therefore, there are parts of the application that cannot be automated, or they will entail huge costs for implementation. We do not need and not profitable. Therefore, we continue to test manually.
And since we are a mobile application, we must be dynamic, respond quickly to everything that happens around, give people new features constantly; we try to keep to the minimum release dates. This is about one month. Faster, we can not, unfortunately, precisely because of the fact that we have a rather complicated interior, and features are quite a big thing that needs to be thoroughly tested. We can not be faster, but we do not want any longer either. We try to fit in a month, but sometimes it is quite difficult.
As a result, we have a standard problem that many people face: we want to meet deadlines and don’t sacrifice quality.
What is quality for us? This is a very broad concept, and all interpret it in their own way. When we think about quality, we present the following picture: that in the store (store) people only complain that 2GIS does not yet have their city. They have no other problems with the product. This is what we are striving for.
5 solutions that we use in your product.
How to interact with the device, so that in the end everything was good?
For example, we follow a simple rule: when a tester goes home, he should put all the devices he works with. Because otherwise in the morning: Android is dead, there is nothing to work with. It is necessary to avoid such situations. It would seem that this is elementary, but this thing helps save time and simplifies the process of work.
In testing uses more than 30 devices. We are trying for all major versions to keep their devices, and do not update them, because We value them. Therefore, many devices accumulate. And we also buy various top devices to make sure that everything works. Now we have about 37-40 devices in active work. And among them it is necessary to find what you need. This is an important task.
We have a big stand with devices. What to do with them? Go through each? We solved this problem by creating a spreadsheet in which all the main characteristics are indicated. There are certain fields in the tables. We enter each new device immediately into this table, we indicate its characteristics. Some characteristics have remained, as a legacy from past platforms, for example, a magnetic compass and a touchscreen (from Symbian). Some have appeared recently, for example, OpenGL support.
Among these fields, the operating system is the most important thing, so that with an accuracy of 3 digits we can find the exact version. Sometimes it is very important.
At the same time, we still practice such a thing as keeping a checklist. For example, someone from another department takes devices, we want them to be returned to us later. So that the devices are not lost, we, like real watchmen, began to write it down. What-where-who-where, and then discharged accordingly. If the device is taken for a long time, we put a certain mark in the table next to the name of the device; then it is immediately clear that this device is not there now.
Rack for storage devices.
The developers tried to optimize it and replace it with a cabinet, but do not agree. The cabinet is worse because it has doors, and you have to open them. And this is not very convenient. At the counter from the workplace you can see where that lies. Most likely, you all know them in person, for example, Androids are easy enough to distinguish, because they are all different. There are a lot of charge, and nothing more.
iPads
2nd and 3rd iPads cannot be distinguished. There are no notes, nothing is written. We tried to memorize the wallpaper, looked at the screens: retina or not retina. But in the end, it's all tired, and we came up with writing on the iPad itself on the lock screen to indicate all its characteristics. Poked one button and immediately found out everything about the device. Why we do not do this for the iPhone'v. Because they are easy and simple to distinguish. For example, how to distinguish 4 from 4s? Elementary. The 4s have a strip on the side of the case.
We turn to the process part.
How we organize the processes. First, we do not have a cross-platform application. For each platform we create our own application, it is assembled separately, and has its own separate parts. We decided so, because There is an experience that says that users love native design.
How do we understand this? We used to have a design that was developed for Symbian first. It was universal and was used on Android, WinCE, and Symbian, and we received a lot of feedback that it was uncomfortable. We did a complete redesign when iOS was added. It was clear that the old design can not be transferred to iOS. On Android, it remained about the same, took over the main features, the only thing we added is the standard drop-down menu. On iOS, we use elements familiar to users, such as a touchbar, for example. Accordingly, it turns out native design both there and there.
Secondly, the application is written in C ++. We do not write in Objective-C or Java, only C ++. It interacts with the system at the kernel level. We work with the kernel and system libraries, and since the systems are different, the applications are different.
For each platform we have our build server. This allows us to avoid any downtime in testing. Everyone probably faced a situation when you come to work in the morning, and the build server does not work for unknown reasons. In this case there is always a backup option: a second build server with another application that you can take and test. This is not much, but it optimizes the process of our work.
Here we have assembled the application on the buildserver, found the device we need, we want to install the assembly on it. To do this, there is the simplest option: take the phone, take the assembly, connect, throw, disconnect, install, run, everything seems good, but for a long time.
Now, if you have 10 devices and one assembly, and you need to install it on all devices, what should you do? Everyone to connect long and uncomfortable. We decided we wanted to do it quickly.
And we have screwed QR codes to all build servers. For each assembly, a QR code is generated, which can be downloaded by any reader, it will appear in your tray, you will poke it, it will be installed. Everything. It is much faster because you do not need to connect anything anywhere. It used to be only for Android, and now on iOS.
All set up, all set, ready for testing. What else could be the pitfalls?
Since there are two applications, it is necessary to test not one feature, but two. To do this, we usually parallelize this process, it allows us to find common bugs for the entire feature, so that the developer does not waste time duplicating his work. If the feature is shared, it will be repaired in both places at once. If it is implemented differently, but it manifests itself in the same way, the developer will fix it in two places at once and he will not have to return twice to the same context. We also find some specific problems. The result is a more thoroughly tested feature.
Everyone knows that there are many, many Androids. A huge number of devices, each has its own screen size, resolution, etc. And all these millions of devices must somehow be tested.
For millions of Android screens, we've made five standard themes. For each device, the optimal theme is automatically selected (along the long side of the screen, there are 5 of them, and we can test these 5 pieces). You can drive on the narrowest, widest screens and see that everything is displayed correctly.
It is important to ensure that we do not discharge the phone to the user. Everyone, of course, knows that Android has the ability to discharge the phone. Our task is not to aggravate the situation, so that the phone does not discharge even faster, because people use 2GIS.
We are planning to test battery consumption in advance, with a focus on new features and on old critical places. We try to carry out such testing before each release.
iPad. Background: when we just released full support for the iPad, it was only vertical. Users wrote that it was inconvenient to use and asked to turn on. Turning the screen on the iPad is not just one tick in the settings. Our layout is a bit non-standard, there were a lot of problems with the layout, this is a bottleneck, so we are laying in advance for testing. We try to include in the plan a free testing session on iPads with an emphasis on layout. We use free testing, because the code is complex, and often works on the principle: “I bought bread - the balcony door stopped opening on Tuesdays”. Therefore, just functional testing is not enough.
Pre-installation on Android. Android'a assembly is installed in the factory-manufacturers on the phone, as the default application. This is about ten vendors. In the process of testing, in addition to the build for the update server, the build for Google Play, you need to make more than 10 builds for vendors and ideally everything should be checked.
We screwed a buildserver script that assembles these assemblies with one click. Assemblies are obtained absolutely identical, only those places that differ from one another are different. We can just run the build, check this key, and since they are the same, we don’t have to test every single one. If they run, then they are correctly assembled. On several assemblies, we do a full run of the functional in order not to test it at all.
When we are finished with functional testing, we begin verification and regression. There is a problem with the App Store, there you can’t just pick up the app. It must be sent, 4 days on average, it will be in the queue and approve to get into the store. That is, first we do everything for the iOS assembly. Since the BlackBerry is not so critical, because he may come out late, because he is not a flagship. iOS is important to release on time. If it happens that the deadlines are tight, the bugfixing has been delayed, we first send the assembly to the App Store, and then we start the regression. If we do not find bugs, then everything is fine, and if we find it, we will have time to correct it, and we will not fall out of line.
If the release was supposed to be yesterday, we write a letter in the App Store with a request to download us without a queue. They sometimes allow this to be done. But it is better not to use this method, it is only for emergency situations.
Sometimes we need more devices than there is an opportunity to purchase and keep in the office. For example, when we released support for OpenGL on Android, we encountered the problem that we are very dependent on hardware and firmware.
We had a sad situation on Android, that we worked quickly on old and weak devices, and we worked slowly on large and high-end ones, because the screens are large and we need to process a lot of pixels, and this caused problems. We solved these problems by connecting hardware acceleration.
There were many difficulties, so we came to beta testing. We made it a two-milestone.
The first stage was carried out within the company. It was necessary to collect the biggest problems. This stage justified its goal: we have compiled a black list of devices that will not support OpenGL in our application. They repaired everything they could and released an external beta.
External beta - build in the state of final readiness published on Google Play. He provided the opportunity to publish public beta releases. To do this, you had to create a group on Google+, place the application, give the link to users, and they could download the application.
We posted an article on Habré that needs help in beta testing. About 3.5 thousand subscribers took part, they brought many specific problems. We debugged everything we could.
This gave us additional bonuses, for example, when specific problems occur, users return and tell us about these problems.
Our application is in the top, and we want to hold positions, so it is necessary not only to quickly test, but also to quickly respond to problems. If something went wrong, you can see at once in the markets. We have a lot of users, and 1% of people with a problem are about 85 thousand users with a problem.
A typical review in the market (usually does not convey the essence): “Everything is broken, fix it!”.
Implemented communication with technical support through the application. You can go to “About the program” and write to the developer, and insert a log with a bunch of useful information into the letter. We "treat" even special cases for a particular device with a specific firmware. We send the user a new assembly, and everything starts to work.
As a result, all this information may be useful to you if you are working with the kernel, you have Android, and not only Android, and if there is no automation. Reduce the associated time costs accordingly.
Presentation: www.slideshare.net/VLDCORP/ss-33705537
')
The task is to support as many different configurations as possible: in testing several platforms, for each platform several versions of the operating system, for each platform several screen sizes and resolutions. There are a lot of devices, and testing is only manual.
In this article I will talk about several techniques that allow you to transparently and simply solve this problem.
Baseline : manual testing only, without automation.
Problem: we want to qualitatively test the product, while meeting the deadlines.
Why is this a problem? 2GIS is a detailed and up-to-date directory of organizations with a detailed city map.
First, there is a lot of functionality in the mobile application. Secondly, a large number of users (8.5 million). Thirdly, we support the main four platforms (iOS, Android, Blackberry, MeeGo). The latter two are partially supported (this is porting versions from Android). Basically, the article will deal with iOS and Android.
We strive to provide maximum comfort to users, we try to ensure that everything works on all versions of operating systems, of course, with certain reasonable limitations.
The problem is aggravated by a small number of testers: two specialists are engaged in testing external products. Only manual testing is applied. So historically, because the application was originally created for Symbian, then expanded on WinCE, and later added Android. After that, they refused WinCE and Symbian, and added iOS. We have abandoned active support for WinCE and Symbia, but as applications they exist. All applications have common internal parts; there is a huge complex and ornate architecture, which when they wrote, did not lay it under automatic testing. Therefore, there are parts of the application that cannot be automated, or they will entail huge costs for implementation. We do not need and not profitable. Therefore, we continue to test manually.
And since we are a mobile application, we must be dynamic, respond quickly to everything that happens around, give people new features constantly; we try to keep to the minimum release dates. This is about one month. Faster, we can not, unfortunately, precisely because of the fact that we have a rather complicated interior, and features are quite a big thing that needs to be thoroughly tested. We can not be faster, but we do not want any longer either. We try to fit in a month, but sometimes it is quite difficult.
As a result, we have a standard problem that many people face: we want to meet deadlines and don’t sacrifice quality.
What is quality for us? This is a very broad concept, and all interpret it in their own way. When we think about quality, we present the following picture: that in the store (store) people only complain that 2GIS does not yet have their city. They have no other problems with the product. This is what we are striving for.
5 solutions that we use in your product.
Administration
How to interact with the device, so that in the end everything was good?
For example, we follow a simple rule: when a tester goes home, he should put all the devices he works with. Because otherwise in the morning: Android is dead, there is nothing to work with. It is necessary to avoid such situations. It would seem that this is elementary, but this thing helps save time and simplifies the process of work.
In testing uses more than 30 devices. We are trying for all major versions to keep their devices, and do not update them, because We value them. Therefore, many devices accumulate. And we also buy various top devices to make sure that everything works. Now we have about 37-40 devices in active work. And among them it is necessary to find what you need. This is an important task.
We have a big stand with devices. What to do with them? Go through each? We solved this problem by creating a spreadsheet in which all the main characteristics are indicated. There are certain fields in the tables. We enter each new device immediately into this table, we indicate its characteristics. Some characteristics have remained, as a legacy from past platforms, for example, a magnetic compass and a touchscreen (from Symbian). Some have appeared recently, for example, OpenGL support.
Among these fields, the operating system is the most important thing, so that with an accuracy of 3 digits we can find the exact version. Sometimes it is very important.
Watchman syndrome
At the same time, we still practice such a thing as keeping a checklist. For example, someone from another department takes devices, we want them to be returned to us later. So that the devices are not lost, we, like real watchmen, began to write it down. What-where-who-where, and then discharged accordingly. If the device is taken for a long time, we put a certain mark in the table next to the name of the device; then it is immediately clear that this device is not there now.
Rack for storage devices.
The developers tried to optimize it and replace it with a cabinet, but do not agree. The cabinet is worse because it has doors, and you have to open them. And this is not very convenient. At the counter from the workplace you can see where that lies. Most likely, you all know them in person, for example, Androids are easy enough to distinguish, because they are all different. There are a lot of charge, and nothing more.
iPads
2nd and 3rd iPads cannot be distinguished. There are no notes, nothing is written. We tried to memorize the wallpaper, looked at the screens: retina or not retina. But in the end, it's all tired, and we came up with writing on the iPad itself on the lock screen to indicate all its characteristics. Poked one button and immediately found out everything about the device. Why we do not do this for the iPhone'v. Because they are easy and simple to distinguish. For example, how to distinguish 4 from 4s? Elementary. The 4s have a strip on the side of the case.
We turn to the process part.
Build Management
How we organize the processes. First, we do not have a cross-platform application. For each platform we create our own application, it is assembled separately, and has its own separate parts. We decided so, because There is an experience that says that users love native design.
How do we understand this? We used to have a design that was developed for Symbian first. It was universal and was used on Android, WinCE, and Symbian, and we received a lot of feedback that it was uncomfortable. We did a complete redesign when iOS was added. It was clear that the old design can not be transferred to iOS. On Android, it remained about the same, took over the main features, the only thing we added is the standard drop-down menu. On iOS, we use elements familiar to users, such as a touchbar, for example. Accordingly, it turns out native design both there and there.
Secondly, the application is written in C ++. We do not write in Objective-C or Java, only C ++. It interacts with the system at the kernel level. We work with the kernel and system libraries, and since the systems are different, the applications are different.
For each platform we have our build server. This allows us to avoid any downtime in testing. Everyone probably faced a situation when you come to work in the morning, and the build server does not work for unknown reasons. In this case there is always a backup option: a second build server with another application that you can take and test. This is not much, but it optimizes the process of our work.
Here we have assembled the application on the buildserver, found the device we need, we want to install the assembly on it. To do this, there is the simplest option: take the phone, take the assembly, connect, throw, disconnect, install, run, everything seems good, but for a long time.
Now, if you have 10 devices and one assembly, and you need to install it on all devices, what should you do? Everyone to connect long and uncomfortable. We decided we wanted to do it quickly.
And we have screwed QR codes to all build servers. For each assembly, a QR code is generated, which can be downloaded by any reader, it will appear in your tray, you will poke it, it will be installed. Everything. It is much faster because you do not need to connect anything anywhere. It used to be only for Android, and now on iOS.
Balanced testing
All set up, all set, ready for testing. What else could be the pitfalls?
Since there are two applications, it is necessary to test not one feature, but two. To do this, we usually parallelize this process, it allows us to find common bugs for the entire feature, so that the developer does not waste time duplicating his work. If the feature is shared, it will be repaired in both places at once. If it is implemented differently, but it manifests itself in the same way, the developer will fix it in two places at once and he will not have to return twice to the same context. We also find some specific problems. The result is a more thoroughly tested feature.
Everyone knows that there are many, many Androids. A huge number of devices, each has its own screen size, resolution, etc. And all these millions of devices must somehow be tested.
For millions of Android screens, we've made five standard themes. For each device, the optimal theme is automatically selected (along the long side of the screen, there are 5 of them, and we can test these 5 pieces). You can drive on the narrowest, widest screens and see that everything is displayed correctly.
It is important to ensure that we do not discharge the phone to the user. Everyone, of course, knows that Android has the ability to discharge the phone. Our task is not to aggravate the situation, so that the phone does not discharge even faster, because people use 2GIS.
We are planning to test battery consumption in advance, with a focus on new features and on old critical places. We try to carry out such testing before each release.
iPad. Background: when we just released full support for the iPad, it was only vertical. Users wrote that it was inconvenient to use and asked to turn on. Turning the screen on the iPad is not just one tick in the settings. Our layout is a bit non-standard, there were a lot of problems with the layout, this is a bottleneck, so we are laying in advance for testing. We try to include in the plan a free testing session on iPads with an emphasis on layout. We use free testing, because the code is complex, and often works on the principle: “I bought bread - the balcony door stopped opening on Tuesdays”. Therefore, just functional testing is not enough.
Pre-installation on Android. Android'a assembly is installed in the factory-manufacturers on the phone, as the default application. This is about ten vendors. In the process of testing, in addition to the build for the update server, the build for Google Play, you need to make more than 10 builds for vendors and ideally everything should be checked.
We screwed a buildserver script that assembles these assemblies with one click. Assemblies are obtained absolutely identical, only those places that differ from one another are different. We can just run the build, check this key, and since they are the same, we don’t have to test every single one. If they run, then they are correctly assembled. On several assemblies, we do a full run of the functional in order not to test it at all.
When we are finished with functional testing, we begin verification and regression. There is a problem with the App Store, there you can’t just pick up the app. It must be sent, 4 days on average, it will be in the queue and approve to get into the store. That is, first we do everything for the iOS assembly. Since the BlackBerry is not so critical, because he may come out late, because he is not a flagship. iOS is important to release on time. If it happens that the deadlines are tight, the bugfixing has been delayed, we first send the assembly to the App Store, and then we start the regression. If we do not find bugs, then everything is fine, and if we find it, we will have time to correct it, and we will not fall out of line.
If the release was supposed to be yesterday, we write a letter in the App Store with a request to download us without a queue. They sometimes allow this to be done. But it is better not to use this method, it is only for emergency situations.
Beta testing
Sometimes we need more devices than there is an opportunity to purchase and keep in the office. For example, when we released support for OpenGL on Android, we encountered the problem that we are very dependent on hardware and firmware.
We had a sad situation on Android, that we worked quickly on old and weak devices, and we worked slowly on large and high-end ones, because the screens are large and we need to process a lot of pixels, and this caused problems. We solved these problems by connecting hardware acceleration.
There were many difficulties, so we came to beta testing. We made it a two-milestone.
The first stage was carried out within the company. It was necessary to collect the biggest problems. This stage justified its goal: we have compiled a black list of devices that will not support OpenGL in our application. They repaired everything they could and released an external beta.
External beta - build in the state of final readiness published on Google Play. He provided the opportunity to publish public beta releases. To do this, you had to create a group on Google+, place the application, give the link to users, and they could download the application.
We posted an article on Habré that needs help in beta testing. About 3.5 thousand subscribers took part, they brought many specific problems. We debugged everything we could.
This gave us additional bonuses, for example, when specific problems occur, users return and tell us about these problems.
Current support
Our application is in the top, and we want to hold positions, so it is necessary not only to quickly test, but also to quickly respond to problems. If something went wrong, you can see at once in the markets. We have a lot of users, and 1% of people with a problem are about 85 thousand users with a problem.
A typical review in the market (usually does not convey the essence): “Everything is broken, fix it!”.
Implemented communication with technical support through the application. You can go to “About the program” and write to the developer, and insert a log with a bunch of useful information into the letter. We "treat" even special cases for a particular device with a specific firmware. We send the user a new assembly, and everything starts to work.
As a result, all this information may be useful to you if you are working with the kernel, you have Android, and not only Android, and if there is no automation. Reduce the associated time costs accordingly.
Source: https://habr.com/ru/post/238707/
All Articles