Geekly Articles each Day
📜 ⬆️ ⬇️

What do mobile QA and octopus have in common?



Hello! I am Katya, and I am a workaholic, the tester of the most popular dating app.

So early in the morning, you are mobile QA. You come to work, brew strong coffee and want to take a couple of mobile devices to test a new feature, knowing what torments of choice are ahead of you. What will these devices be?
')
Sooner or later, each mobile tester wonders how many devices to test the new functionality in order to catch the maximum number of device-dependent bugs, spending a minimum of time. Auto tests are not written yet, in front of you are absolutely new features. And if there is at least some clarity with iOS, and the list of devices is limited, then Android has “bred” to hell. You'd be surprised, but for happiness you need only three or four Android devices. I want to tell how from the point of view of an experienced tester to choose them.


Why test on different devices and how dangerous fragmentation


If your application is just starting the way to the top of the app market, then it would seem that only one Google reference device should be enough for testing. However, if thousands of people use your application every day, then you will have to think about expanding the "zoo" of devices and face-to-face with the features and problems associated with the variety of Sam Android devices.

Let your application work fine under the emulator, or on your unreal blue Pixel, but in the statistics of popular Android devices there are none that occupy more than 10% of the market, unlike, for example, from iOS devices. So such a point testing does not guarantee the absence of problems for most users. Different devices have different “improvements” from manufacturers, different OS versions and iron characteristics - all this can cause device-dependent bugs.

The more popular your application, the more devices you need for a full scan. And the more functionality in your application, the more likely that you will encounter problems. And the way out in this situation is to save devices.


Have you tried playing jenga from smartphones?

Our company has a large fleet of mobile devices (at the moment we have about 60 different Android smartphones and tablets) and a stand-up automator where auto tests run on real devices.


The tester is sleeping, auto tests work!

What bugs are we looking for?


The bugs that you may encounter on some devices, but not on others, can be divided into three categories:


Most often, these problems are not interrelated, so pairwise testing is not necessary.
Do you know everything about such bugs? Then go straight to the section on device selection .

Manufacturers bugs


Manufacturers of modern mobile devices love to reinvent the wheel. This applies to all kinds of firmware and interface modifications. At best, you will simply encounter bugs with the display of the interface that the manufacturer has changed for certain devices. For example, such bugs may appear due to custom fonts or increasing their default size.

image Due to this implementation, the UI begins to distort (as on the screen), which means that developers will either have to take this into account, or use their own font and not allow third parties to use it. Previously, this problem was encountered only on cheap Chinese devices, but now it has reached Korean flagships (I don’t want to give specific examples and make manufacturers anti-advertising). Should I take into account such devices, statistics on your application will tell.
Fantastic K and where to look for her work and education

In the worst case (for developers), the manufacturer will already be associated with more interesting deep-seated problems and application crashes due to the use of some kind of self-written libraries and methods. In addition, bugs can be associated with custom applications with which your software will interact. Such as, for example, the camera or the built-in file manager.


image In this case, the developers have a choice: to write their own tool or to send users from your ideal application to the manufacturer’s subsystems, from where it will not return as before. Problems with file managers are mainly found on Chinese devices, you may not need to worry about them. But with the camera complexity arise more often.
Persistent health care!

Any features of the camera that are implemented by the manufacturer can affect the operation of your application and lead to crashes, problems with autofocus, an inverted image, and so on. As in the previous case, you will have to make a decision based on the percentage of affected users, and here there are two options: either write your own custom camera, or correct photos on the fly.
Thus, for quality testing, we need a device with a modified, non-vanilla firmware and a lot of our own applications and presets from the manufacturer. Perfect Samsung and Sony devices.

Android bugs


Developers love to use everything new and interesting, but they don’t always like to google compatibility tables of different APIs and OS versions. Developers sleep soundly, but we, testers, have no peace - a large number of potential bugs are associated with incompatibility of versions.

Here we just have to arm ourselves with devices with the most popular OS versions. They are usually not very much. My favorite site Android Developers will help you with the choice.

It is worth considering your own statistics on devices, because your application can leave its mark on the user base.

image The search for bugs for the latest version of the OS is certainly very important: in about a year this version will become the most popular. Still, your application should not fall on Google's native devices. On the other hand, for the time being, there are few devices with the latest Nougat OS version at the current moment, and updates are rarely rolled out. Therefore, one can either entrust the search for such bugs to autotests, or sometimes carry out a manual run of regression tests on such devices. The main thing is not to miss any potential danger, such as the aggressive Doze Mode and its consequences , especially if your application monitors your Internet connection. Not only developers, but also testers should follow the changes that have occurred in the latest versions of Android.
Not all Androids love ovals!

Another group of problems arises on weak devices. The Android operating system does not impose restrictions on the hardware used, which many manufacturers did not fail to take advantage of in order to save on everything. Almost any application will fall if there is a shortage of memory on a weak mobile device. It’s impossible to fully defend against this, but it’s worth considering, especially if you use some “voracious” features like video calls or video recording.
For testing, you definitely need a weak device - you need to ensure that the application on it functions adequately. Interesting cases will be tests of work with a small amount of RAM or lack of internal memory.

Resolution Curves and Modes



Similar or different?

Strictly speaking, these problems do not always relate to device-dependent, more often these are just flaws related to the interface of your application. However, the popularity of those or other resolutions and screen sizes is closely related to the popularity of devices, and these indicators are constantly changing. Also, manufacturers can add something interesting, for example, the ability to select Pixel Density, as in OnePlus 3 (and in Nougat, by default, in general, right from the settings).

Despite the statistics on popular resolutions and screen sizes, the tablet should be used for testing: it allows you to catch such problems as forgotten in form interface elements (which will be out of the screen on a small screen, but will be visible on a large screen), forgotten alignment for texts or pictures that on a small screen will appear in place. Perhaps these errors are not very critical, but they can badly affect the reputation of your application.

Results: choose devices




As a rule, the tester starts working with morning coffee positive tests, so first you need the perfect device for positive testing. The most suitable devices for this are the most popular devices of the users of your application. Device statistics can be found in the Google Play Store Developer Console or Google Analytics. I will share other useful links at the end of the post.

For positive tests, native google devices are also well suited. They will give you clean tests, which means that if any errors are detected on other devices, it will be immediately clear that they are somehow connected to this device.

We will choose this way:


Plus: choose the most popular tablet (as an additional device).
As a rule, only three devices cover most of the various differences.

This is our choice!


Now Badoo has more than 333 million registered users worldwide, and the number of installations on the Android platform has exceeded 100 million. Let's run this algorithm on the data relevant to Badoo, while the coffee is cooling.

The most popular devices (according to our own statistics; I will not indicate the exact percentages, since this value changes very quickly, the popularity here is not more than 5% for each of the devices):


The most popular versions of Android (this data from Google is very close to our own statistics):




The most common screen sizes and resolutions of devices (this data from Google is also very close to our own statistics):



Decryption of this table.

Popular manufacturers of mobile devices (statistics on popular manufacturers depends on the countries in which your application is currently popular. For us it is):


If we analyze all this data, then, for the time being, the ideal device for positive tests for us is the Samsung Galaxy S6 with Android 6.0 and XXHDPI large / normal screen .

Since the Samsung Galaxy S3 with Android 4.4 and XHDPI normal screen ( UPD: SGS3, which is S3 Neo) is a popular device with the smallest screen ( UPD: compared to SGS4 and SGS6) and, while being quite weak (this is a rather old device of 2014 release), then it is great for negative testing.

Plus, we take Samsung Galaxy S4 with Android 5.0 and XHDPI normal screen , it will ideally complement the set with a popular version of Android 5.0 and an average screen size.

Based on these three devices, we get the following coverage of Android fragmentation:

- Android versions: 26.3 + 24 + 10.8 = 61.1%, but you need to add 23.2 here and get as much as 84.3% (because bugs that are found only on version 5.0, but not found 5.1 - a rarity);
- screen size: 6.7+ 88.3 = 95% ;
- screen resolution: 32.4 + 15.8 = 48.2%, but you need to add 38.8 here and get as much as 87% ( UPD: because we we use different SGS3 models for testing, SGS3, SGS3 Neo and SGS3 mini).

That is, with the help of only three or four devices, we get a coverage of more than 80% for all the differences between Android devices that interest us. If you fail to cover about 80% for any indicator, then you need to add another device. But as a rule, three is enough.

Good additions can be:
- the tablet;
- a device with pure Android;
- A device that will increase coverage for popular device manufacturers.

I usually add the Asus Nexus 7 (2013) as a tablet with pure Android. Also suitable for Huawei MediaPad M2.

image That is, four devices are the perfect set of Android testers, which means the octopus will be the perfect testers (two tentacles for each device)!

Take the Asus Nexus 7, Samsung Galaxy S3, S4 and S6 and go to drink coffee!
As it turned out, everything is not so scary, but the coffee is not even cold.

Related Links



PS Stickers from the post.

Ekaterina Mikheeva, Android QA

Source: https://habr.com/ru/post/317964/

More articles:


All Articles