Testing mobile apps: tips & tricks
Our new article is a list of recommendations and tips. From it you will learn:
For beginners, testers can help grow faster, and more experienced ones can streamline knowledge. The article will also be useful to developers, product managers and project managers, in a word - to everyone who wants to improve the quality of the product and establish interaction between departments.
1. Use the principles of heuristics and mnemonics - they help keep in mind all aspects that need to be considered when testing a feature or application.
2. Screenshots, logs and videos are the best arguments of the tester!
Unfortunately, with the logs of "communication" with the server, often not everything is as smooth as with client logs. Usually they are added rather for the convenience of the developer when debugging work with the server, than for the work of the tester.
Logs for the release version of the application must be turned off for reasons of speed and security.
')
3. Use "monkeys" to search for crashes and freezes while you are testing the functionality more intelligently. It is most effective to combine monkeys and telemetry removers to speed up problem localization, for example, with TestFairy. Recently, TestFairy also supports iOS, but for now the functionality is limited.
4. In order to feel more comfortable before releases, use the safety net in the form of beta versions. Two or three people will not be able to physically cover all combinations of cases and various devices (especially for Android), and you can get help from beta users all over the world, which will allow you to unload the testing team. For convenience, I highly recommend wrapping the beta version into the TestFairy shell.
5. It is very convenient to have a debug menu with functions that facilitate the work of developers and testers (especially the automation team). For example, with the ability to simulate responses from the server, open certain users, set certain flags, clear and lose sessions, reset the cache. A multi-functional debug menu has been created in our mobile applications, and I can’t imagine either manual or automated testing without it.
6. iOS and Android development menus are your best friend. Enable for iOS , Android .
In iOS, for example, it allows you to:
The Android picture is even more rosy - there are a lot of settings for any needs: from displaying CPU and memory usage to changing the animation speed of the interface.
7. If the application supports portrait and landscape mode, then pay special attention to changing the orientation of the device. This can cause crashes, memory leaks, a return to a previous state.
8. Switch between screenshots many times.
It is best to switch between screenshots while the application is interacting with the network:
9. Do not neglect testing on emulators and simulators - a very handy thing that allows you to facilitate some checks. For example, for iOS, it is much easier to test changes in location and background location updates, as well as simulate memory warning for hot keys, including slow animations. For Android, you can configure the exotic characteristics of the "iron": the screen resolution, pixel density, the amount of RAM, heap size, the size of the internal and external memory.
10. Fill the device RAM before running the application. This will help, firstly, to conduct stress testing and check the speed of work, and secondly, to check the preservation and resumption of the application state (where we will return after the application is minimized, whether all the necessary services will start).
11. Run the application under the debugger. Why?
1. The application should work stably with:
Use chain cases "network problems", to the maximum using:
2. If you need a proxy server, the easiest way is to use CharlesProxy (there are instructions for devices and emulators on iOS and Android, supports binary protocol, rewrite, throttling traffic, and generally worth every nested cent ) or Fiddler (it's free).
1. If there is a third-party service - he will fail. The recent accident at FB has affected the work of some applications and sites. For example, a couple of versions ago, the Habrahabr application “shared” articles with UI blocking without an activity indicator. At the moment when Facebook or the Internet “braked”, the “sharing” hung the entire application until the process was completed.
It is better to identify the maximum number of abnormal situations in advance and consider steps to eliminate them: provide for handling unexpected responses (errors, garbage, no response, empty response) from third-party services with indication of a problem to the user, add timeouts to the necessary requests, etc.
2. If there are third-party libraries, they will definitely cause problems. In particular, Twitter, PayPal, Facebook quite often contain bugs. As an example, in one of the versions of the Twitter SDK, there was a crash when receiving 503 errors from its own backend — the library simply crashed and dragged the application along. The Facebook SDK also often crashes on Android (you can see a process called com.facebook.katana from time to time in crash alerts).
3. Parsers URI and data should try to take into account all sorts of surprises - there were cases when the auto check of the validity of files and (or) URI broke off, and applications had to pull out the information bit by bit.
For example:
In all these cases, the application may not parse the unexpected response and fall.
In addition to working on the stability of the application, in the above cases it is important to give the user some visual response: alert, toast notification, placeholders instead of data, etc.
4. If your application updates the data on static or easily generated URLs, then you can use Dropbox or Google Drive until the logic to the server is ready or run. Uploading or updating files directly on the device is a dubious pleasure.
Therefore, we did this:
5. Do not forget to check the migration of data and caches when updating the application. It is important to remember that users can skip versions and it will be necessary to check for updates of older versions. As an example, I can give a crash at the start of the LinkedIn application in June 2015: some users could not launch the application until they released a new version (fortunately, it was released on the same day).
1. Set the custom screen resolution of the emulator - this will reveal problems with the layout, if you have a certain lack of devices or need to check whether the layout is written correctly. Also, the screen resolution and pixel density can be edited via ADB and on a physical device, for example, on Nexus 10 .
2. If the keyboard is redefined (using custom), then pay extra attention to this. There are both keyboard errors that cannot be bypassed, and logical or graphical errors.
3. Staged rollout will make it easy to find problems that might have been missed when testing the release version: you can roll out 5-10% and monitor graphics and crashes, if necessary, roll back or re-allocate the version with fix.
4. Use do not keep activities when testing and make sure applications are ready for unexpected activity terminations, which can lead to crashes or data loss.
1. Check if the standard gestures are overridden. For example, when you activate "Universal Access", additional gestures are activated, they may conflict with the gestures of your application (for example, a three- and four-finger gesture).
2. Also pay attention to third-party keyboards. For example, in iOS9 there is a bug that will cause the application to crash if you enter text using a third-party keyboard in a modal window in WebView.
3. Show the developers the rollout.io service, which allows you to patch some crashes on production, override parameters, show apologies with apologies, or make some buttons inactive. He has already saved us more than once.
4. For interactive testing of the layout or checking that all screens have been removed from the hierarchy, you can use the standard Xcode or Spark Inspector , RevealApp tools .
5. Ask to integrate the Memory Warning call into the debugger menu. It is usually hanged on a certain gesture (tap with several fingers, pressing the status bar or navigation) or the volume buttons . This is necessary to check the adequacy of the behavior of the application during Memory Warning, if it cleans up the resources and how well it is done. For example, we had an unpleasant bug, when, after Memory Warning, our Image service unloaded the image from memory and the screen remained blank.
These tips will help you move faster in testing mobile applications and will teach you to bypass the pitfalls of the relationship with developers.
1. Enter a Pre-QA culture. Before sending the ticket to the review, sit down with the developer for his car and test the feature for 5-10 minutes under the debugger on one or two devices - most of the most stupid mistakes will be found right away. It also allows developers to teach basic testing skills: at a minimum, they will repeat actions for you, at most they will penetrate and test more meaningfully. No one wants to make stupid mistakes and put them on public display.
2. At least fluently look through the diffs of each branch (features) and ask as many questions as possible to the developers.
Thus, you, first of all, raise your prestige as a tester - you try to understand the code and areas that are affected by this feature. Even now, mobile application testers are sometimes perceived by developers as monkeys who poke into the phone and juggle them in order to “drop” the application.
If there are no free developers, you can act as a reviewer: the developer, while explaining the work, features sometimes find flaws or cases that he did not take into account.
Secondly, you will gradually begin to learn a programming language and will understand better what is happening “under the hood” of the application.
3. Examine the life cycle of the application's entities and the application itself (Activity for Android 1 , 2 , 3 ; ViewController for iOS 1 , 2 , 3 ) to understand which state the application screen can go from and to itself. The better you know how the application works from the inside, the more fully you will be able to test it.
4. If you have apps for iOS and Android, then it is important to maintain the right balance of resources for testing them.
Errors in applications threaten to re-lay, which has certain consequences. At the same time, the price of an error in the Android platform is often lower.
But, on the other hand, iOS applications can be tested much faster, because The ecosystem is not as fragmented as Android.
1. What should I do if the hour X sneaks up at the most inopportune moment and an unstable version of the application hits production? We use an update screen system to speed up the user migration process. Such a system is useful:
Our update system works in two modes :
Not all users will have the physical ability to update the application, so for certain versions the logic is intentionally disabled. Take, for example, iPhone4 users who will not be able to upgrade to iOS8, while we in the application plan to stop supporting iOS7.
2. Monitoring of the most important application metrics in production is required:
3. Sometimes problems arise only for certain users with certain devices or in certain countries. For example, Vodafone UK had problems with WebP images. To test such cases, you can use cloud solutions for renting devices: DeviceAnywhere (paid service), PerfectoMobile (paid service), Samsung Device Lab (free, but there is a point system that replenishes with time).
4. Also, do not forget about time zones and user locations. Perhaps your application is not designed to work in certain countries (although it was posted there by mistake, or you, as a user, came to a time in another country). Location on iOS can be "fake" in the settings of the simulator (Debug> Locations), and on Android there are applications that allow you to do this .
If the application works with data and there are several data centers in different time zones, you need to make sure that everything works correctly and there are no collisions when switching between data centers.
5. Learn how to update and “downgrade” firmware - platforms are fragmented, especially Android and Blackberry. Cloud services are good, but they cost money, so not all companies have the opportunity to use them due to lack of funding or security policy.
6. Found a bug after calculating the features, but do you have to re-upload the new version? In this case, you can help enable, disable or modify features on the fly. In our applications a lot of features can be turned off from the server side directly or, depending on the team’s solution, using a special interface.
Such a list of recommendations, approaches and tools, in our opinion, can be useful for both beginners and experienced mobile application testers. I hope the developers and managers will also learn something useful for their purposes.
Compiling this list, my colleagues and I were based on our own experience and would be happy to see your opinion in the comments.
Alexander Host
Lead Mobile QA Engineer
- how to facilitate the process of testing mobile applications in general;
- the specifics of working with the network, internal and external services, iOS and Android platforms;
- what process decisions and changes will allow you to grow faster and introduce a testing culture in the development department;
- What are some useful tools and solutions for testing, debugging, monitoring and migrating users?
For beginners, testers can help grow faster, and more experienced ones can streamline knowledge. The article will also be useful to developers, product managers and project managers, in a word - to everyone who wants to improve the quality of the product and establish interaction between departments.
How to facilitate the testing process?
1. Use the principles of heuristics and mnemonics - they help keep in mind all aspects that need to be considered when testing a feature or application.
- A set of mind maps for all occasions: test insane , ministry of testing
- Heuristics, mnemonics: I SLICED UP FUN (my favorite), COP FLUNG GUN , SFDPOT , LONG FUN CUP
2. Screenshots, logs and videos are the best arguments of the tester!
Unfortunately, with the logs of "communication" with the server, often not everything is as smooth as with client logs. Usually they are added rather for the convenience of the developer when debugging work with the server, than for the work of the tester.
- Ask client and server developers to display all server requests and responses from it in a convenient and clear interface for viewing logs. So it will be easier to analyze requests and responses from the server, identify duplicates, find more convenient ways to update data. For example, to update a part of a profile, a developer can re-request the entire profile instead of using a more lightweight request. In situations where it is not clear where the problem is, a combination of server and client logs will in most cases help to deal with the problem faster.
- Learn how to add client logs yourself (however, the developer’s help may still be needed) - an excess of logs is also harmful and pollutes the console.
Logs for the release version of the application must be turned off for reasons of speed and security.
')
3. Use "monkeys" to search for crashes and freezes while you are testing the functionality more intelligently. It is most effective to combine monkeys and telemetry removers to speed up problem localization, for example, with TestFairy. Recently, TestFairy also supports iOS, but for now the functionality is limited.
- Android: Application / UI Monkey Exerciser
- iOS: UIAutoMonkey / CrashMonkey (based on UIAutoMonkey). Unfortunately, AntEater completely abandoned - error 404 was on the site. Last time I fixed the bugs and updated for iOS8 ( if anyone is interested ), the developers did not dare to transfer the source code to open source, although they asked them repeatedly.
4. In order to feel more comfortable before releases, use the safety net in the form of beta versions. Two or three people will not be able to physically cover all combinations of cases and various devices (especially for Android), and you can get help from beta users all over the world, which will allow you to unload the testing team. For convenience, I highly recommend wrapping the beta version into the TestFairy shell.
- Android with a beta program is still much better than iOS: you can invite (or wait for bids from) users via Google+. The number of beta users is limited to 10,000.
- IOS with TestFlight (which Apple bought) has some artificial limitations: a maximum of 2000 users, a serious “review” of the first beta version. Distribution services can also be used for beta programs. Excellent overview of distribution services: 1 , 2 , 3 , 4 .
5. It is very convenient to have a debug menu with functions that facilitate the work of developers and testers (especially the automation team). For example, with the ability to simulate responses from the server, open certain users, set certain flags, clear and lose sessions, reset the cache. A multi-functional debug menu has been created in our mobile applications, and I can’t imagine either manual or automated testing without it.
6. iOS and Android development menus are your best friend. Enable for iOS , Android .
In iOS, for example, it allows you to:
- Enable Network Link Conditioner ;
- enable logging of traffic and energy consumption;
- more convenient to test iAd advertising.
The Android picture is even more rosy - there are a lot of settings for any needs: from displaying CPU and memory usage to changing the animation speed of the interface.
7. If the application supports portrait and landscape mode, then pay special attention to changing the orientation of the device. This can cause crashes, memory leaks, a return to a previous state.
8. Switch between screenshots many times.
- On iOS, the correctness of working with memory is checked (have you turned to the wrong memory location; has the screen not been updated while it was invisible), memory leaks.
- There may be memory leaks on Android. the previous activity (activity) is holding something.
It is best to switch between screenshots while the application is interacting with the network:
- requests must be canceled if they are not completed;
- server response to a screen that is deleted from memory (invisible) should not "crash" the application.
9. Do not neglect testing on emulators and simulators - a very handy thing that allows you to facilitate some checks. For example, for iOS, it is much easier to test changes in location and background location updates, as well as simulate memory warning for hot keys, including slow animations. For Android, you can configure the exotic characteristics of the "iron": the screen resolution, pixel density, the amount of RAM, heap size, the size of the internal and external memory.
10. Fill the device RAM before running the application. This will help, firstly, to conduct stress testing and check the speed of work, and secondly, to check the preservation and resumption of the application state (where we will return after the application is minimized, whether all the necessary services will start).
11. Run the application under the debugger. Why?
- There is a chance to know Zen :)
- Allows you to work slowly with the application, which sometimes reveals bugs.
- If a crash or exception happens in the application, it will stop, and then you can call the developer to debug "without departing from the cash register."
Work with network
1. The application should work stably with:
- unstable connection;
- missing connection;
- exceptionally slow speed (1-2Kb / s);
- no response from the server;
- incorrect server response (garbage or errors);
- Changing the type of connection Wi-Fi - 3G - 4G - Wi-Fi on the fly.
Use chain cases "network problems", to the maximum using:
- custom firmware for the router. At one time, I was fine with the Tomato firmware for the Linksys WRT54G. The router cost a penny, and with the help of the firmware you can set the necessary Wi-Fi speed on the fly without losing the connection with the device;
- proxy;
- WANEm ;
- Network Link Conditioner - easily installed on Mac and embedded in iOS from version 6.0. They can “sew” traffic and distribute it by creating an access point on both an iOS device and a Mac ;
- for Android, you can use connection speed presets in the emulator or more flexibly configure via netspeed .
2. If you need a proxy server, the easiest way is to use CharlesProxy (there are instructions for devices and emulators on iOS and Android, supports binary protocol, rewrite, throttling traffic, and generally worth every nested cent ) or Fiddler (it's free).
Work with application data, external and internal services
1. If there is a third-party service - he will fail. The recent accident at FB has affected the work of some applications and sites. For example, a couple of versions ago, the Habrahabr application “shared” articles with UI blocking without an activity indicator. At the moment when Facebook or the Internet “braked”, the “sharing” hung the entire application until the process was completed.
It is better to identify the maximum number of abnormal situations in advance and consider steps to eliminate them: provide for handling unexpected responses (errors, garbage, no response, empty response) from third-party services with indication of a problem to the user, add timeouts to the necessary requests, etc.
2. If there are third-party libraries, they will definitely cause problems. In particular, Twitter, PayPal, Facebook quite often contain bugs. As an example, in one of the versions of the Twitter SDK, there was a crash when receiving 503 errors from its own backend — the library simply crashed and dragged the application along. The Facebook SDK also often crashes on Android (you can see a process called com.facebook.katana from time to time in crash alerts).
3. Parsers URI and data should try to take into account all sorts of surprises - there were cases when the auto check of the validity of files and (or) URI broke off, and applications had to pull out the information bit by bit.
For example:
- error 404 returned in html format;
- 200 has HTML or JS in the answer;
- returned an empty answer;
- the server responds to the API request with a standard web server stub (“It works!” in the case of nginx);
- returned empty data structure (JSON, XML, PLIST);
- instead of one data structure, another returned (HTML instead of XML);
- Returned leading not there or invalid URI.
In all these cases, the application may not parse the unexpected response and fall.
In addition to working on the stability of the application, in the above cases it is important to give the user some visual response: alert, toast notification, placeholders instead of data, etc.
4. If your application updates the data on static or easily generated URLs, then you can use Dropbox or Google Drive until the logic to the server is ready or run. Uploading or updating files directly on the device is a dubious pleasure.
Therefore, we did this:
- made all URLs configurable and rendered to a separate entity so that the development or testing team can easily rebuild applications with specific URLs for updated data;
- then they changed all the necessary files and replaced them with existing ones (manually or with the help of the simplest scripts). To roll back to the previous version, you can write another script that will write reference files (you can also use the versioning of files that Dropbox provides).
5. Do not forget to check the migration of data and caches when updating the application. It is important to remember that users can skip versions and it will be necessary to check for updates of older versions. As an example, I can give a crash at the start of the LinkedIn application in June 2015: some users could not launch the application until they released a new version (fortunately, it was released on the same day).
Android
1. Set the custom screen resolution of the emulator - this will reveal problems with the layout, if you have a certain lack of devices or need to check whether the layout is written correctly. Also, the screen resolution and pixel density can be edited via ADB and on a physical device, for example, on Nexus 10 .
2. If the keyboard is redefined (using custom), then pay extra attention to this. There are both keyboard errors that cannot be bypassed, and logical or graphical errors.
3. Staged rollout will make it easy to find problems that might have been missed when testing the release version: you can roll out 5-10% and monitor graphics and crashes, if necessary, roll back or re-allocate the version with fix.
4. Use do not keep activities when testing and make sure applications are ready for unexpected activity terminations, which can lead to crashes or data loss.
iOS
1. Check if the standard gestures are overridden. For example, when you activate "Universal Access", additional gestures are activated, they may conflict with the gestures of your application (for example, a three- and four-finger gesture).
2. Also pay attention to third-party keyboards. For example, in iOS9 there is a bug that will cause the application to crash if you enter text using a third-party keyboard in a modal window in WebView.
3. Show the developers the rollout.io service, which allows you to patch some crashes on production, override parameters, show apologies with apologies, or make some buttons inactive. He has already saved us more than once.
4. For interactive testing of the layout or checking that all screens have been removed from the hierarchy, you can use the standard Xcode or Spark Inspector , RevealApp tools .
5. Ask to integrate the Memory Warning call into the debugger menu. It is usually hanged on a certain gesture (tap with several fingers, pressing the status bar or navigation) or the volume buttons . This is necessary to check the adequacy of the behavior of the application during Memory Warning, if it cleans up the resources and how well it is done. For example, we had an unpleasant bug, when, after Memory Warning, our Image service unloaded the image from memory and the screen remained blank.
Adjusting processes
These tips will help you move faster in testing mobile applications and will teach you to bypass the pitfalls of the relationship with developers.
1. Enter a Pre-QA culture. Before sending the ticket to the review, sit down with the developer for his car and test the feature for 5-10 minutes under the debugger on one or two devices - most of the most stupid mistakes will be found right away. It also allows developers to teach basic testing skills: at a minimum, they will repeat actions for you, at most they will penetrate and test more meaningfully. No one wants to make stupid mistakes and put them on public display.
2. At least fluently look through the diffs of each branch (features) and ask as many questions as possible to the developers.
Thus, you, first of all, raise your prestige as a tester - you try to understand the code and areas that are affected by this feature. Even now, mobile application testers are sometimes perceived by developers as monkeys who poke into the phone and juggle them in order to “drop” the application.
If there are no free developers, you can act as a reviewer: the developer, while explaining the work, features sometimes find flaws or cases that he did not take into account.
Secondly, you will gradually begin to learn a programming language and will understand better what is happening “under the hood” of the application.
3. Examine the life cycle of the application's entities and the application itself (Activity for Android 1 , 2 , 3 ; ViewController for iOS 1 , 2 , 3 ) to understand which state the application screen can go from and to itself. The better you know how the application works from the inside, the more fully you will be able to test it.
4. If you have apps for iOS and Android, then it is important to maintain the right balance of resources for testing them.
Errors in applications threaten to re-lay, which has certain consequences. At the same time, the price of an error in the Android platform is often lower.
- Android has a staged rollout . The Android application can be redrawn at least on the same day or rolled back the staged rollout (up to 50% of the layout can be completely rolled back to the previous version). But you should not re-allocate very often, because users will begin to complain and give low ratings;
- For iOS, rebinding is done, at best, through expedited review (which is not recommended to be abused). The application will be redirected:
- the earliest - on the same day (usually in in review goes on the same day, but is available only for the next one);
- in the worst case, if they did not allow expedited review, the period will increase to 5-10 days.
But, on the other hand, iOS applications can be tested much faster, because The ecosystem is not as fragmented as Android.
miscellanea
1. What should I do if the hour X sneaks up at the most inopportune moment and an unstable version of the application hits production? We use an update screen system to speed up the user migration process. Such a system is useful:
- in case of a critical bug that was missed during development and testing;
- with a quick update of the version of interest to us:
- running features on all platforms simultaneously (also useful if changes break backward compatibility);
- faster and more uniform A / B testing results;
- offloading the server command, which is forced to support outdated versions of the API due to a certain number of users using the (very) old version of the application.
Our update system works in two modes :
- soft (soft-update), when there are “Update” and “Skip” buttons on the screen (the screen can be hidden for 24 hours; also in this mode, you can ask users to enable the auto-update of applications for iOS and Android in the system settings - some of the users turn off auto updates);
- hard (hard-update) - only the “Refresh” button is displayed on the screen, which leads immediately to the page of our application in the store.
Not all users will have the physical ability to update the application, so for certain versions the logic is intentionally disabled. Take, for example, iPhone4 users who will not be able to upgrade to iOS8, while we in the application plan to stop supporting iOS7.
2. Monitoring of the most important application metrics in production is required:
- charts daily / monthly / ... active users to respond to accidents faster;
- crash log collection and analysis systems: Crashlytics (now part of Twitter Fabric), HockeyApp , Crittercism , BugSense (now part of Splunk);
- feedback systems from users through the application (embedded feedback forms or sending a letter) with the ability to send a description of the device and screenshots;
- application usage statistics ( GoogleAnalytics , Flurry , Splunk , Heatmaps.io , MixPanel );
- Digests of downloads, grouping reviews, understanding where and when you were recorded ( AppAnnie - bought by Distimo).
3. Sometimes problems arise only for certain users with certain devices or in certain countries. For example, Vodafone UK had problems with WebP images. To test such cases, you can use cloud solutions for renting devices: DeviceAnywhere (paid service), PerfectoMobile (paid service), Samsung Device Lab (free, but there is a point system that replenishes with time).
4. Also, do not forget about time zones and user locations. Perhaps your application is not designed to work in certain countries (although it was posted there by mistake, or you, as a user, came to a time in another country). Location on iOS can be "fake" in the settings of the simulator (Debug> Locations), and on Android there are applications that allow you to do this .
If the application works with data and there are several data centers in different time zones, you need to make sure that everything works correctly and there are no collisions when switching between data centers.
5. Learn how to update and “downgrade” firmware - platforms are fragmented, especially Android and Blackberry. Cloud services are good, but they cost money, so not all companies have the opportunity to use them due to lack of funding or security policy.
6. Found a bug after calculating the features, but do you have to re-upload the new version? In this case, you can help enable, disable or modify features on the fly. In our applications a lot of features can be turned off from the server side directly or, depending on the team’s solution, using a special interface.
Conclusion
Such a list of recommendations, approaches and tools, in our opinion, can be useful for both beginners and experienced mobile application testers. I hope the developers and managers will also learn something useful for their purposes.
Compiling this list, my colleagues and I were based on our own experience and would be happy to see your opinion in the comments.
Alexander Host
Lead Mobile QA Engineer
Source: https://habr.com/ru/post/269189/
All Articles