8 interesting beta bugs iOS 12 and how we searched for them

Hello!

I'm Kate, testing mobile apps in Badoo. Today I will talk about why beta testing is so important, how we conduct manual testing of such versions, and what interesting things we discovered during the test of iOS 12 Beta.


')
Every September, Apple releases a new version of iOS, and at every WWDC, company representatives boast that iOS users install updates much faster than Android users.

The reason for this is not only that iOS is cool in itself, but also because Apple knows how to get users to update:

• changing the position of the "Remind me later" button, so that some inattentive users accidentally click on the update button;
  
• not allowing downgrade version ;
  
• showing a red marker on the settings icon;
  
• Finally, my favorite move is the daily display of update messages.
  

By the way, the beta version had an annoying bug. The update notification pops up whenever the screen is unlocked or the notification center is closed; there were no available updates.

That is why our users are updated at the speed of sound - and we should be ready for this.


As can be seen from the graph, in August last year, the beta version of iOS 11 was installed only in 1.05% of our users, and by the end of October, this figure reached almost 60%.

An example from personal practice

In September 2017, I updated the version of my device to iOS 11, and the email application, which I actively use, stopped sending push notifications. Yes, you didn’t dream: BREAKED. SEND. PUSHES. Thus, to find out whether I received a letter or not, I had to open the application and check the Inbox folder. It was terribly annoying, so I wrote about the problem in technical support, and it was solved. It took three weeks! After the release of the fix, the notifications returned, but the counter on the icon disappeared. With a growing wave of discontent, I reported on this bug. In short, I began to use the new application (I also use the old one, but not on a permanent basis).

Mobile software is constantly changing, and we can not keep up. That is why, when a new beta version of iOS comes out, we have to test it. Almost immediately.

Starting with which beta version is worth testing?

This year we started with Beta 6. “Why not with the first?” - You ask. The fact is that the beta versions at the very start are very “raw”, and actively testing an unstable version is rather pointless. You may encounter many different errors that Apple will soon correct. We saw it with our own eyes: between versions 5 and 9, many problems were solved.

Example. On the screen with video previews, we had crashes. Such a problem existed in Beta 5 and disappeared through Apple’s efforts in Beta 6.

What is the beta testing process

Step 0. Install the OS beta on your device.

Check out the list of devices that support iOS 12 . The oldest and least productive iPhone is 5s, the newest and most productive is XS Max. So, if you have not done so yet, update your collection of gadgets. In addition, I advise you to regularly check the rating of the most frequently used devices for your application.
After the device is selected, you can install the beta. Here is the available instruction.

Step 1. Start with new OS features.

Under the link you will find their list. Read it carefully and note the features applicable to your application. In our case, they were about notifications, uptime and performance improvements.



Step 2 . Manual regression.

In principle, the manual regression of the entire application is not my favorite activity, but in the beta versions even it is a joy. Constant hangs and crashes are like a breath of fresh air, like immersion into a startup atmosphere.

Tips

• Check out the latest release version.
  
• Keep track of all the bugs you encounter and hope that they will be corrected by Apple’s omnipotent technical support. Of course, when the new beta comes out, you will need to double-check them all.
  
• It will be useful to review the list of problems that occurred during the previous transition (from iOS 10 to iOS 11) and make sure that everything works as it should. Under this item, I advise you to pay special attention to making payments (through Sandbox and other providers, as well as single purchases and subscriptions), access permissions (to the camera, location, etc.), camera operation (photo and video, downloads from the gallery, preview), location (online and offline), correspondence (all kinds of messages and offline synchronization) and network operation (switching between Wi-Fi and mobile Internet).
  
• Before releasing a new version of iOS, Apple usually releases a new version of Xcode. It is logical to assume that support for iOS 12 with the latest version of your application means that it was compiled using the latest Xcode. However, working in applications compiled in different versions of Xcode, you may encounter a variety of bugs. Therefore, check everything carefully.
  

Step 3 . Non-functional testing.

Usually for non-functional testing, we use Xcode tools. However, if you have iOS 12 beta installed, you will need to use the Instruments beta to track energy consumption (Energy Log) and monitor system status (Activity Monitor). By downloading Xcode from here , you can use Instruments normally.

CPU:


iOS 12 Beta


iOS 11

Memory:


iOS 12 Beta


iOS 11

Battery:


iOS 12 Beta


iOS 11

As our research has shown, there have been no significant changes in the operation of the processor, memory and battery with the transition to iOS 12 Beta in the Badoo application.

Step 4. Make sure your autotests pass on a new beta.
In particular, install the beta version of Xcode on the agents and update Facebook WebDriverAgent. Read more about this item in the next article.

Step 5. Turn on iOS 12 Beta testing in your testing process.
I advise you to include a check on iOS 12 Beta as soon as the beta version becomes stable. We included such checks in testing feature-brunch, their integration, as well as in release testing.

What bugs we found in badoo and other applications

Here are just the most worthy representatives:

1. Crash when you start the application - a classic of the genre. So, for example, the messenger that I use to communicate with my parents was painted. This problem was fixed by the application developers themselves, so after the update, the bug could not be reproduced.
   
2. Other crashes. While working on the article, I encountered the following problem: the review collection application crashed while trying to respond to a comment when a low battery warning popped up. In this case, updating the application did not help.
   
3. The Badoo application crashed while trying to open your own profile in an application compiled with the SDK 12.0. This was caused by an error when using enum Objective-C initialized with an incorrect rawValue value. Departure occurred when, during the execution of the program, there was no corresponding value in the switch statement . This bug (see gifku below) was fixed by our developers (we still need, by the way ).
   
   
   
   
   
4. Hangs when connecting to third-party applications (for example, when uploading photos or when verifying accounts in social networks). We also solved this problem on our own.
   
5. The message "Gallery is empty" when trying to upload a photo. This bug has touched an application that I use to send photos to friends and other people from my contact list. I expect that its developers will fix this problem, but for now I use the standard Share dialog, which seems to me very inconvenient.
   
6. Wrong order of notifications. This problem is observed in the application in which I read blogs. Still waiting for a fix.
   
7. Duplicate the dialogue about the access request in the simulator. This bug was in our application, and our automation team was very unhappy. Having corrected it, we can again rely on the test results.
   
8. Bugs with animation and keyboard. The navigator application I use started to flash like crazy, left without access to the Internet. In addition, one of the banking applications broke the layout of the page when opening a message from technical support via a push notification. Waiting for developers to fix both problems.
   

Note that we tested Badoo on applications compiled with:

• the old version of Xcode 9 on the latest iOS Beta (6-12) and GM (iOS 12);
  
• the latest Xcode 10 beta on the latest iOS Beta (6-12) and GM (iOS 12);
  
• a fresh beta version of Xcode 10 on older versions of iOS (11).
  

Other applications were tested out of curiosity with the application compiled using the old version of Xcode 9 on the latest iOS 12 Beta (6-12).

Amazing conclusions

Testing beta versions is incredibly useful - it helps to protect users from very unpleasant bugs. Some bugs were found in iOS itself and Apple has already been fixed, but most of them were fixed by our gallant developers. And now, after the retest of fixes, we can be sure that iOS 12 users will be able to enjoy the Badoo application to the full.

In addition, it is very nice to see how bugs fix in other applications. Many of the applications that I use still have iOS 12-specific bugs, although the official release took place in September. It would be great for developers to hurry with fixes, since after the active update phase this month, most users will be under attack.

In general, guys, test the beta - this is useful both from the standpoint of your personal growth, and for your users.