Windows Phone as an experimental platform
Most IT companies do not make versions of their products under Windows Phone, because the share of Microsoft’s mobile operating system is lower than that of iOS and Android. Nevertheless, it is the third largest mobile operating system on the market, and the application has been existing in Badoo Windows Phone for a long time.
It was originally written by an outsource developer on Silverlight and has not been updated for a long time, and for several years the product has not had a dedicated department. In 2014, a Windows Phone developer came to the company and began to support this application, and I began to receive some of its tasks for testing.
After a couple of months, it was decided to write a completely new application, abandoning almost all the old developments, largely because the code was far from ideal, and a lot of time was spent on its support. The first new application for Windows Phone was a pilot project Badoo - Hot or Not. By concept, it is very similar to Badoo, but it has much less functionality and is somewhat different. In just 3 months, the two of us made the Hot or Not application from scratch in one branch of the Git repository, which was eventually locked into Master. And the TeamCity application was set up so that each developer commit was going as a separate application, which allowed me, as a tester, to see the history of client changes.
After several releases of Hot or Not, we returned to the development of Badoo, deciding to make both applications with a common core. This meant that Hot or Not and Badoo would have a common Git repository and a common Visual Studio project. At that time, besides Windows Phone 8, we still supported Windows Phone 7. And because of the specifics of Visual Studio, it turned out that these are two completely different applications, and I had 2 projects, each of which consisted of two applications. For one tester, this is quite a lot and, most likely, we would have made a lot of mistakes when using the old development scheme, when we had one branch in Git for one release. And we began to think how to improve the development process.
')
In the course of reflection, we put forward two main requirements for the working process:
We noticed that our mobile development teams for iOS and Android use Git Flow. This is a fairly large system in which testing is divided into three stages.
Initially there are two Git branches: Master and Dev. The first stage of testing is a feature of the branch. The developer creates a new Git-branch from the Dev branch, makes it cool functionality and gives it for testing. When the task is verified and everything is fine with it, it is merged back into the Dev branch.
When in this way a certain number of tasks, for example, 10, is merged in Dev, then it is time for the second stage of testing - integration. Here, tasks are tested for the presence of bugs that appear due to the integration of several branches among themselves. The problems found at this stage are corrected in the first priority right in the integration branch.
The last stage of testing passes before you want to publish your new version of the application to the store. This is release testing. It is similar to integration testing. The bugs found in the release also have the highest priority. When all the tasks of this release have been checked, release testing has been carried out and the application has been published, the release branch is merged into Dev and Master.
Git Flow is a very interesting and good system that is suitable for large development teams of iOS and Android (we have more than 15 people in each). But it would have been too difficult for one tester and one developer to work with it, and we decided to remove the first stage of testing, and the resulting system was called “Windows Phone Team Flow”.
Thanks to this, we successfully released the new Badoo application just 3 months after the start of development.
In 2015, our team increased by 2 times - we hired another developer and tester, which is quite strange, because the classic ratio of testers to developers is 1/3. But we had reasons for this:
Due to the high speed of delivery of new versions to users, the best experiment platform in our company is the web. We have real continuous integration: 2 times a day we post a new build for production, and each layout takes less than a minute. But today the mobile industry is developing in leaps and bounds, and an increasing number of users prefer mobile applications. Therefore, the company's priority has shifted to mobile platforms. And there was a big problem. The results of experiments running on the web, in most cases, are unsuitable in the realities of mobile clients, so the experiments need a special mobile platform.
How is Windows Phone good for experimenting?
Let the number of users of our mobile resource on Windows Phone be only 3% of the total number, there are still hundreds of thousands of them on a day. This number allows you to run A / B tests or even A / B / C / D tests, where they are divided into 4 groups, including the control. On the other hand, there are not so many users, so that, in the event of an error, they could affect the activity of a large number of users. A decrease in activity on a large platform would entail a noticeable decrease in the activity of the entire resource.
Unlike iOS, where the review time in the AppStore is a week or even two, in our experience, a review in the Windows Phone Store takes only 20 minutes. According to Microsoft, in 16 hours the application will be available for download worldwide. And in our experience, in just 2 hours it will be possible to download it in all significant markets for us. This means that Windows Phone has a fast delivery rate for new versions of applications. 60% of users will update within 24 hours.
Android has a very large fragmentation of hardware installed on mobile phones. Plus, each vendor creates its own version of this operating system, while changing its fundamentals. Microsoft also set strict limits for the hardware used in the phones, and there are no third-party versions of the Windows Phone OS. Therefore, we have almost no bugs that are dependent on the operating system or hardware.
Windows Phone OS was developed after the launch of iOS and Android. And there is one big difference - this is a graphical interface. Metro-design is greatly simplified, unlike iOS and Android OS. What was rounded becomes square. It almost lacks complex animation. On the other hand, it's all the same mobile operating system with multitasking and touch screen. And applications for all these operating systems may look exactly the same.
Thanks to all the above-described advantages of Windows Phone, we have become a mobile experimental platform. Our team quickly implements the ideas of product managers and, having collected statistics, we determine whether this functionality is successful. If everything is good, this idea is given for development to the iOS and Android teams, which implement this functionality in the usual flow. If the statistics says that the idea has some problems, our product manager thinks how to modify it. Comes up, we are finalizing and run again. If nothing helps, then we simply remove this functionality from the client. In 2015, thanks to experiments, we implemented 55 new features, which allowed us to significantly increase the revenue from mobile platforms.
In each of the teams of translators, designers, mobile server development and billing development there are people who deal with the tasks of our experiments in the first place. This allows us to create new features faster.
We are still a small team that has no problems with communication. We have almost no “meetings”, because we can discuss the task right away without leaving the workplace. Specifically, we only meet once a week to discuss the weekly release cycle with the product manager.
There are no test cases in our company at all. Because a huge amount of time is spent on their development and maintenance. In addition, the work of a person according to the instructions reduces the desire to conduct any research and experiments, and after all testing is a creative process. Especially in the experimental team, where every week we create several new features. Instead of test cases, we use notes, separated by features, in which you can find anything, some of them have mind maps attached, which allow you to see the relationship between the application components and their input parameters.
Badoo has created a very large number of different utilities that help test and develop faster, then we look at the main ones.
Interface to view server logs
I am sure that almost every tester faced a situation when it was necessary to determine on which side the bug - client or server side. And to simplify this task, we have created a very convenient interface for viewing server logs, which allows you to see them filtered by user or device ID, as well as real-time logs or history. Additionally, you can add separators, hide or show different parts of it.
QAPI
Sometimes it is difficult to prepare a test account or test environment to reproduce the problem. For example, you need to check how a letter looks and works, which comes to the user's mail a week after registration. The real test case looks like this: register a new account, wait a week, receive a letter and finally check it. But this is too long, and the first thought of the tester is to go to the server developer and ask him to change something in the database so that he can get the letter quickly. But distracting developers on such trifles is inefficient, Besides, there can be a lot of such “trifles”. And in order for everyone to be happy, we created a QaAPI. This is a collection of API methods that allow you to quickly prepare a test environment or account. In the case of a letter that is sent a week after registration, it would be enough to run one QaAPI method to get a letter within a second. You can learn more about QaAPI from the report of Dmitry Marushchenko .
Server mock
In our company, the server is usually developed faster than mobile clients. But, due to the rapid development speed in Windows Phone, our team has a problem - you cannot test and develop new functionality, because the server is not ready. In order to get rid of this problem, an interface called Server Mock was created. It allows you to create a special test environment in which the server returns predefined answers to the specified client requests. This was made possible thanks to a protocol that is developed with the mandatory participation of all server and client commands.
About the development of the web, mobile clients in general and the protocol in particular, you can find out in the report of Pavel Dovbush .
Badoo Internal App Store
In addition to apps published to app stores, each of our mobile development teams has one more app, which we call “internal app stores.” They allow you to install client test builds without installing xCode for iOS, Visual Studio for Windows Phone OS and Android SDK for Android OS. In situations where your product manager wants to study the work of the new functionality, which will soon be released, or the designer wants to see what this new functionality looks like, or the translator wants to find out what the newly translated text looks like on a real phone, simply select and install the desired application assembly.
Debug_menu
Each test version of our applications has a large Debug_menu section. For example, in Windows Phone Debug_Menu, you can view the client log, send it by email, clear the cache of photos or the shared cache, if you want, you can even paint the application. In general, opportunities that facilitate testing and development, the mass.
As I wrote earlier, on our Windows Phone many experiments are being conducted in our company, but in order to understand the results of these experiments, we collect a lot of statistics.
Hotpanel
We have our own utility for statistics, which is developed by a separate team called BI-team (business intelligence team). Having such a team allows you to create fully configurable statistics that will satisfy the needs of our product managers.
Appannie
For a quick response to problems reported by users, leaving reviews in mobile app stores, we use AppAnnie. This is a service that collects and processes ratings and reviews in the AppStore, Google Play and the Windows Phone Store.
HockeyApp
One of the main metrics of the success of the new version of the application is “crashrate”. And for collecting crash logs from real users, we use HockeyApp.
Key graphics
We also have a large number of graphs showing the key metrics of our resource, for example, the activity of user voting, authorization, the number of active users and so on.
A couple of months ago we implemented the new Double Tap To Like task. On one of our main screens of the People Nearby application, where there is a large number of user profiles, it’s enough to double tap the user’s icon so that Badoo automatically counts a positive vote for it. By launching this feature, we expected that many users would actively vote. By collecting statistics, we learned that the majority of users actually increased the voting activity, the total activity of the female audience even increased, while the male part of the resource decreased. Therefore, we decided to leave Double Tap to Like for girls only and gave it to the iOS and Android teams for implementation. This example once again proves that statistics is very important for making decisions about the success of experiments.
These are, perhaps, all the methods and utilities that we wanted to tell you about. Together and individually, they can significantly speed up the development and testing of mobile applications. Verified on badoo. Now you!
Vyacheslav Loktik
Senior Mobile QA Engineer (Windows Phone)
It was originally written by an outsource developer on Silverlight and has not been updated for a long time, and for several years the product has not had a dedicated department. In 2014, a Windows Phone developer came to the company and began to support this application, and I began to receive some of its tasks for testing.
After a couple of months, it was decided to write a completely new application, abandoning almost all the old developments, largely because the code was far from ideal, and a lot of time was spent on its support. The first new application for Windows Phone was a pilot project Badoo - Hot or Not. By concept, it is very similar to Badoo, but it has much less functionality and is somewhat different. In just 3 months, the two of us made the Hot or Not application from scratch in one branch of the Git repository, which was eventually locked into Master. And the TeamCity application was set up so that each developer commit was going as a separate application, which allowed me, as a tester, to see the history of client changes.
After several releases of Hot or Not, we returned to the development of Badoo, deciding to make both applications with a common core. This meant that Hot or Not and Badoo would have a common Git repository and a common Visual Studio project. At that time, besides Windows Phone 8, we still supported Windows Phone 7. And because of the specifics of Visual Studio, it turned out that these are two completely different applications, and I had 2 projects, each of which consisted of two applications. For one tester, this is quite a lot and, most likely, we would have made a lot of mistakes when using the old development scheme, when we had one branch in Git for one release. And we began to think how to improve the development process.
')
In the course of reflection, we put forward two main requirements for the working process:
- First, testers and developers should not block each other. There were only two of us, and if one blocked the work of the other, then half the team would not work;
- secondly, each task should be checked several times at different stages (preferably by different people).
We noticed that our mobile development teams for iOS and Android use Git Flow. This is a fairly large system in which testing is divided into three stages.
Initially there are two Git branches: Master and Dev. The first stage of testing is a feature of the branch. The developer creates a new Git-branch from the Dev branch, makes it cool functionality and gives it for testing. When the task is verified and everything is fine with it, it is merged back into the Dev branch.
When in this way a certain number of tasks, for example, 10, is merged in Dev, then it is time for the second stage of testing - integration. Here, tasks are tested for the presence of bugs that appear due to the integration of several branches among themselves. The problems found at this stage are corrected in the first priority right in the integration branch.
The last stage of testing passes before you want to publish your new version of the application to the store. This is release testing. It is similar to integration testing. The bugs found in the release also have the highest priority. When all the tasks of this release have been checked, release testing has been carried out and the application has been published, the release branch is merged into Dev and Master.
Git Flow is a very interesting and good system that is suitable for large development teams of iOS and Android (we have more than 15 people in each). But it would have been too difficult for one tester and one developer to work with it, and we decided to remove the first stage of testing, and the resulting system was called “Windows Phone Team Flow”.
Thanks to this, we successfully released the new Badoo application just 3 months after the start of development.
2015
In 2015, our team increased by 2 times - we hired another developer and tester, which is quite strange, because the classic ratio of testers to developers is 1/3. But we had reasons for this:
- We have almost no automation. Now it is at the initial stage of development and in fact, we do all the checks manually.
- Developers have excellent standard libraries from Microsoft, which, together with the libraries of Telerik, allow developing new UI interfaces very quickly. The design of Windows Phone is very simple, and we try to avoid complex animation. All this allows our developers to do more in less time.
- Due to the large number of experiments, we create more new functions in the application than our large development teams, although we are smaller.
Experiments
Due to the high speed of delivery of new versions to users, the best experiment platform in our company is the web. We have real continuous integration: 2 times a day we post a new build for production, and each layout takes less than a minute. But today the mobile industry is developing in leaps and bounds, and an increasing number of users prefer mobile applications. Therefore, the company's priority has shifted to mobile platforms. And there was a big problem. The results of experiments running on the web, in most cases, are unsuitable in the realities of mobile clients, so the experiments need a special mobile platform.
How is Windows Phone good for experimenting?
Let the number of users of our mobile resource on Windows Phone be only 3% of the total number, there are still hundreds of thousands of them on a day. This number allows you to run A / B tests or even A / B / C / D tests, where they are divided into 4 groups, including the control. On the other hand, there are not so many users, so that, in the event of an error, they could affect the activity of a large number of users. A decrease in activity on a large platform would entail a noticeable decrease in the activity of the entire resource.
Unlike iOS, where the review time in the AppStore is a week or even two, in our experience, a review in the Windows Phone Store takes only 20 minutes. According to Microsoft, in 16 hours the application will be available for download worldwide. And in our experience, in just 2 hours it will be possible to download it in all significant markets for us. This means that Windows Phone has a fast delivery rate for new versions of applications. 60% of users will update within 24 hours.
Android has a very large fragmentation of hardware installed on mobile phones. Plus, each vendor creates its own version of this operating system, while changing its fundamentals. Microsoft also set strict limits for the hardware used in the phones, and there are no third-party versions of the Windows Phone OS. Therefore, we have almost no bugs that are dependent on the operating system or hardware.
Windows Phone OS was developed after the launch of iOS and Android. And there is one big difference - this is a graphical interface. Metro-design is greatly simplified, unlike iOS and Android OS. What was rounded becomes square. It almost lacks complex animation. On the other hand, it's all the same mobile operating system with multitasking and touch screen. And applications for all these operating systems may look exactly the same.
Thanks to all the above-described advantages of Windows Phone, we have become a mobile experimental platform. Our team quickly implements the ideas of product managers and, having collected statistics, we determine whether this functionality is successful. If everything is good, this idea is given for development to the iOS and Android teams, which implement this functionality in the usual flow. If the statistics says that the idea has some problems, our product manager thinks how to modify it. Comes up, we are finalizing and run again. If nothing helps, then we simply remove this functionality from the client. In 2015, thanks to experiments, we implemented 55 new features, which allowed us to significantly increase the revenue from mobile platforms.
What helped us achieve such results?
In each of the teams of translators, designers, mobile server development and billing development there are people who deal with the tasks of our experiments in the first place. This allows us to create new features faster.
We are still a small team that has no problems with communication. We have almost no “meetings”, because we can discuss the task right away without leaving the workplace. Specifically, we only meet once a week to discuss the weekly release cycle with the product manager.
There are no test cases in our company at all. Because a huge amount of time is spent on their development and maintenance. In addition, the work of a person according to the instructions reduces the desire to conduct any research and experiments, and after all testing is a creative process. Especially in the experimental team, where every week we create several new features. Instead of test cases, we use notes, separated by features, in which you can find anything, some of them have mind maps attached, which allow you to see the relationship between the application components and their input parameters.
Utilities
Badoo has created a very large number of different utilities that help test and develop faster, then we look at the main ones.
Interface to view server logs
I am sure that almost every tester faced a situation when it was necessary to determine on which side the bug - client or server side. And to simplify this task, we have created a very convenient interface for viewing server logs, which allows you to see them filtered by user or device ID, as well as real-time logs or history. Additionally, you can add separators, hide or show different parts of it.
QAPI
Sometimes it is difficult to prepare a test account or test environment to reproduce the problem. For example, you need to check how a letter looks and works, which comes to the user's mail a week after registration. The real test case looks like this: register a new account, wait a week, receive a letter and finally check it. But this is too long, and the first thought of the tester is to go to the server developer and ask him to change something in the database so that he can get the letter quickly. But distracting developers on such trifles is inefficient, Besides, there can be a lot of such “trifles”. And in order for everyone to be happy, we created a QaAPI. This is a collection of API methods that allow you to quickly prepare a test environment or account. In the case of a letter that is sent a week after registration, it would be enough to run one QaAPI method to get a letter within a second. You can learn more about QaAPI from the report of Dmitry Marushchenko .
Server mock
In our company, the server is usually developed faster than mobile clients. But, due to the rapid development speed in Windows Phone, our team has a problem - you cannot test and develop new functionality, because the server is not ready. In order to get rid of this problem, an interface called Server Mock was created. It allows you to create a special test environment in which the server returns predefined answers to the specified client requests. This was made possible thanks to a protocol that is developed with the mandatory participation of all server and client commands.
About the development of the web, mobile clients in general and the protocol in particular, you can find out in the report of Pavel Dovbush .
Badoo Internal App Store
In addition to apps published to app stores, each of our mobile development teams has one more app, which we call “internal app stores.” They allow you to install client test builds without installing xCode for iOS, Visual Studio for Windows Phone OS and Android SDK for Android OS. In situations where your product manager wants to study the work of the new functionality, which will soon be released, or the designer wants to see what this new functionality looks like, or the translator wants to find out what the newly translated text looks like on a real phone, simply select and install the desired application assembly.
Debug_menu
Each test version of our applications has a large Debug_menu section. For example, in Windows Phone Debug_Menu, you can view the client log, send it by email, clear the cache of photos or the shared cache, if you want, you can even paint the application. In general, opportunities that facilitate testing and development, the mass.
Statistics
As I wrote earlier, on our Windows Phone many experiments are being conducted in our company, but in order to understand the results of these experiments, we collect a lot of statistics.
Hotpanel
We have our own utility for statistics, which is developed by a separate team called BI-team (business intelligence team). Having such a team allows you to create fully configurable statistics that will satisfy the needs of our product managers.
Appannie
For a quick response to problems reported by users, leaving reviews in mobile app stores, we use AppAnnie. This is a service that collects and processes ratings and reviews in the AppStore, Google Play and the Windows Phone Store.
HockeyApp
One of the main metrics of the success of the new version of the application is “crashrate”. And for collecting crash logs from real users, we use HockeyApp.
Key graphics
We also have a large number of graphs showing the key metrics of our resource, for example, the activity of user voting, authorization, the number of active users and so on.
A couple of months ago we implemented the new Double Tap To Like task. On one of our main screens of the People Nearby application, where there is a large number of user profiles, it’s enough to double tap the user’s icon so that Badoo automatically counts a positive vote for it. By launching this feature, we expected that many users would actively vote. By collecting statistics, we learned that the majority of users actually increased the voting activity, the total activity of the female audience even increased, while the male part of the resource decreased. Therefore, we decided to leave Double Tap to Like for girls only and gave it to the iOS and Android teams for implementation. This example once again proves that statistics is very important for making decisions about the success of experiments.
These are, perhaps, all the methods and utilities that we wanted to tell you about. Together and individually, they can significantly speed up the development and testing of mobile applications. Verified on badoo. Now you!
Vyacheslav Loktik
Senior Mobile QA Engineer (Windows Phone)
Source: https://habr.com/ru/post/276827/
All Articles