Introduction

The third part of the review will be devoted to the Ubertesters service (the first part of the review , the second part of the review ).

This is a rather “young” service, with ambitions to enter the list of leaders (this will not be easy, given the recent acquisitions of competitors by the largest players in IT). The big difference and plus Ubertesters is the ability to attract on a paid basis external testers (also anyone can register as a tester and get paid for their work).

Separately, I want to note that Ubertesters was the only company from the review that followed its publications, provided additional information and shared plans to update the service, in order to improve the quality of the review.
')
Evaluation system: services are evaluated on a 10-point scale for each of the sections (Registration and Integration, Main Functionality, Additional Functionality, Continuous Integration). The total score will determine the winner (the final conclusion will be included in the last part of the review).

Ubertesters

Registration and Integration

Registration in the service is simple, except for paid subscription options, there is the possibility of free use of the main functionality with restrictions on additional functions and the number of team members, projects, and so on ( more ).

Note from the category of curiosities: after registration, check your profile - in my case, the country was specified correctly, but the subject of the federation was chosen first from the list and it turned out to be the Adygea Republic!

SDK integration is simple ( http://ubertesters.com/download-sdk/ ):


Rating: 7/10.

Main functionality

Note (updated June 15, 2015): speaking of the distribution of iOS applications, you need to remember that at the moment all services except Apple's TestFlight still have to take into account the limit on the number of devices available using AdHoc provisional profiles. Consequently, the maximum of the devices used is 100 for each family of devices (this condition was changed on June 8, 2015, after announcing at WWDC that all developers programs are combined into one), which also includes devices used directly by developers. The biggest limitation is that the list of registered devices can be “zeroed out” only once a year, after the subscription to the iOS Developer Program is renewed (it is possible later that this condition will be changed).

Immediately after completing the registration procedure, the user has the opportunity to choose one of four options for further actions:

After selecting the first option - Start a new project - a brief description of the upcoming steps will be presented:

Note: the seeming step-by-step wizard window is in fact only informative and after completing the first step, the user is taken to a regular page (without specifying the current progress), where he can choose one of the two options offered (or use other functions of the site via the navigation menu) :

Another note about the second option is that the icon may lead to the idea of ​​using social networks as a tool to attract team members, but in fact sending invitations is done via e-mail.

When creating a new project in the first step, the easiest way is to use the form auto-fill feature when downloading the application distribution:

A little inconvenient, in my opinion, is that the downloaded distribution kit is not used by the service for its intended purpose, since the next step is to download the distribution kit for the first revision of the application separately (revision is the term used by the service for designating versions / builds).

The Overview page contains basic information about the project, as well as summary statistics:

Note: system notifications are set by default to Watching mode for users with the Admin or Manager role and Not Watching for users with the QA or Developer role, which assumes notification of only those events in which they are a participant. Ignoring mode is still available, when the system will not send any notifications at all:

For more details on the description of the above modes, please read the frequently asked questions page “What is the notification status? In what case will I be notified? ”.

When adding a new revision in the first step, you need to download the application distribution (* .ipa):

The next step is to set the basic revision information:

After downloading, the revision is inactive, that is, unavailable for installation on test devices. To start testing, click the Start button:

And on the next screen, indicate to which member of the team this revision will be available (and it is possible to send a notification to all participants):

After that, the audit receives the status of In progress and can be installed on the device.

Note: if the project has a group of users by default, then the members of the team included in it will be immediately selected (the switch position in the first On column), if there is no such group, then users will need to be manually selected. Access to each audit can be stopped either for all or for some users, which allows you to effectively solve organizational issues that arise during the work on the project. Also, when initializing Ubertesters, it is possible to allow access to the application, but disable the ability to use service functionality: UbertestersLockingModeDisableUbertesters.

The February update service ( more ) added including the ability to edit the description for the audit:

Invitations to the team, as I said, are sent by e-mail (Administration - Team - Add members), and this task can be facilitated by using the import function from the CSV file:

Users can define one of the three roles, as well as add to groups (used to distribute the application to a limited number of users).

Note. In general, within the service, the roles are hierarchically distributed as follows: at the organization level, there are two roles: Admin and Member ; at the project level, Member can be QA , Developer and Manager . Important nuance: the first revision in each project must be loaded by the user with the Admin or Manager role. Further revisions can be downloaded by any member of the team (including QA ). The service is designed in such a way as to take into account all possible options for building business processes in a particular company. A more detailed description of the roles can be found on the frequently asked questions page.

When registering a device, two applications are installed on it: one native (an icon with a white background), another web (an icon with a black background). All the main functionality is in the native application, so the web application can be deleted after the device has been registered, in order to avoid further confusion:




Note: in the native application there is no possibility to update the contents of the open tab, you have to switch between tabs to see new data (new application revision, new application for testing, etc.).

A new device for testing can be added in several ways:

Rating: 8/10.

Additional functionality

Once again, when testing software it is very important to obtain the most complete and reliable information. Unfortunately, at the moment Ubertesters has no functionality for symbolizing fall reports (in this case it loses in comparison with other services), but it has another, often more useful, functionality in its baggage: the ability to create tickets directly from the application under test, and also provide them with screenshots.

This functionality is available to the user after pressing a round button, which by default is placed in the upper right corner of the screen, but can be freely moved to any other convenient place:



The tracking system for errors in the service may not be a complete alternative to specialized services, but in combination with the ability to accompany each ticket with screenshots, it copes with its tasks perfectly well:



Each screenshot can be accompanied by marks (among the editing tools: pencil, marker, text labels, arrows, and the ability to cut out a part of the screenshot). The screenshots made remain on the device and can be reused. It is also possible to add screenshots from the standard photo library of the device.

Note: the letter with the notice on the new ticket does not contain information on whether screenshots were uploaded during the creation. In my opinion it would be convenient to have such data, at least the number of attached screenshots.

On the website of the service information on all tickets is presented very clearly, you can view the overall statistics for a single revision, and for all revisions in total:

In addition to statistics on the status, the information is also presented by type (Bug, Feature, Improvement, Task, Crash), priority and frequency of error playback.

An important advantage is that each ticket is linked to a specific revision, which facilitates work with testers reviews and increases the value of the information collected.

Note: If, in some case, crashing of crash reports plays a critical role, it is possible to use the SDK of another service, but after initializing the Ubertesters SDK, you must turn off crash processing:

[[Ubertesters shared] disableCrashHandler];

Important! In this case, no information about crashes will be available on the Ubertesters website.

The service provides the ability to integrate with external trackers (Jira, Mantis, Redmine, YouTrack, Unfuddle, HPQC). At the same time, integration can be performed with only one of the available services. The interface for managing this functionality is located in the Administration - Issue Tracker section (settings are made for each project individually).

An example of a JIRA-related ticket:

The work of testers can be more organized by using the Tests functionality. In a special section of the service, you can view statistics on existing tests, as well as create new ones:


Note: The ID field is intended for a unique identifier, it can be numeric or string.

Adding new tests is also possible by importing XLSX files:

On the test device, the user has the opportunity to view a list of available tests, select one of them, conduct a test and report the result:



Note: in case of unsuccessful test, it is possible to create a ticket (issue), which will later be tied to this particular test.

In February, the functionality related to the monitoring and statistics of the activities of testers was significantly updated. The new Activity section provides extensive information on current testing sessions:

The Summary tab displays only the main statuses (Ative, Suspended, Crashed, etc.), and if you select a specific test session — the Activity Stream block — it is possible to track the tester’s work step by step:

Key events from the session history are interactive: thus, it is possible to immediately see the test description, sent a ticket, or get basic information about the application crash.

Note: the date of the sessions in the Activity Stream is indicated at the time of the session start, if the session lasted several days, then this should be taken into account if you want to view the history of the user's actions (that is, what event may occur on the second day, but not this day, and the day the session began - that is, the previous one).

Official guide for the service: http://ubertesters.com/step-by-step-instruction/

Rating: 8/10.

Continuous integration

Note: The following nuances of using services as part of continuous integration (continuous integration) in this review imply that it will be made on the basis of the solution proposed by Apple. That is, with the help of Mac OS X Server and Xcode bots - in detail on the organization of such integration, I will not stop, perhaps it will be a topic for a separate article. Those interested can familiarize themselves with the topic, for example, by referring to Apple’s official guide .

Access to the API to automate the download is included separately in the profile settings (the description of the API is located at http://ubertesters.com/upload-api/ )

Using the API is possible with curl:

curl -X POST http://beta.ubertesters.com/api/client/upload_build.json -H "X-UbertestersApiKey:PERSONAL_API_KEY" -F "file=@upload.ipa" -F "title=build title" -F "notes=build notes" -F "status=in_progress" -F 

Options:

  X-UbertestersApiKey- required, your personal 'API access KEY'. Please see item 1 file - required, path to your build file (.IPA or .APK) title - optional, your build title notes - optional, build description, plain text status - optional - pending - default, create a new revision only - in_progress - create and start revision stop_previous - optional - true - stop all previous revisions ( if any) - false - keep current statuses of previous revisions 

Note: in order to download and publish the new version in fully automatic mode, in the project settings ( Administration - Distribution Groups ) you need to create a default tester group, in this case, with the added key “status = in_progress”, the new version will be available immediately after publication all members of this group. Several groups can be set as default group at once.

Important: before publishing an application in the AppStore, at least remove the initialization of Ubertesters from the code, if the size of the application is also important, it is recommended to completely remove the framework from the project (the difference in the size of the distribution is slightly more than 1 megabyte).

Rating: 9/10.

Total cumulative score for all sections: 32 points.

PS Despite the lack of (yet!) Functional symbolization of fall reports, Ubertesters can be used much more efficiently than its competitors in projects where the importance of receiving feedback from testers and other team members (customers, managers, designers and others). This is especially important for projects with a distributed team.

The developers of Ubertesters have promised to take into account many of the comments and suggestions made in the article, but I, in turn, will try to update the material as these promises are realized.