Geekly Articles each Day
πŸ“œ ⬆️ ⬇️

How automatic testing is arranged in Mail.Ru Mail for iOS

image


Some time ago we told you about automatic testing of our Mail on Android and received a huge number of questions from readers. Today, let us open to you a part of our β€œinner kitchen”, which concerns auto-testing on iOS. To test each assembly, we conduct more than 500 autotests, which are performed in less than one hour. How did we implement them and why? What problems did you encounter and how could you solve them? Read all about it under the cut.


Content


Gerrit Code Review
Command system and CI communication
CI platform
Assembly
Test build - stage 1
Alpha version - stage 2
Beta - Stage 3
Release - Stage 4
Build applications
Application signature and provisioning profile
Problems
Checks and tests


UI tests
What should be the framework
MonkeyTalk modifications
Manage UI tests through scripts
Scaling - parallel launch
First version
Current version
Runs Isolation
Each on the simulator!
Common simulators
Workarounds for xcode, simulator, etc.
Stability of interaction with iOS-simulators leaves much to be desired
Simulator Pool and Load
Asynchronous simctl with no signs of completion of action
System dialogs in the application
Access to photos and contacts
Server response spoofing
Reporting
Documentation generation
System summary
disadvantages
Benefits


Project Checks
Same Targets
Localization
Categories
Test headers
Reporting


Static code analysis
scan-build
fbinfer


Profiling and startup timings
Unit tests


HipChat and Reporting
New Tasks Report
Report on the results of the assembly, tests, checks


Conclusion / Conclusions


Schematically, the workflow is as follows:


image


The diagram shows the initial, most rigid and critical steps of the task on the way to release. Let's start with one of the first components - Code Review.


Gerrit Code Review


For code review, we use the Gerrit Code Review. It works on a separate node, as with some actions it requires a lot of resources. The code is stored in a git repository. In addition, the repository is mirrored to an additional node, and a full disk backup via Time Machine is constantly maintained.


Workflow is closely related to Gerrit. Not only automated checks are conducted, but also QA testing. Approve / Fail at any of the stages is noted in Gerrit as a point: + 1 / –1. If the automated test fails or the test assembly fails to build, -1 is set.


If the first stage is successful, then the QA team takes over. They only interact with Jira, transferring the task to the Open or QA Approved state, so when switching, the corresponding Label is set to Gerrit. For this, a small plugin for Jira is implemented, it performs exactly this task - and nothing more.


Command system and CI communication


Gerrit allows you to use server-side hooks with an additional plug-in . We implement the following hooks on ruby:



Most of the hooks are used to notify the team of changes, as well as to synchronize the status of the task in Jira. For example, when creating a new patchset in Gerrit, the corresponding hook is called in which we translate the Jira task to the Code Review state, and the custom field Merge Approved is set to No, since automated checks on this patchset have not yet been performed. But there are more saturated implementations, for example comment-added . Through it, we run checks and build changes on the CI platform. For this, we have implemented our own system of commands that allows you to run certain jobs on CI on a given patchset. To facilitate the task of expanding the command set, we implemented a small DSL to describe the display of comments in Job'y. For example, this is the description of the command to run automated checks:


JobMapping.register do command 'flow' job BUILD_FLOW_JOB arguments_required processor do |args| { 'CHECKS_ENABLED' => args.key?('checks') } end processor do |args| { 'ANALYSIS_ENABLED' => args.key?('analysis') } end processor do |args| { 'UNITTESTS_ENABLED' => args.key?('unit') } end processor do |args| if args.key?('ui') tags = args['ui'] { 'UITESTS_ENABLED' => true, 'UI_TESTS_TAGS' => tags.nil? ? '' : tags.tr(',', ' ') } else { 'UITESTS_ENABLED' => false } end end end

Automated checks consist of several parts, and I want to be able to run what is needed now. Each JobMapping is tied to a specific team; it is specified by the command method. Also this command should be displayed in Job on Jenkins, it is set to guess which method ( job ). If the team has arguments, in JobMapping, their necessity is indicated by the arguments_required method.


Jenkins jobs are configured via environment variables. To do this, you can add a block processor to the mapping registration process using the processor method and convert the command invocation string into an env variable set.


Most often, these methods are enough to create the necessary mappings. Then laziness arises, and instead of β€œ! Flow ui unit” I just want to write β€œ! Tests” . Therefore, the broadcast appeared:


 JobMapping.register do command 'tests' translates_to '!flow ui unit' rebase end

Everything speaks for itself laziness comfort in its purest form.


CI platform


Most of the platform is written from scratch in Ruby. Each test or task is implemented by a separate module, which can be run both locally and on CI.


To run tasks, we use Rake. Thus, for each module there is a separate Rake-task in Rakefile.


If the developer can rely on console logs when running locally, he expects to see the final report and alert on CI. This is one of the basic principles - Jenkins Job logs are needed for the CI developer, not his client . Therefore, all test modules implement the same interface so that when running on CI, to get the final task state and error description (if any). So, for CI, we need a system that allows you to simply run the Rake task and take on some basic operations like cloning the code of a given revision, keeping the certificates, saving artifacts. Jenkins was mastered as a base. From it we use:



All jobs are a sequence of sh scripts that are executed after cloning our repository. And in most cases, the sh script in Job just calls Rake-task. All other operations β€” running xcodebuild, scan-build, modules for checking the project and source codes, UI-test systems, processing results β€” we implement ourselves within the framework of ruby-scripts.


There are two types of tasks on the CI platform:



Assembly


Build and validation / tests are independent of each other; after a successful build, the application and the task immediately go to the QA team to manually check specific new cases. At each stage on the way to release builds come out:



Only the first two stages work completely automatically, since they are released in HockeyApp, where you can do this as many times as you like. Beta and release are rarer and more demanding events, so they are manually launched if necessary.


Test build - stage 1


When the change passes the Code Review successfully, a test build is being prepared: a version with changes that are not yet merged into the main branch. A successfully assembled application is immediately loaded into HockeyApp under a separate ID of test builds.


Links to the version in HockeyApp are attached to the task in Jira. If the build is successful, the task is transferred to the Ready For Test state; if unsuccessful, go to the Open state for further refinement and correction of errors.


The QA team checks the assembly for compliance with the task on different devices and versions of iOS. If the changes are satisfactory, and by this time the automated checks, the Code Review and the design review, have passed, then pressing the Pass button in the Jira will immediately add the change to the main branch, thereby launching the second stage.


Alpha version - stage 2


When the change is merged into the main branch of the project, we need to release an alpha version of the application. The trigger for this assembly is a hook in Gerrit change-merged . Immediately after the merging change, Job builds the current alpha version of the application.


This Job has a delay of 10 minutes, which is needed in case of several dependent (or independent) tasks. Since the entire Job takes an average of 11-12 minutes, it is easier to wait for several tasks than to spend time on each one separately. The compiled alpha version will include all changes since the last successfully compiled version.


After successful completion, the latest version of the application is loaded into HockeyApp. The task in Jira is considered closed precisely then - as soon as it becomes available in the alpha version of the application. Therefore, after publication in HockeyApp, all tasks included in the version are closed. To do this, we refer to the API Jira. They also contain a comment with links to Job in Jenkins and the app itself in HockeyApp.


Failure to build in stage 2 is unlikely. If this happens, then most often because of the long failure of the API HockeyApp or Jira. This is rare, but ...


image


There are no problems with this, since the next pledged change will trigger the same Job, which will capture the previous changes.


Beta - Stage 3


The third stage is beta. Separate Job on Jenkins, he needs only the name of the branch from which the beta version will be collected. Manual build starts: although the beta is scheduled to within a day, it does not have a specific start time / period or trigger.


The result of the assembly automatically falls into HockeyApp and becomes available to the QA team. Immediately, a Job is launched for the release version from the same branch, the build is loaded into TestFlight and issued to internal and external testing.


Release - Stage 4


The fourth stage is release. A beta-hardened build, already loaded into iTunesConnect and TestFlight in the third stage, is released.


Build applications


Builds are done directly by xcodebuild. Each stage is lined with a separate configuration and arguments. For example, right at the beginning we completely disconnect Link-Time Optimizations to save time before an exit of a task to testing. Also at the first stage in the application there is a functional available that is activated in compile-time, which helps to track and debug problems, and also makes it easier for the QA-team to do some tasks:



Application signature and provisioning profile


At the moment with the automated assembly, we rigidly set the certificate used and the provisioning profile. Before the assembly, we go through the profiles available on the node, select the one we need exactly for the current assembly and set it in the project settings via environment variables. This step, of course, is done automatically by the scripts.


There were reasons for this. We are trying to upgrade to the new major versions of XCode as soon as they appear, and on one of the beta versions of XCode 7.0, we stumbled upon a bug in xcodebuild: it did not allow the automatic definition of the provisioning profile to be used consistently. Therefore, we implemented on the same Ruby module, which finds among all the available profiles the necessary Bundle Id, and then added to the project in place of the profiles env-variables. When assembling, we put profile identifiers in them.


On Xcode 7.0, this did not cause any inconvenience. The profile was still automatically substituted by the development environment if the env variable was empty. But with XCode 8.0 and Automatic Signing, our method could not work. With Automatic Signing, the environment automatically substitutes both in the debug and release versions of the development certificates and, accordingly, the profiles. If you set it hard on a release-version distribution-certificate, then xcodebuild refuses to work.


That is, it is assumed that when creating xcarchive, there is always a version subscribed to development. And then the second stage follows - exporting IPA, for it you can now set the configuration via -exportOptionsPlist and re-sign the application according to the desired distribution. We were initially not satisfied with such rules, so we turned off Automatic Signing and still substitute profiles manually. The configuration is as follows:


image


Problems


Xcodebuild has a limitation that we did not manage to get around: without prejudice to the duration of the execution, more than one process cannot be started. We played different env-variables and paths to the cache and data, but to no avail. Somewhere he still rests on shared resources.


With simultaneous assemblies, the successive work of each process is strongly noticeable. The total duration with two parallel xcodebuild is almost doubled. Therefore, we install one executor for each Jenkins slave. This eliminates the interference of two tasks without additional plug-ins, which do not always adequately work in Jenkins. At the same time, all our Job's use the dedicated executor to the maximum, there is no particular reason to run several tasks in parallel, they will only interfere with each other.


Checks and tests


For checks and tests there are corresponding Rake-tasks - for each check is separate. More specifically about the purpose and implementation of checks further, but for now - a high-level job description.


Jenkins uses the BuildFlow plugin, which allows you to run parallel tasks, merge them, create chains, etc. Any automated checks and tests run as part of the BuildFlow task. In the full version, in parallel, run:



The first three tasks are performed in one Jenkins Job, but the UI test system is slightly more complicated than running tests through xcodebuild. This is a separate chain, its first step - prebuild - is as follows:



image


Then, based on the distribution results, BuildFlow runs the required number of tasks in Jenkins, each with its own group of tests. Each Job receives UI tests, configuration, build, and scripts from Prebuild Job’s artifacts in one archive. After BuildFlow proceeds to the next task, the essence of which is to create a final general report and execute notifications in Jira, HipChat, Gerrit. For the results of the work of each of the ways, this task turns into artifacts. BuildFlow is configured, you can turn off any of the paths if necessary.


UI tests


What should be the framework


We believe that the framework for UI testing is needed:


  1. Elementary syntax with which not only iOS-developer and programmer is able to cope.
  2. The ability to reuse their work. When writing a UI test, commands are targeted using accessibility identifiers (most often). Ideally, we want to build our own library on top of the standardly available commands. This will allow us to abandon duplication of the elementary code, as well as ease the task with minor corrections in the application code. Has the ID changed? No Find and Replace: go to the interlayer and change one identifier there.

MonkeyTalk (currently purchased by Oracle, its further path is not yet known) came up according to our criteria and significantly outpaced the remaining bidders. It has its own elementary syntax, simple, which allows you to take common fragments of tests into separate scripts and reuse them. To write a test in this language, you can not understand programming at all, you just need to find out Accessibility Identifier - extract it with a utility like Reveal , Accessibility Inspector or Flex , and in extreme cases, check with the developer.


The final script almost completely repeats the test case described by the QA team. At the same time, MonkeyTalk supports JavaScript. Here, the scope of actions allows you to implement the very layer on top of the standard commands and element identifiers. Gradually, something very similar to Page Object was created . Each screen, dialogue and application element eventually got its own module in JavaScript, which greatly accelerated the development of UI tests.


At the same time, all the advantages of programming (cycles, conditions, functions) are available, although we are against it. The test should be simple, straightforward, and let it be slightly verbose. If the test fails, sometimes you have to contact the script, and you need the opportunity to clearly and precisely say that the letter’s name verification did not pass after deletion through the Edit mode. And to do this in the presence of a cycle in modes with three conditional divisions by commands and a pair of lambdas in the arguments can be very difficult and extremely unpleasant. However, in some tests we still use (but in moderation) cycles, which is really great shortening the length of the test script.


MonkeyTalk has another major advantage, which was very useful later in optimizing and speeding up the performance of a greatly increased base of UI tests.


MonkeyTalk modifications


MonkeyTalk does not depend on XCTest. Thanks to this key feature, we were able to implement a distributed system for performing UI tests. Other frameworks ( KIF , EarlGrey , and the official XCUITest) are launched via xcodebuild. With XCUITest everything is quite hard. If you look at the processes that are running during the tests, you will notice testmanagerd . He is fully engaged in current testing alone. If you run another session in parallel, it is because of testmanagerd that nothing will come out: the tests do not even start.


KIF and EarlGrey are based on XCTest. When we only began to think about parallel launching within a single node, there were not so many developments. FBSimulatorControl began to develop, only then pxctest appeared. Own attempts to anything more or less usable did not lead. The version with MonkeyTalk was easier and more accessible.


MonkeyTalk does not depend on either XCTest or XCUITest. It has a separate system for running and running a test. An agent with an http server is built into the application. On the side of the scripts, the runner is used, which parses the UI test script, sends the commands to the specified address and waits for a response with the result of the command execution.


As a result, we only need to run the application on the simulator, wait for the launch, then run the runner and process the results.


Over time, and the next iOS updates, we encountered some difficulties; to solve them, we modified MonkeyTalk for ourselves and implemented:



We also rewrote the interaction with some elements of UIKit and part of the network interaction, significantly speeding up the entire framework.


UI tests should work as close as possible to actual conditions and preserve the principles of the black box. Tapes and gestures in real conditions are performed by the user's hand. The closest thing we can achieve with automation is an imitation of clicks at the event level in the application. Initially, MonkeyTalk directly sent messages to UIControls and UIGestureRecognizers, which did not suit us at all, and sometimes simply did not allow us to cover the case.


We can also control all network interaction through the test script, enabling / disabling stubs, including when the application is started; more on that later. An additional advantage is an absolutely similar process for a real device, if we suddenly decide to switch to them.


Manage UI tests through scripts


Since MonkeyTalk has a special way to run, I had to implement additional control scripts. Their tasks are:



For the UI test script itself, we entered a title, which we will discuss in detail below. Now it’s worth pointing out that the device type and the iOS version on which the test is to be performed is specified in the header. If several arguments are given, then the Cartesian product will be taken. It is also possible to pass various arguments to the application to configure the application before testing.


All this is quite simple to implement, but UI testing is a long process, and our goal is to check every change in the project.


Scaling - parallel launch


First version


Initially, our self-written UI test execution system was characterized by low utilization of resources. We smashed 70 tests into folders. Each folder corresponds to a UI test category (for example, authorization). When launched, the folders were distributed to the Mac Mini available to us. At each of them, only one simulator worked at the same time, which was restarted and completely erased under the new UI test. Obvious disadvantages:



Then, slightly expanding the base of tests, covering the most necessary and basic, we began to look for ways to maximize the utilization of resources. And found.


Current version


image


, . :


  1. , .
  2. / , , .


- HTTP- MonkeyTalk . MonkeyTalk , . environment variable launch argument.


, . i7, 16 SSD - . , . . , env-, . UI-.


Jenkins API. env- . , , , . API.


β€” . , . , , . , Jenkins. :


  1. .
  2. json .
  3. .
  4. .

json UI-. UI- , . UI-. :


  1. .
  2. json , .
  3. ( ) .
  4. β€” Β« Β» => «», .
  5. UI- Jenkins , .
  6. , .
  7. .

UI- :


  1. .
  2. .

. , UI- . , . , . , .


UI- . , . , :



json , :


  INFO IOSMail::UITestsGrouper : Getting all available tests... INFO IOSMail::UITestsGrouper : Found 506 available tests. Took 0.8894939422607422 secs. INFO IOSMail::UITestsGrouper : Reading stats json from ./ui-tests/uitests_stats.json... INFO IOSMail::UITestsGrouper : Gluing available tests and stats... WARN IOSMail::UITestsGrouper : No statistics for ScrollToFirstLetterInListFromMiddle.mt_iPhone5s. Using mean tests time. INFO IOSMail::UITestsGrouper : Statistical total duration = 33603.8670472377 INFO IOSMail::UITestsGrouper : Computing capacities counters... INFO IOSMail::UITestsGrouper : Capacities counters received and computed. Took 3.4377992153167725 secs. INFO IOSMail::UITestsGrouper : Partitioning for 7 parts... INFO IOSMail::UITestsGrouper : Allocated partitions = [{:capacity=>4, :label=>"4sim", :tests=>[]}, {:capacity=>4, :label=>"4sim", :tests=>[]}, {:capacity=>5, :label=>"5sim", :tests=>[]}, {:capacity=>4, :label=>"4sim", :tests=>[]}, {:capacity=>5, :label=>"5sim", :tests=>[]}, {:capacity=>3, :label=>"3sim", :tests=>[]}, {:capacity=>4, :label=>"4sim", :tests=>[]}] INFO IOSMail::UITestsGrouper : Partitions constructed, took 0.039823055267333984 secs. INFO IOSMail::UITestsGrouper : Partitions sizes = [72, 70, 87, 73, 83, 46, 75] INFO IOSMail::UITestsGrouper : Partitions durations = [1157.0547642111776, 1160.5415018200872, 1157.1358834599146, 1159.2893925905228, 1158.044078969956, 1161.813697735469, 1158.7458768486974]

. 4β€”5 , API Jenkins. UI-, . , .


UI-. . , β€” 40 . . , 5 , , , . , , 10β€”15 UI-. . : 10 , .



UI- . . UI- .


!


, . ruby wrapper simctl β€” xcode iOS-. :


 simulator = Simulators::MRSimulator.new(test_name, type, runtime) simulator.launch simulator.prepare(app) simulator.install_app(app) If access_allowed simulator.enable_access(app) else simulator.disable_access(app) end simulator.run_app(app, arguments, env) #  MonkeyTalk runner simulator.shutdown #   (, -, , , ) simulator.destroy

- , simctl, , , . ́ syslog' . . β€” .



, . :


  1. Run
  2. .
  3. .
  4. .
  5. β€” , keychain, , (TCC).
  6. , , β€” 2.

, . , 20 . , .


keychain. data- appgroup- . . keychain . keychain-2-debug.db. , - , . keychain:


 def clear_keychain `sqlite3 ~/Library/Developer/CoreSimulator/Devices/#{@id}/data/Library/Keychains/keychain-2-debug.db "delete from genp;"` end

, ( , ), keychain β€” .


xcode, simulator . .


C iOS-


iOS- , . , , , , . , , - . . iOS- , dlopen, . β€” , Jenkins , . . , - .



, . -, , watchdog springboard. β€” 20 . , FBSimulatorControl Facebook. , preferences springboard , - Bundle Id. , springboard, preferences , ? β€” .


β€” . File IO. CI, , launchd, kernel_task CoreSimulator. . . XCode 8.0 - , -, 160 . , .


simctl


simctl, XCode, β€” . , , launch screen. (. β€” watchdog). simctl , syslog' . , , :




. MonkeyTalk , . . sqlite TCC.db, .



, , . . , .


, /. sqlite TCC.db, :


 def self.setup_access(simulator_id, app, disabled, simulator_set_path: "#{ENV['HOME']}/Library/Developer/CoreSimulator/Devices") tcc_path = File.join(simulator_set_path, "#{simulator_id}/data/Library/TCC/TCC.db") Utilities::FileMonitoring.wait_for_file_creation(tcc_path, :max_wait => 30) { raise Simulators::Errors::SimulatorTimeoutError.new("Timed out while waiting for TCC!") } rights = disabled ? '0' : '1' ['kTCCServiceAddressBook', 'kTCCServicePhotos', 'kTCCServiceCalendar'].each do |access_target| `sqlite3 #{tcc_path} "replace into access(service, client, client_type, allowed, prompt_count) values ('#{access_target}','#{app.bundle_id}',0,#{rights},1)" 2>&1` end end


UI- . CI , , , , . . URLProtocol, . URLProtocol' FakeResponse. UI- FakeResponse , . . :




Jira Hipchat UI- . , , Jenkins Job'. HTML- UI-, UI-, , UI- . ( UI- β€” //) :



UI- debug-. HTML- json-, , N . Job' json- - , , .


HTML- Jenkins, Jira :


image


Jira, , HipChat:


image


HTML- :


image



β€” -, :



HTML- . UI-. CI-.



BuildFlow β€” UI-. :



Prebuild, 5β€”6 , Gerrit. CI 506 UI- 55 .


disadvantages


, XCode. , . β€” :



- , . .


Benefits



, , :




Same Targets


alpha- . , , iTunes.


: Alpha, Beta AppStore. Alpha Alpha-. , - . . (, ) , . CI . - , . . , , . Alpha-, . json. :


 { "compare-sets": [ { "targets": ["MRMail-Alpha-Enterprise", "MRMail-Pub-Beta-Enterprise", "MRMail-AppStore", "MRMail-MonkeyTalk"], "exclusive": { "MRMail-Alpha-Enterprise": [ "^pod-packages/Reveal-iOS-SDK/", "^src/infrastructure/MainThreadGuard/UIKitThreadGuard\\.m$",

, xcodeproj , .


Localization


, . , . , , :



Build Phase, . . . - , CI, , Open.


Categories


Xcode β€” , , , . /. :



, , . , , . , , , . β€” Ruby, :



β€” , . :



, . . β€” β€” , , , . :


 { "rules": [ { "name": "GeneralExternalImplementation", "type": "ExternalImplCheck", "imports": { "AccountEnvironmentImpl+Protected.h": "AccountEnvironmentImpl.m", …….

β€” AccountEnvironmentImpl+Protected.h AccountEnvironmentImpl.m.


CI- .
xcodeproj .



, UI- β€” . . UI- :



UI-:


 # tasks: IOSMAIL-6020 # name:   # category: Compose # description:  ,   «»  ,  , ,    # .


, Jenkins Job, , Rake-. , , .


, HTML- , BuildFlow. It looks like this:


image
image


Static code analysis


scan-build


scan-build + xcodebuild. : . scan-build . , , , . β€” .


scan-build, , HTML-.


fbinfer


CI infer . β€” . 1 10 . . , . ́ . CI-, , , .


, CI, , . β€” , infer scan-build, .



. , , .


alpha- Downstream Job, .


Job , .
250 . InfluxDB, Grafana. , , , , - : , ( , ).


image


Unit-


Unit- β€” xcodebuild + xcpretty . HTML- xcpretty, , . Unit- β€” BuildFlow. 1451 Unit-, 10β€”11 .


HipChat



Jira + HipChat. , , .


, ,


β€” CI-. . . :



, . CI β€” Unit-, UI-, , β€” 55β€”60 .


, , /. , . , , ( ): , CI, . Continuous Reporting , . , , , .


/


. :



, , , , . , , , , . .


́ , , . .


image


')

Source: https://habr.com/ru/post/325552/

More articles:


All Articles