Run instrumental tests in the Firebase Test Lab. Part 1: iOS project

My name is Dmitry, I work as a tester in the company MEL Science . Most recently, I finished exploring a relatively fresh feature from the Firebase Test Lab — namely, instrumental testing of iOS applications using the native XCUITest testing framework.



Before that, I already tried out the Firebase Test Lab for Android and I really enjoyed it, so I decided to try putting the test infrastructure of the iOS project on the same rails. It was necessary to google a lot and not everything worked out the first time, so I decided to write an article-tutorial for those who still have it.

')

So, if you have UI tests on an iOS project, you can already try running them on real devices, kindly provided by the Good Corporation, today. Interested - welcome under cat.



In the story, I decided to build on some of the source data - a private repository on GitHub and the build system CircleCI. The name of the application is AmazingApp, bundleID - com.company.amazingapp. I cite these data immediately, to reduce the subsequent confusion.



If you have implemented different decisions in your project differently - share your experience in the comments.

1. Sami tests

Create a new project branch for UI tests:

$ git checkout develop $ git pull $ git checkout -b “feature/add-ui-tests” 

Open the project in Xcode and create a new Target with UI tests [Xcode -> File -> New -> Target -> iOS Testing Bundle], give it the talking name AmazingAppUITests.







Go to the Build Phases section of the created Target and check for Target Dependencies - AmazingApp, in the Compile Sources - AmazingAppUITests.swift.



A good practice is to highlight different build options into separate Schemes. Create a schema for our UI tests [XCode -> Product -> Scheme -> New Scheme] and give it the same name: AmazingAppUITests.



Build the created schema should include Target of the main application - AmazingApp and Target UI tests - AmazingAppUITests - see screenshot







Next, create a new build configuration for UI tests. In XCode, click on the project file, go to the Info section. Click on “+” and create a new configuration, for example XCtest. We will need this later in order to avoid dancing with a tambourine when it comes to code signing.







There are at least three Target in your project: the main application, unit tests (because they exist, aren’t they?) And the Target UI tests we created.



Go to Target AmazingApp, Build Settings tab, section Code Signing Identity. To configure XCtest, select iOS Developer. In the Code Signing Style section, select Manual. We have not yet generated the Provisioning profile, but we will definitely return to it later.



For Target AmazingAppUITests we do the same, but in the column Product Bundle Identifier we enter com.company.amazingappuitests.

2. Setting up the project in the Apple Developer Program

Go to the Apple Developer Program page, go to the Certificates section, Identifiers & Profiles and then in the App IDs column of the Identifiers item. Create a new App ID named AmazingAppUITests and bundleID com.company.amazingappuitests.







Now we have the opportunity to sign our tests with a separate certificate, but ... The build procedure for the build for testing involves building the application itself and building the test runner. Accordingly, we are faced with the problem of signing two bundle IDs with one provisioning profile. Fortunately, there is a simple and elegant solution - Wildcard App ID. Repeat the procedure for creating a new App ID, but instead of Explicit App ID, select Wildcard App ID as in the screenshot.







At this stage, work with developer.apple.com is over, but we will not minimize the browser window. We go to the site with documentation on Fastlane and read about the Match utility from cover to cover.



The attentive reader noted that in order to use this utility we will need a private repository and an account that has access to both the Apple Developer Program and Github. We create (if there is no such thing) an account of the type InfrastructureAccount@your.company.domain, create a powerful password, register it with developer.apple.com, assign it as the project administrator. Next, give your account access to the github repository of your company and create a new private repository with a name like AmazingAppMatch.

3. Configure Fastlane and match utilities

Open the terminal, go to the folder with the project and initialize fastlane as indicated in the official manual . After entering the command

 $ fastlane init 

you will be prompted to select available usage configurations. Select the fourth item - manual configuration of the project.







A new fastlane directory has appeared in the project, in which two files lie - Appfile and Fastfile. In a nutshell - in Appfile we store service data, and in Fastfile we write jobs, in the terminology of Fastlane, called lanes. I recommend reading the official documentation: one , two .



Open the Appfile in your favorite text editor and bring it to the following form:

 app_identifier "com.company.amazingapp" # Bundle ID apple_dev_portal_id "infrastructureaccount@your.company.domain" #   ,     iOS   Apple Developer Program. team_id "LSDY3IFJAY9" # Your Developer Portal Team ID 

We return to the terminal and start setting up match according to the official manual.

 $ fastlane match init $ fastlane match development 

Next, enter the requested data - repository, account, password, etc.



Important: when you first start the match utility will ask you to enter a password to decrypt the repository. It is very important to keep this password, at the stage of setting up the CI server it will be useful to us!



A new file has appeared in the fastlane folder - Matchfile. Open in your favorite text editor and bring it to the form:

 git_url("https://github.com/YourCompany/AmazingAppMatch") #       . type("development") # The default type, can be: appstore, adhoc, enterprise or development app_identifier("com.company.amazingapp") username("infrastructureaccount@your.company.domain") # Your Infrastructure account Apple Developer Portal username 

We fill it in this way if we want to use match in the future to sign builds for display in Crashlytics and / or the AppStore, that is, to sign the bundle ID of your application.



But, as we remember, we created a special Wildcard ID to sign the test build. Therefore, open the Fastfile and enter the new lane:

 lane :testing_build_for_firebase do match( type: "development", readonly: true, app_identifier: "com.company.*", git_branch: "uitests" #     development     . ) end 

We save, we enter into the terminal

 fastlane testing_build_for_firebase 

and see how fastlane created a new certificate and put it in the repository. Fine!



Open Xcode. Now we have the required provisioning profile of the Match Development com.company. * Type, which must be specified in the Provisioning profile section for AmazingApp and AmazingAppUITests.







It remains to add lane to build tests. We go to the repository of the plugin project for fastlane, which facilitates the configuration of export to the Firebase Test Lab and follow the instructions.



Copy-paste from the source example so that our lane testing_build_for_firebase looks like this:

 lane :testing_build_for_firebase do match( type: "development", readonly: true, app_identifier: "com.company.*", git_branch: "uitests" ) scan( scheme: 'AmazingAppUITests', # UI Test scheme clean: true, # Recommended: This would ensure the build would not include unnecessary files skip_detect_devices: true, # Required build_for_testing: true, # Required sdk: 'iphoneos', # Required should_zip_build_products: true, # Must be true to set the correct format for Firebase Test Lab ) firebase_test_lab_ios_xctest( gcp_project: 'AmazingAppUITests', # Your Google Cloud project name (    ) devices: [ # Device(s) to run tests on { ios_model_id: 'iphonex', # Device model ID, see gcloud command above ios_version_id: '12.0', # iOS version ID, see gcloud command above locale: 'en_US', # Optional: default to en_US if not set orientation: 'portrait' # Optional: default to portrait if not set } ] ) end 

For complete information on setting up fastlane in CircleCI, I recommend reading the official documentation once or twice .



Do not forget to add our config.yml new task:

 build-for-firebase-test-lab: macos: xcode: "10.1.0" working_directory: ~/project shell: /bin/bash --login -o pipefail steps: - checkout - attach_workspace: at: ~/project - run: sudo bundle install #   - run: name: install gcloud-sdk #  mac    gcloud command: | ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null ; brew install caskroom/cask/brew-cask 2> /dev/null brew cask install google-cloud-sdk - run: name: build app for testing command: fastlane testing_build_for_firebase #  lane     firebase 

4. What about our test bench? Configure Firebase.

Let's proceed, in fact, to what the article was written for.



Perhaps your application uses Firebase on a free tariff plan, perhaps it does not use it at all. There is absolutely no fundamental difference, because for testing needs we can create a separate project with a year of free use (cool, right?)



Login to our infrastructure account (or any other, without a difference), and go to the Firebase console page . Create a new project named AmazingAppUITests.



Important: In the previous step in the Fastfile in lane firebase_test_lab_ios_xctest, the gcp_project parameter must match the project name.







The default settings are fine with us.



We do not close the tab, registering with Gcloud under the same account - this is a necessary measure, since communication with Firebase is performed using the gcloud console interface.



Google gives $ 300 a year, which is equivalent to a year of free use of the service in the context of performing autotests. Enter the payment data, wait for the test write-off of $ 1 and get $ 300 to the account. After a year, the project will be automatically transferred to a free tariff plan, so it’s not worth worrying about losing money.



Let's go back to the tab with the Firebase project and transfer it to the Blaze tariff plan - now we have something to pay if the limit is exceeded.



In the gcloud interface, select our Firebase project, select the “Catalog” main menu item and add the Cloud Testing API and the Cloud Tools Result API.







Then go to the menu item "IAM and administration" -> Service accounts -> Create a service account. We issue the rights to edit the project.







Create API key in JSON format







We will need the downloaded JSON a bit later, but for now the Test Lab setup will be considered finished.

5. Setting CircleCI

A reasonable question is brewing - what to do with passwords? Reliably save our passwords and other sensitive data will help us the mechanism of environment variables of our build machine. In the CircleCI project settings, select Environment Variables.





And we get the following variables:

• key: GOOGLE_APPLICATION_CREDENTIALS
  
  value: the contents of the json gcloud service account key file
• key: MATCH_PASSWORD
  
  value: password to decrypt github repository with certificates
• key: FASTLANE_PASSWORD
  
  value: Apple Developer Portal Infrastructure Account Password

Save the changes, create the PR and send it to the review of its team name.

Results

As a result, the implementation of these simple manipulations we got a good, stable working stand with the ability to record video on the device screen at the time of passing the test. In the test case, I specified the iPhone X device model, but the farm provides a rich selection from a combination of different models and iOS versions.



The second part will be devoted to the step-by-step configuration of the Firebase Test Lab for the Android project.