How we test Sberbank Online for iOS

In the previous article, we met with the pyramid of testing and the benefits of automated tests. But theory is usually different from practice. Today we want to talk about our experience in testing the application code used by millions of iOS users. And also about the difficult path that our team had to go to achieve stable code.

The situation is this: suppose the developers managed to convince themselves and the business of the need to cover the code base with tests. Over time, the project has become more than ten thousand unit- and more than a thousand UI-tests. Such a large test base gave rise to several problems, the solution of which we want to talk about.
')
In the first part of the article we will familiarize with the difficulties encountered when working with pure (non-integration) unit-tests, in the second part we will consider UI-tests. To find out how we improve the stability of the test runs, welcome to cat.

In an ideal world with unmodified source code, unit tests should always show the same result regardless of the number and sequence of launches. And constantly falling tests should not pass through the barrier Continuous Integration server (CI).


In fact, you may encounter the fact that one and the same unit test will show either a positive or a negative result, which means “blinking”. The reason for this behavior lies in the poor implementation of the test code. Moreover, such a test can pass CI with a successful run, and later it will begin to fall on other people's Pull Request (PR). In such a situation, there is a desire to disable this test or play roulette and start the CI run again. However, this approach is anti-productive, since it undermines the credibility of the tests and loads CI with meaningless work.

This issue was highlighted this year at the WWDC Apple's international conference:

• This session describes parallel testing, analysis of code coverage for an individual target with tests, and also about the procedure for launching tests.
• Here, Apple told about testing network requests, mocking, testing notifications and speed of test execution.

Unit tests

To combat the flashing tests, use the following sequence of actions:



0. We evaluate the quality test code according to the basic criteria: isolation, correctness of mocks, etc. We follow the rule: when the test is flashing, we change the test code, and not the test code.

If this item did not help, then we proceed as follows:

1. We fix and reproduce the conditions under which the test falls;
2. Find the reason for the fall;
3. Change the test code or the code under test;
4. Go to the first step and check whether the cause of the fall is eliminated.

Reproduce the fall

The simplest and most obvious option is to run a problem test on the same iOS version and on the same device. As a rule, in this case the test is executed successfully, and the thought appears: “Everything works locally for me, I will restart the assembly on CI”. But in fact, the problem was not solved, and the test continues to fall for someone else.

Therefore, in the next verification step, you need to locally run all the unit tests of the application in order to identify the potential impact of one test on another. But even after such a test, your test result may be positive, but the problem remains undetected.

If the entire test sequence has completed successfully and failed to fix the expected drop, you can repeat the run a significant number of times.
To do this, on the command line, you need to start the loop with xcodebuild:

#! /bin/sh x=0 while [ $x -le 100 ]; do xcodebuild -configuration Debug -scheme "TargetScheme" -workspace App.wcworkspace -sdk iphonesimulator -destination "platfrom=iOS Simulator, OS=11.3, name=iPhone 7" test >> "report.txt"; x=$(( $x +1 )); done 

As a rule, this is enough to reproduce the fall and move on to the next step - identifying the cause of the recorded fall.

Causes of the fall and possible solutions

Consider the main reasons for the flashing unit tests that you may encounter in your work, the tools that enable them to identify, and possible solutions.

There are three main groups of reasons for the drop in tests:

Weak insulation

By isolation we mean a special case of encapsulation, namely: the language mechanism, which allows to limit the access of some program components to others.

The isolation of the environment plays an important role, since for the purity of the test nothing should affect the test entities. Particular attention should be paid to tests that are aimed at checking the code. They use entities with a global state, such as: global variables, Keychain, Network, CoreData, Singleton, NSUserDefaults, and so on. It is in these areas that the greatest number of potential places for the manifestation of weak insulation occurs. Suppose that when creating an environment for a test, a global state is set, which is implicitly used in another test code. In this case, the test verifying the code under test may begin to “blink” - because, depending on the test sequence, two situations may arise - when the global state is set and when it is not set. Often, the described dependencies are implicit, so you can accidentally forget to set / reset such global states.

In order for the dependencies to be clearly visible, you can use the principle of Dependency Injection (DI), namely: pass the dependency through the parameters of the constructor, or by the property of the object. This makes it easy to substitute mock dependencies instead of a real object.

Call asynchrony

All unit tests run synchronously. The difficulty of asynchronous testing arises due to the fact that the test method call “freezes” in the test, waiting for the completion of the unit-test's skoup. The result will be a stable drop test.

 //act [self.testService loadImageFromUrl:@"www.google.ru" handler:^(UIImage * _Nullable image, NSError * _Nullable error) { //assert OCMVerify([cacheMock imageAtPath:OCMOCK_ANY]); OCMVerify([cacheMock dateOfFileAtPath:OCMOCK_ANY]); OCMVerify([imageMock new]); [imageMock stopMocking]; }]; [self waitInterval:0.2]; 

To test such a test, there are several approaches:

1. Run NSRunLoop
2. waitForExpectationsWithTimeout

Both options require an argument with a wait time. However, it cannot be guaranteed that the selected interval will suffice. Locally, your test will pass, but on a high-loaded CI, there may not be enough power and it will fall - “blinking” will appear from here.

Suppose we have some data processing service. We want to check that after receiving a response from the server, it sends this data for processing further.

To send requests over the network, the service uses the client to work with it.

Such a test can be written asynchronously, using a mock server to ensure stable network responses.

 @interface Service : NSObject @property (nonatomic, strong) id<APIClient> apiClient; @end @protocol APIClient <NSObject> - (void)getDataWithCompletion:(void (^)(id responseJSONData))completion; @end - (void)testRequestAsync { // arrange __auto_type service = [Service new]; service.apiClient = [APIClient new]; XCTestExpectation *expectation = [self expectationWithDescription:@"Request"]; // act id receivedData = nil; [self.service receiveDataWithCompletion:^(id responseJSONData) { receivedData = responseJSONData; [expectation fulfill]; }]; [self waitForExpectationsWithTimeout:10 handler:^(NSError * _Nullable error) { expect(receivedData).notTo.beNil(); expect(error).to.beNil(); }]; } 

But the synchronous version of the test will be more stable and will allow you to get rid of work with timeouts.

For him, we need a synchronous mock APIClient

 @interface APIClientMock : NSObject <APIClient> @end @implementation - (void)getDataWithCompletion:(void (^)(id responseJSONData))completion { __auto_type fakeData = @{ @"key" : @"value" }; if (completion != nil) { completion(fakeData); } } @end 

Then the test will look easier and work more stable.

 - (void)testRequestSync { // arrange __auto_type service = [Service new]; service.apiClient = [APIClientMock new]; // act id receivedData = nil; [self.service receiveDataWithCompletion:^(id responseJSONData) { receivedData = responseJSONData; }]; expect(receivedData).notTo.beNil(); expect(error).to.beNil(); } 

Asynchronous work can be isolated by encapsulating into a separate entity that can be tested independently. The rest of the logic needs to be tested simultaneously. This approach will avoid most of the pitfalls introduced by asynchrony.

Alternatively, in the case of updating the UI-layer from the background-stream, you can check whether we are in the main stream, and what will happen if we make a call from the test:

 func performUIUpdate(using closure: @escaping () -> Void) { // If we are already on the main thread, execute the closure directly if Thread.isMainThread { closure() } else { DispatchQueue.main.async(execute: closure) } } 

A detailed explanation, see the article D. Sandell .

Testing code beyond your control
Often we forget about the following things:

• the implementation of the methods may depend on the localization of the application,
• there are private methods in the SDK that can be called by the framework classes,
• method implementation may depend on the SDK version

The above cases introduce uncertainty when writing and running tests. To avoid negative consequences, you need to run tests on all locales, as well as on versions of iOS supported by your application. Separately, it should be noted that there is no need to test the code, the implementation of which is hidden from you.

With this we want to complete the first part of the article on automated testing of the Sberbank Online iOS application devoted to unit testing.

In the second part of the article, we will talk about the problems encountered when writing 1500 UI tests, as well as recipes for overcoming them.

The article was written together with regno - Anton Vlasov, the head of the direction and iOS developer.