Geekly Articles each Day
📜 ⬆️ ⬇️

Library for automating acceptance testing in mobile applications

Preamble


I work for a company that makes quite a lot of and, I’m not afraid of this word, a cumbersome mobile application with a solid history for several years and, therefore, with a fairly solid and monstrous code.
The flow of wishes from the customer is varied and abundant, and in connection with this, from time to time it is necessary to make changes even in those places that are not intended for this purpose, such as. Some of the problems that arise with this — regression bugs — deliver quite a few complicated clocks from time to time.
At the same time, for one reason or another, there is only manual testing and a fairly impressive number of testers on the project, and quite naive attempts to automate it remained only at the level of several rather trivial unit tests at the “Hello world” level.
In particular, the testing department has an impressive cycle of tests for regression search, which is conducted quite regularly and takes a decent amount of time. Accordingly, once the task arose to somehow optimize this process. This will be discussed.

Honestly, I do not remember which tools for automated acceptance testing I watched and why they did not fit me. (I would be very grateful if someone in the comments prompts interesting solutions to this - I probably missed something very worthwhile) I can say one thing for sure - because our application, in fact, a thin client, is very many cases impossible (or at least I don't know how to) cover with unit tests and need something else. One way or another, it was decided to write my own library to automate acceptance testing.


About the system itself and its use


So, this system must meet the following requirements:

')
Large-block, the testing system should consist of the following blocks:



A potential tester should act as follows:


In this article I would like to dwell in more detail on the TestCore module. It can be described in approximately the following scheme:



Sequencing:


And from this moment more:

The test case is a structure of the following form:

         .


The test action can be a macro call, arithmetic operations or assignments. For example, here are some simple examples that I used to check the correctness of the system:

 #simpleTest /*simple comment*/ send("someSignal") waitFor("response")->timeOut(3.0)->failSignals("signalError")->onFail("log fail") #end 


 #someTest paramA,paramB log("we have #paramA and #paramB") #end 


 #mathTest foo = 1 + 2 bar = foo * 3 failOn(bar == 9, "calculation is failed") failOn(((1 + 2) < 5),"1 + 2 < 5 : false") failOn(NOT((1 + 2) > 5), "1 + 2 > 5 : true") failOn(NOT("abc" == "def"), "true equality of abc and def") failOn("abc" == "abc", "false equality of abc and abc") failOn(1 == 2, "compeletly wrong equality") #end


The send, log, waitFor, and other operators — these are macros — about which I wrote above — are essentially native methods, the writing of which lies on the programmer’s shoulders, and not on the tester’s.
Here, for example, the logging macro code:

 @implementation LogMacros -(id)executeWithParams:(NSArray *)params success:(BOOL *)success { NSLog(@"TESTLOG: %@",[params firstObject]); return nil; } +(NSString *)nameString { return @"log"; } @end


But the macro code FailOn is essentially an assertion.

 @implementation FailOnMacros -(id)executeWithParams:(NSArray *)params success:(BOOL *)success { id assertion = [params firstObject]; NSString *message = nil; if (params.count > 1) message = params[1]; if ([assertion isKindOfClass:[NSNumber class]]) { if ([assertion intValue] == 0) { *success = NO; TCLog(@"FAILED: %@",message); } } return nil; } +(NSString*)nameString { return @"failOn"; } @end


Thus, by writing a number of custom macros (macros from the above example and several others), it is possible to provide access to application data to check them with these tests, perform some UI actions, and send screenshots to the server.

One of the key macros is the waitFor macro, which expects the application to react to its actions. He, at the moment is one of the main points where the application code affects the execution of test examples. That is, for comfortable work of the library, it is necessary not only to write a certain number of project-specific macros, but also to introduce various status signals into the application code, about changing the state, sending a request, receiving a response, and so on. That is, in other words, prepare an application to ensure that it will be tested in this way.

Under the hood


And under the hood, the fun begins. The main part (lexer, parser, executor, execution tree) is written in C, YACC and Lex - so it can be compiled, run and quite successfully interpret the tests not only on iOS, but also on other systems that can C. If interest - I would try to write in a separate article about the intricacies of splicing non-native iOS languages ​​with my favorite IDE Xcode - all development was done in it, but in this article I’ll only briefly tell you about the code.

Lex

As you already understood, a small, but very proud interpreted scripting language was written to solve the problem, which means that the task of interpretation arises to its full height. For a qualitative interpretation of samopisnyh bicycles means not so much and I used a bunch of YACC and LEX.
On Habré there were several articles on their topic (to be honest, they didn’t have enough to start. There was a lack of some not too complicated, but not too obvious, use example. And I would like to believe that if someone there will be such a task - my code will help to take some kind of start):

A series of articles on writing a compiler with immersion in how it works ;
A small article about one simple example ;
Wiki about lexers ;
Wiki about YACC .

Well, and many other useful and not very links ...

In short, the Lexer task is to ensure that the input to the parser is not a symbol-for-symbol, but a sequence of tokens already defined with already defined types.

In order not to litter the article with long listings - here is the code of one of the lexers:
Lexer
In fact, he distinguishes between arithmetic signs, numbers and names, and he also passes them to the input of the parser.

YACC

In fact, YACC is a magic thing that translates the once written description of a Backus-Naur Form into an interpreter of the language.

Here is the code of the main parser:
Parser

Consider a piece of it, for understanding:

 program: alias module END_TERMINAL {finalizeTestCase($2);} ; module: /*this is not good, but it can be nil*/ {$$ = NULL;} | expr new_line {$$ = listWithParam($1);} | module expr new_line {$$ = addNodeToList($1,$2);} ;


YACC generates a syntax tree, that is, in fact, the entire test case collapses into one program node, which in turn consists of a case declaration, an action list, and a final terminal. The list of actions in turn can be minimized from function calls, arithmetic expressions, and so on. For example:

 func_call: NAME_TOKEN '(' param_list ')' {$$ = functionCall($3,$1);} | '(' func_call ')' {$$ = $2;} | func_call '->' func_call {$$ = decorateCodeNodeWithCodeNode($1,$3);} ; param_list: /*May be null param list */ {$$ = NULL;} | math {$$ = listWithParam($1);} | param_list ',' math {$$ = addNodeToList($1,$3);} ; math: param {$$ = $1;} | '(' math ')' {$$ = $2;} | math sign math {$$ = mathCall($2,$1,$3);} | NOT '(' math ')' {$$ = mathCall(signNOT,$3, NULL);} | MINUS math {$$ = mathCall(signMINUS,$2, NULL);} ;


In particular, for example, a function call is its name, its parameters, its modifiers.
In general, YACC itself is just passing through the token nodes and folding them. In order to do something about it, logic is hung on each pass according to some syntactic construction, which in parallel creates a tree in memory that can be used later. For understanding - in YACC notation
$$ is the result that is associated with the given expression
$ 1, $ 2, $ 3 ... are the results associated with the corresponding phonemes of these expressions.

And I’ll call listWithParam, mathCall, and so on - to generate and connect the nodes in memory.

Nodes

Source codes, as generators can be read here:
Node generation logic
Heading node

In fact, the node is required so that the graph, which they represent in aggregate, can be bypassed and, by the results of the detour, get some conclusion about the test performance. In fact, they must be an abstract syntax tree.

After folding the program expression in $$, we have just this very tree and we can calculate it.

Executor

Executor's code is stored here right here:
In fact - this is a recursive analysis of the tree in depth from left to right. Depending on its type, a node is interpreted either as a mathNode (arithmetic) or as a operationalNode (macro call, compiling a list of parameters).
The leaves of this graph are either constants (string, numeric), or variable names that form a lookup table at the initial parsing stage and acquire quick access indexes to memory cells in them, and at the computation stage they simply refer to these cells, or in the same way, via the bridge module, it requests the execution of a macro with the given name and a list of parameters (here you should not forget that the parameters should already be calculated by this point). And there are a lot of other routine and not so moments connected with the management of memory, data structures, etc.

Call example


Well, as a conclusion, I will give an example of how this miracle, in fact, is called from the native code:

 -(void) doWork { TestReader *reader = [[TestReader alloc] init]; [reader processTestCaseFromFile:[[NSBundle mainBundle] pathForResource:@"testTestSuite" ofType:@"tc"]]; [reader processTestHierarchyFromFile:[[NSBundle mainBundle] pathForResource:@"example" ofType:@"th"]]; [self performSelectorInBackground:@selector(run) withObject:nil]; } -(void) run { [[[TestManager sharedManager] hierarchyForName:@"TestFlow"] run]; }


All execution of the test plan is carried out in a separate thread - not in the main thing - and when writing macros that require access to application data, it is recommended not to forget about it. And at the output of the Run method there will be an object of the TestHierarchy class, which contains a tree of objects of the TestCase class with the name and execution status, and of course a bundle of beeches in the logs.

As PS


Oddly enough, testers took this thing with joy and now this thing is slowly preparing for implementation on the project. It would be great then to write about this wonderful process.
The source code for the TestCore module for iOS can be found at the github link: github.com/trifonov-ivan/testSuite
In general, a significant part of the work was done rather for self-education, but at the same time it was brought to some logical conclusion - so I will be very grateful if you can tell me some weak points in the idea - there may be some means that are more effective this task. What do you think - is it worth developing the idea to a full-fledged test service?

Well, yes - if any of the parts would need more detailed explanations - I would try to write about it, because a breakthrough appeared in the process. “But the fields of this article are too narrow for them” (c)

Source: https://habr.com/ru/post/231731/

More articles:


All Articles