Geekly Articles each Day
📜 ⬆️ ⬇️

Nine Circles Automated Testing



I want to tell you about our automated testing system. The system in my understanding is not only code, but also hardware, processes and people.

I will answer the questions: What are we testing? Who is doing this? Why is this all happening? What do we have?
And then I will tell how everything works: I will describe the testing circles - from the first to the ninth.
')
What?

Our product is a corporate web application Service Desk, written in java.

Who!

I am the lead of the automated testing group; programmers whose code we are testing; manual testers, whose routine we eradicate; managers who believe that if the tests have passed, then everything is not so bad.

What for?

The goal of my group is to protect the product from the regression spiral of death .
The task of the group is the non-detection of defects by a maximum of interesting methods with a minimum amount of manual labor.

What do we already have?

900 short and not so application usage scenarios coded into tests.
CI Jenkins on six servers, three DBMS, two OS families and three browsers for which we write the product.

How it works?



Circle one - test script.

The smallest unit - the test begins with the script.
Javadoc test method, which we then collect and show the manual testers and PMMs.

  1. The title of the test - briefly and clearly describes what we check.
  2. Reference to the formulation for which the test was written; if it is not there, then the case is just a game of the sore imagination of the tester, we have corporate software, we are not joking.
  3. Test script in Russian, describing the three phases of the test - preparation, execution of the action and verification.

Note: The script is not a translation from java to Russian, on the contrary, the script is primary, and it takes longer to write a quality case than to encode it. In our case, it is usually written by the wrong person who encodes the test.

The first phase of the test is the preparation of the testing system.
Catalog catalog = DAOCatalog.create(true, false); catalog.set(Catalog.TITLE, “          ”) DSLCatalog.add(catalog); CatalogItem item = DAOCatalogItem.create(catalog); DSLCatalogItem.add(item);

The concept is a system model described by methods from the DAO; There are DSL prefixed methods that lead the system under test to the state described by the model.
Methods from DSL change the application through the system API, in our case it is the restfull service. The entire first phase of the test is carried out through the API.
Note: The rule is to write a DSL for each encoded test action in order to further execute it through the API.

The model is a map string, we try to avoid excessive complication of the testing system.

The second phase of the test is a test action using selenium, through a browser.
 DSLCatalog.goToCard(tester, item.get(CatalogItem.PARENT_CODE)); tester.clickElement(ADD_ELEMEMT); DSLForm.assertFormAppear(tester, DSLForm.DIALOG); tester.sendKeys(DSLForm.TITLE_INPUT, item.get(CatalogItem.TITLE)); tester.sendKeys(DSLForm.CODE_INPUT, item.get(CatalogItem.CODE)); tester.clickElement(DSLForm.SAVE_ON_FORM); DSLForm.assertFormDisappear(tester, DSLForm.DIALOG);

Note: Also, to ensure testability, each new interface element in the system has its own unique id within the page. Therefore, our xpath look like this:
  "//*[@id='description-value']"
or
  "//*[@id='description-input']"

Not so:
 "/html/body/table/tbody/tr/td[6]/table/tbody/tr[4]/td/div"

A class of problems with layout changes is solved.


The third phase is verification or verification. We perform it at least once through the interface:
 Assert.assertEquals("      ", template.get(ReportTemplate.TITLE), tester.getTextElement(DSLForm.TITLE_VALUE));

And then you can check already through the API.
 ModelMap map = SdDataUtils.getObject(sc.get(Bo.METACLASS_FQN), Bo.UUID, sc.getUUID()); String message = String.format("   uuid- '%s'   .", sc.getUUID()); Assert.assertEquals(message, team.get(Bo.TITLE), SdDataUtils.getMapValue(map, "responsibleTeam").get("title"));

Note: Our application has an asynchronous interface; in the community of automators, they write a lot about the problems with the expectation of the appearance of elements; Significantly accelerates the testing process.

The second and third phases can pass without using a browser, such as almost a hundred API tests.
Note: Someone should have a thought: is n’t too much attention paid to testing through the UI and on a raised application?
I will answer: the feature of our software is the high complexity of not individual functions, but business logic. Almost every test requires a database and a large context. In this case, unit-tests are not much gain in time and significantly lose on the complexity of writing.


The second circle. Environment test

Before each launch of the testing system:

Before each class with tests

After each test:


The third circle is the test hierarchy.




Note: Javadoc , going to the team
 mvn -f sdng-inttest/pom.xml javadoc:test-javadoc -e

It helps manual testers and PMAs to find out what tests are, allows not to search for git in the web-based interface and not to install an IDE.
The root looks like this:



Circle four - the organization of the project testing system.


60,000 lines of testing system code


Note: This is how the browser settings for tests look, I hope someone will be useful:
Firefox profile
  /** *  Firefox . * http://code.google.com/p/selenium/wiki/FirefoxDriver * @return    FirefoxDriver,   WebDriver. * @exception WebDriverException    . */ private WebDriver openFirefox() { FirefoxProfile firefoxProfile = new FirefoxProfile(); //   firefoxProfile.setPreference("browser.sessionhistory.max_total_viewer", "1"); // 3   ,     firefoxProfile.setPreference("browser.sessionhistory.max_entries", 3); firefoxProfile.setPreference("browser.sessionhistory.max_total_viewers", 1); firefoxProfile.setPreference("browser.sessionstore.max_tabs_undo", 0); //    firefoxProfile.setPreference("network.http.pipelining", true); firefoxProfile.setPreference("network.http.pipelining.maxrequests", 8); //  firefoxProfile.setPreference("nglayout.initialpaint.delay", "0"); //    firefoxProfile.setPreference("browser.download.manager.scanWhenDone", false); //   firefoxProfile.setPreference("browser.tabs.animate", false); // firefoxProfile.setPreference("browser.search.suggest.enabled", false); //  firefoxProfile.setPreference("image.animation_mode", "none"); //   firefoxProfile.setPreference("browser.bookmarks.max_backups", 0); // firefoxProfile.setPreference("browser.formfill.enable", false); //       //firefoxProfile.setPreference("browser.cache.memory.enable", false); firefoxProfile.setPreference("browser.cache.disk.enable", false); //     tslogs firefoxProfile.setPreference("browser.download.folderList", 2); firefoxProfile.setPreference("browser.download.manager.showWhenStarting", false); firefoxProfile.setPreference("browser.download.dir", new File("").getAbsolutePath()); firefoxProfile.setPreference("browser.helperApps.neverAsk.saveToDisk", "application/xml,application/pdf,application/zip,text/plain,application/vnd.ms-excel"); return new FirefoxDriver(firefoxProfile); }

We also have such a wonderful thing as unit tests for the core of the testing system.
Note: The testing system has been living for a year and a half, survived the move from svn to git, 300 commits, therefore 100 unit tests are vital

The two main usage scenarios are run from Jenkins and from eclipse.
Algorithm of work with launch configuration:
  1. Test system settings transmitted from maven - the highest priority. We write them to the config. If there is no config, then create.
  2. The remaining settings are taken from the config. If there is no config, then by default.
  3. Thus, the developers of the application and tests set up their personal config, and in CI, all the necessary parameters are passed through maven.

My configuration file:
 allowscreenshot=true needinit=true superlogin=naumen allowsaveconfig=true clickertime=3600000 testerpassword=atpassword testerlogin=atlogin superpassword= needdelete=true webaddress=http://localhost:9090/sd/ browsertype=firefox



Note: The vehicle is all grown-up - it has its own statements and usage scenarios.
“As an auto tester, I want to be able to run a testing application on a local computer in order to perform certain tests on a local or client stand, to further analyze the data obtained, or to develop new tests.”
“As a developer, I want the TS to check the project after each change, in order to identify problems and errors in a timely manner.
And so on.


We perform balancing of tests for their parallelization in a rather ugly way.
Startup profile in pom.xml
  <profile> <id>smoke-snap2-branch</id> ... <execution> <id>integration-tests</id> <configuration> <excludes> <exclude>all</exclude> </excludes> <includes> <include>**/selenium/**/advlist/*Test.java</include> <include>**/selenium/**/bo/*Test.java</include> <include>**/selenium/**/advimport/*Test.java</include> <include>**/selenium/**/user/*Test.java</include> <include>**/selenium/**/rights/*Test.java</include> </includes> ... </profile>


There are several such profiles for which packages are manually broken.

The fifth circle. Source Code Management.

We use this git branching model as the most appropriate for our needs.
Note: Tests are in the same place as the application code. Thus, uncoupling the branch, the developer unhooks the test code as well, this ensures the synchronization of the tests and the application. Even to this obvious decision, we did not come immediately.


The sixth circle. Separate build in CI Jenkins.

Assembly begins with a description. I believe that the rules should be as close as possible to the place of their application. Immediately - useful links.
Look


Useful plugins to customize builds:


Running tests:
 export DISPLAY='localhost:'$DISPLAY_NUM'.0'; mvn verify -P war-deploy,$TEST_PROFILE -Dext.prop.dir=$WORKSPACE/$PROP_DIR -Dwebaddress=http://localhost:$DEPLOY_PORT/sd -Dselenium.deploy.port=$DEPLOY_PORT -Dcargo.tomcat.ajp.port=$ARJ_PORT -Dcargo.rmi.port=$RMI_PORT

where war-deploy profile is raising an application using maven-cargo-plugin

Post-assembly operations.


The seventh circle. Assembly hierarchy.

A group of assemblies that provides services to programmers is called a branch.

Parent assembly - compile the project.
Second level child builds are unit tests for postgresql, mssql, oracle; static code analysis (CPD, PMD, findbugs); build war file (under one browser to save time) for UI tests.
The third level - UI tests in three streams and tests for migration.

Typical third branch

Note: Build migration testing is a simple and effective test in which we take backup of an ancient database with data and go up to the last war file. Allows you to make sure that all data migrations are completed and the application is upgraded to a non-empty database.


Eighth round, continuous integration system level

Our CI is 4 branches in which programmers can run tests by manually starting and pointing to their branch. And one main branch running on a commit to develop differs from the others with the presence of test builds on all supported DBMSs. Not earlier than two weeks ago, each branch was located on a separate server (we chose EX4 , since we do not need reliability, but the processor speed is critical; experiments showed that the three Jenkins nodes per server are optimal). After reconfiguring virtual screens, ports, and DBMS settings, we now have a common node space.
Download schedule looks like this:


Jenkins promoted builds plugin , allowing you to draw beautiful stars for assemblies, all of whose daughters passed without fallen tests. It turned out to be more useful than three hundred tests through the interface or a thousand unit tests. Before him, some irresponsible PMs released releases, despite the presence of fallen tests, persuasion and threats did not help. But as soon as each build has asterisks - the first signal system worked and once every two weeks the leitmotif of the day - wait for two stars . This resulted in increased demands on the speed of error correction by programmers and on the quality of the code of the testing system — random triggers now cost more.
PMs are meditating on this picture on the day of release.


The assembly using the Performance Plugin - monitoring the performance of typical operations, allows you to quickly find the worst errors related to system performance.
The result looks like this:


In the development of the assembly, automatically conducting a full load testing. But this is a topic for a separate article.

Note: Some more useful stuff:


Round nine, people and processes.

The most difficult thing in automated testing is to get a good case. It takes up to half the time. The main supplier is manual testers, we eat their cases and ask for more, but they have lots of other interesting work. Therefore, we write tests for defects, and in general each performed task in JIRA passes through my gaze - is it possible to write a test for it? We also have an Emma Plugin and three build counting builds - for uint tests, for UI tests and total. According to these reports, we write tests on the system API. But for the rest of the functionality, the report on coverage does not have any value - the specificity of our software is such that you need to focus on covering tests with not code, but requirements.
Simple coverage schedule


Development of features in the context of automated testing looks like this:
Unhook the git branch. Write feature code. Run tests in the branch. Fix tests in the thread. Get two stars. Drain with develop.

The principle that slowed down the speed of development, increased the readiness of the code for release and saved a lot of time for my group:
The programmer has the moral right to commit, the assembly of testing in the branches of which received two stars.
If the programmer uncoupled from the commit in which all tests passed, then his commit should have the same property.
At first, the principle was reinforced by a revert of commits and a ban on commits in develop.

This development process has imposed a limit on the total time for passing all tests - 2 hours. We do not have nightly tests and tests running on weekends, absolutely all tests pass a maximum of 2 hours. The automated testing group for the last six months worked in the following mode:
Write tests up to 2 hours - buy servers and parallelize - write tests up to 2 hours - optimize test code - write tests up to 2 hours - change servers for faster - write tests ...

Afterword


All that I have said is not my merit. My contribution to the test code is a third of the lines. In CI - a quarter of configs. Two dozen people worked on the testing and continuous integration system - from technical support and testers to development managers and analysts.

I want to hear from you not only questions, but also wise advice, useful settings and cool ideas. Just remember about my role in the project - I can change anything in the code, a lot of things in CI, something in the processes. People I can not change.

If there is a willing not only to give valuable guidance, but for example, learn how to do it all yourself - or teach me how to work properly - welcome to the PM, I'm looking for a colleague.

Source: https://habr.com/ru/post/168451/

More articles:


All Articles