Load Testing with Selenium

Introduction

In this article I will talk about the application of the tool originally intended for functional testing when testing the load web part of the electronic document management system (EDMS).

Why do you need it at all? We pursued two goals - the introduction of automated tests for our web applications and the creation of load tests based on functional tests.

Why was Selenium used for the test, and not a more suitable tool - LoadRunner, jMeter? With LoadRunner's, a load test was carried out, but the results were questioned - when emulating two hundred users, the page was loaded in 2 seconds plus or minus 2%, although when opening the same pages from the browser, the display occurred in more than 3 seconds. So I wanted to carry out load tests as close to reality as possible, and this can be done only with the help of full emulation of user behavior. Tools for functional testing with their work with browsers were just suitable for this — the site would open through a normal browser, i.e. as the user would do it.

About Selenium

For functional testing, Selenium was chosen for a simple reason - it is the best of free tools for functional testing. More specifically, it has good remote control support (Selenium Server, Selenium Grid), a lot of documentation (including in Russian ( habrahabr.ru/post/151715 habrahabr.ru/post/152653 ) and support for all major browsers (although this is more the merit of WebDriver).
')

Common architecture

The application is divided into levels (for clarity on the diagram, the elements of each level have meaningful names, and not just Test 1, Methods 2).

The first level - the level of "runner" tests. It just runs the tests. The settings configure the number of threads, the number of test runs, test classes.

The second level is the tests themselves. They perform business transactions - they log in, open lists of documents, open documents, go through the document tabs.

The third level is the level of work with web-elements. It contains atomic user operations for working with the system - opening a list of documents, moving to a specific document, working with document tabs.

To begin, the above actions will be sufficient to ensure minimal work with the system. In the future they will be added.

The division into these levels gives the following benefit - you can run tests with both the "runner" and without it - just running one test from the development environment. Bringing atomic user operations to a separate level will allow you to abandon writing tests in Java in the future, and develop your own DSL ( ru.wikipedia.org/wiki/Project-oriented_programming ) so that any people can write tests.

Running tests

The program for running jUnit tests is quite simple and consists of three classes - a class that performs the specified tests in its thread; jUnit test “listener” class to calculate test run time; class for the formation of threads and their launch.
Runner code

final int threadCount = readThreadCount(); final int invocationCount = readInvocationCount(); final List<Class> testClasses = readTestClasses(); ExecutorService taskExecutor = Executors.newFixedThreadPool(threadCount); for (int i = 0; i < threadCount; i++) { taskExecutor.execute(new TestRunner(invocationCount, testClasses)); } taskExecutor.shutdown(); taskExecutor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS); 

Test class code

 public class TestRunner implements Runnable { public void run() { JUnitCore jUnitRunner = new JUnitCore(); jUnitRunner.addListener(new TimeListener()); for (int i = 0; i < invocationCount; i++) { for (Class clazz : testClasses) { jUnitRunner.run(clazz); } Thread.sleep(invocationTimeOut); } } } 

Listener code

 public class TimeListener extends RunListener { private EtmPoint point; public void testStarted(Description description) throws Exception { point = EtmManager.getEtmMonitor().createPoint(description.getClassName() + "." + description.getMethodName();); } public void testFinished(Description description) throws Exception { point.collect(); } public void testFailure(Failure failure) throws Exception { log.error("Error in test.", failure.getException()); } } 

Tests

Tests were written simple, but, nevertheless, reflecting the user's work - opening a list of documents, opening a document card, navigating through the tabs of a card.

For writing tests used jUnit. Although you can also use TestNG, which supports parallel test execution (and this is a mandatory requirement for load testing). But jUnit was chosen for two reasons: 1) it was widely distributed in the company and used for a long time 2) you still had to write your own “runner” who would allow, without changing tests, run them in different threads (in TestNG, the parallel launch is configured in the tests themselves) and collect statistics on their implementation.

In addition to the tests, additional modules were written - pool webdrivers (here the word webdriver is used in Selenium terminology), pool of users, pool of documents, rule (in jUnit terminology) for taking screenshots in case of error, rule for issuing a webdriver test, rule authorization.

Pool webdrivers are the class that manages the receiving of a webdriver from Selenium's server and distributes them between tests. It is needed in order to abstract work with Selenium - tests will get webdrivers and give them to this pool. Webdrivers do not close at the same time (the close method is not called). This is because not to restart the browser. Those. Thus, the “reuse” of webdrivers by other tests is obtained. But reuse has its drawbacks - when you return the webdriver to the pool, you need to “clean up” behind you - delete all cookies or, if this cannot be done, perform a logout.
Just as it turned out later, this pool should restart the webdrivers for which the session ended. This is possible when an error occurred on the server side.

Pool of users is needed mainly during load testing, when you need to run the same tests under different users. He is only in a circle gives the username / password of the next user.
Pool of documents, as well as users, is needed mainly during load testing - it returns the id of documents of a certain type in a circle.

Rule for taking screenshots in case of an error is needed, as the name implies, to take a screenshot when a test fails. He saves it to a folder and writes to the log the name of the screenshot with the error stacktrace. It helps to further "see" the error, and not just read it in the logs. ( internetka.in.ua/selenium-rule-screenshot )
Rule for issuing webdriver'a test is needed in order to automate the receipt before the start of the test method and return at the end of the webdriver'a from the pool of webdrivers.

Rule authorization is also needed for automation, only now authorization - so that in each test method does not write login \ logout.

Statistics collection

To collect statistics, it was decided not to reinvent the wheel, but to use some of the ready-made frameworks. Search on the Internet, unfortunately, did not give a wide choice - only one tool - JETM (http://jetm.void.fm/), and it has not changed since 2009. Although it has good documentation and small advantages - remote connection via HTTP to view statistics in real time.
The configuration code for the monitor and the launch of the http console:

  BasicEtmConfigurator.configure(); EtmMonitor etmMonitor = EtmManager.getEtmMonitor(); etmMonitor.start(); HttpConsoleServer server = new HttpConsoleServer(etmMonitor); server.start(); 

Statistics were collected from two places: the total time was taken for the execution of test methods (from the “start-up” level) and the time for performing atomic user operations (from the third level). To solve the first problem, a RunListener successor was used, in which the methods for starting and ending the test were redefined and information about execution was collected in them.

The solution of the second problem could be carried out “in the forehead” - at the beginning and at the end of each method, the execution time of which needs to be written down, write the code for starting this time. But since there are already more than five methods now, and in the future they will be much more, I would like to automate this. For this, I used AOP, specifically AspectJ. A simple aspect was written that added counting the execution time of all public methods from classes with custom operations. Time was counted only by successfully executed methods, so that the methods that took off with an error in the middle of the execution did not spoil the statistics. One drawback was also found when collecting statistics on method names - since the methods for working with user operations were universal and were called by all tests, but the statistics needed to be collected by document types. Therefore, statistics were collected not only by the name of the methods, but also by their arguments that identify the document type.

Aspect Method Code

  @Around("execution(public static * <    >.*.*(..))") public Object profileMethod(ProceedingJoinPoint thisJoinPoint) throws Throwable { EtmPoint point = EtmManager.getEtmMonitor().createPoint(getPointName(thisJoinPoint)); Object result = thisJoinPoint.proceed(); point.collect(); return result; } 

The getPointName method generates the name of the time slice point based on the name of the method and its parameters.

Browsers for load testing

After writing all the tests, the question arose on which browsers to run it. In the case of functional testing, everything is simple here - you need to run tests on those browsers on which users will work. For us, this is IE 9. Therefore, I tried to run tests on IE with multiple instances of the browser on the machine, so that one computer could emulate the work of several users (In Selenium, one WebDriver is one instance of the browser). As a result, on my machine (4GB of RAM, 2.3 Core 2 Duo), only 4 IEs worked normally. That is not very good - to emulate two hundred users will need 50 machines. It was necessary to look for an alternative. And these are: a) other desktop browsers b) headless browsers.

From desktop browsers, FF and Chrome were tested. With Chrome, the situation was the same, plus for his work he demanded that WebDriver be launched in a separate process for each Chrome instance. What increased resource requirements. With FF, the situation was slightly better - 5 browsers worked normally without an additional launch of WebDrivers. But this situation is not much improved.

Then we would have to test headless browsers - browsers that fully work with the site (build DOM, run JS), but do not display it. In theory, they should work faster. Of all the headless browsers stopped at 2 - PhantomJS and HttpUnit. Prospectively looked like PhantomJS, based on Webkit. But in fact, he did not differ from FF in terms of resource consumption, but had the following disadvantages - sometimes he didn’t find the elements on the page and didn’t display the site correctly on screenshots. So it was not possible to understand why an error occurred. With HtmlUnit, everything is much simpler - its webdriver did not support alert, and this was critical for our web application.

As a result, returned to the use of FF in load testing. Although it also had problems with alert'ami - sometimes there were errors java.lang.Boolean cannot be java.lang.String (java.lang.ClassCastException) (here is a link to Google's error code code.google.com/p / selenium / issues / detail? id = 3565 ). Fix this error did not work, but it turned out to completely abandon the alert. So in the future you can try again to use HtmlUnit. Although all headless browsers have one common disadvantage associated with their specificity - they do not display the pages and so simply can not understand why an error occurred. The ability to take a screenshot does not help much - sometimes it is not informative.

Selenium configuration

Selenium's server supports launching in two modes - as a standalone server (default startup mode) and as part of a common network from Selenium servers - Selenium Grid (startup modes with –role hub and –role node). Since we needed to use a large number of computers, the first mode is not very suitable - in this case, each server will need to be managed separately. Ie, in fact, write your server manager. Although, at first, I was impressed by this option - in this case, we will have full control over which browser to run on which machine. But in the future I refused it - full control over the launch of browsers was not needed, plus Selenium Grid bribed with its ease of use. (link to Selenium Grid configuration page code.google.com/p/selenium/wiki/Grid2 )

As a result, we came to the following configuration: On one computer, Selenium was launched in hub mode with the additional parameter –timeout 0. This was necessary because sometimes the sessions were closed by timeout due to the long inactivity of the tests. On other computers, Selenium was started in node mode. For powerful computers capable of running 15 browsers, node Selenium was launched with an additional setting that allows you to run 15 copies of FF and indicating that you can work with 15 sessions at the same time.

Testing

The tests were carried out as follows: one instance of the browser was launched on one computer, which executed the test scripts several times and from which the execution time was taken. On the other computers, the same test scripts were run, but on several browsers. Such a division for measuring time is necessary in order that simultaneous operation of browsers does not affect the measurement result. As if doing the same measurements on several running browsers, then the time will be slightly longer.

A few words need to be said about the test scenarios and the timing of their execution. Each scenario included the opening of documents of each type. Those. first opened the incoming document, then the outgoing document, etc. Here you need to take into account the following situation: if you only need to remove the opening time of an incoming document, and only run this script on all the execution machines, then the time will be significantly less (50%) than if you removed the time while all scenarios were executed. In my case, most likely it was related to caching at the web application and DBMS levels. And the fact that few unique documents were opened. Perhaps, with a large number of different documents, the differences will not be so significant.

Ideally, I would like to get the distribution of users and documents as it will be in a really working system. Ie, for example, in a real system there will be 10 people working with incoming and 30 with outgoing. And in the load test also reflect this ratio - the number of tests for outgoing is three times more than with incoming documents. But since the system under test has not yet been commissioned and this data is not yet available, testing has taken place without taking them into account.

Summarizing

As a result of tests for 1 st, 16, 26 and 70 users, a schedule was drawn up for each scenario. So far, the number of users is not too large to draw accurate conclusions, but now we can trace the growth rate of time.
Dependence of the opening time of documents on the number of working users:


The dependence of the time of the list of documents on the number of working users:

Further tests will continue to build a schedule for up to 200 users. The result should be a graph similar to this one (taken from msdn.microsoft.com/en-us/library/bb924375.aspx ):


According to it, it will be possible to accurately determine the capabilities of the system and find its bottlenecks.