Geekly Articles each Day
📜 ⬆️ ⬇️

TailSampler - parallel GET requests sent to Apache.JMeter



1. Purpose of the plugin «HTTP Request Tail»


The plugin makes it easy to load embedded resources, and allows you to perform specified GET requests in parallel . By making the test as close as possible to the work of the browser in terms of the composition of the downloaded resources and the method of loading these resources.

TailSampler helps out if needed:
')

2. Instructions for use


HTTP Request Tail converts the list of links into an HTML document, loading the embedded resources of which will create a GET request for each of the specified links.

TailSampler </ b> plugin and using it
Figure 1. Data preparation process for the TailSampler plugin and its use

  1. Launch Fiddler .
  2. Open a page in a web browser, the download of which must be simulated in detail in a test - it can be a page with Adobe Flash , with Adobe AIR , with Microsoft SilverLigth , with ActiveX components.
  3. Perform the necessary actions on the page.
  4. Copy from Gid -requests from Fiddler , which need to be executed in the test in parallel, as if they were executed by the browser.
  5. Insert links into TailSampler , parameterize if necessary.
  6. Get parallel execution of specified queries in the test, the degree of parallelism is configured.



3. Operation of standard JMeter html parsers


The standard way to get an HTML document using the HTTP protocol in JMeter is to use the HTTP Request sampler. In HTTP Request there is an easy way to request the embedded resources of the page - tick [v] Retrieve Embedded Resource s. The standard parser that forms links to the page resources is LagartoBasedHtmlParser . The parser can be changed in the htmlparser.classname setting of the jmeter.properties file.

The study of Apache.JMeter parsers for five popular sites is given in the article “Choosing an html-parser for Apache.JMeter”:


When using parsers, almost all the necessary resources are loaded. But for different sites the download is different. So if the website is implemented on Microsoft Silverlight , then the efficiency of the Apache.JMeter parser will be about 0%. Whereas, using TailSampler , it will be possible to apply a load similar to the work of the browser in a simple way.

3.1. An example of working with a complex site atlas.mos.ru



Figure 2. Website atlas.mos.ru in the browser - 85 queries to the main domain
Figure 3. Website atlas.mos.ru in JMeter - 2 requests to the main domain

Figure 4. Tracing query execution can be seen at webpagetest.org

Figure 5. The browser performs 85 requests to the main domain, and JMeter only 2 - efficiency 2,35% , see logs in google docs

When you open the site atlas.mos.ru , actively using AJAX , you can see the difference:


Thus, when using HTTP Request with setting [v] Retrieve Embedded Resources for the address 83 GET request will not be sent.


Not enough to be
"Last year's snow fell"

To simulate sending 83 GET requests, you will need to add 83 HTTP Request components to the JMeter script:


Added 83 HTTP Request will be processed one after the other - the browser sends requests for embedded resources in parallel :




Figure 6. Report on the download site pflb.ru in Firefox - http://www.webpagetest.org/result/160319_RQ_Q3W/1/details/ - you can see parallel download groups of 6 requests

Thus, using only HTTP Request, it will not be possible to completely reload the embedded resources so as to accurately measure the load time of the html page with all the subqueries.


Figure 7. HTTP Request does not allow to see the whole picture; HTTP Request Tail will help to realize the tail of subqueries

4. History of creation


The HTTP Request Tail plugin for JMeter was created as an alternative to the Tile Server - python service, see description below.

The first acquaintance with the Tile Server service happened when colleagues Zhenya Borodenkov and Maxim Konyshev talked about load testing a web project loading large images in slices and tiles. Then we solved the problem of load testing one of the versions of this web project using SilverLight on the client and streaming content from the server using the SOAP / MSBin1 protocol . We did not know how to send requests from the JMeter via the SOAP / MSBin1 protocol and process the answers to them. We discussed the options.

The story was like this:
- If Vova were here, he would say to use an intermediate service to form the necessary requests.

- Intermediate service, it is too difficult (answered them). Let's write a plugin for JMeter . Plugin - simple and reliable.

- That was when the previous testing was done, Andrey Pischulin wrote the Tile Server service in python , we still use this service, the service for working with tiles:
  • JMeter sends a list of links to the Tile Server server using the POST method via HTTP Request ;
  • Tile Server implements a web server, accepts a list of links, generates from links an html-document with a list of iframes that point to links and returns an html-document JMeter ;
  • JMeter parses the html document and performs the necessary GET requests as subqueries.
Let's do the same.

- Well, let's do an intermediate service to implement the work with SOAP / MSBin1 .


Figure 8. Tile Server

Looking ahead, I said, we did as well as I once did. We made a proxy service on .NET, which formed requests for SOAP / MSBin1 - JMeter sent commands to this service via SOAP / XML , and the service sent requests to the loadable node already via SOAP / MSBin1 and returned answers to JMeter .

And with a high load, the proxy service became a bottleneck, it could not generate requests and process responses so that the loadable servers would be depressed and lay down. The test servers received the load, but if they answered the proxy service in 10 seconds, the proxy service responded to JMeter in 110 seconds. According to statistics, the JMeter logs showed that the loadable service was depressed, and yes, the load was good, but the loadable service responded briskly, more cheerfully, as the JMeter logs showed. Prompt addition of detailed logging to the proxy service corrected the situation, but when the proxy service hung, then logging on it was delayed - it was necessary to scale the intermediate service or rewrite it completely, one copy did not pull on the role of “Tsar Cannon”.


Figure 9. Tsar Cannon

Proxy service has become the point of failure. Then we returned to the idea of ​​a plug-in for JMeter , and made a JSR223 JNA library and a proxy client on a .NET machine gun to work on the SOAP / MSBin1 protocol , it turned out great. The plugin no longer has to handle multiple incoming streams. Of course, there are more overhead costs for RAM, but it works faster.

Then the idea arose to write a sampler JMeter to java, to replace the Tile Server , all of a sudden and it is a point of failure under load. Even the name of the future sampler appeared “HTTP Request Tail” or “Tail Sampler”. Due to the poor knowledge of the English language, I heard the word “Tile” as “Tail”, did not understand a little, and here “tiles” and the word, which is translated as “tail”. Deaf phones, tail and tail, the image of a mermaid added to the picture. The idea was fixed, the idea is clear, the name is. It remained the least - to do. Here Sasha Perevozchikova Sanchez92 , a plugin development expert, helped.

I have no complaints about Tile Server , my colleagues say - this is a fast and reliable tool. The plugin was created out of curiosity and interest in the JMeter tool for me, and Sasha had to do something, or the girl missed.

All the names in this story are non-fictional, made explicitly or indirectly by the plugin “HTTP Request Tail”:



5. Description


5.1. Default settings



Figure 10. Default Settings

The default settings are:


Unused settings are settings for POST requests, the values ​​are not used in any way by the main request or by subqueries:


The main request is generated, but not sent, the settings for POST requests do not apply to it. Subqueries use the GET method; the settings for POST requests also do not work for them.

The remaining settings affect subqueries.

HTTP Request Tail is a successor to HTTP Request , a description of the settings can be found in the documentation for the HTTP Request :



5.2. Customized HTTP Request Tail



Figure 11. Configured HTTP Request Tail
Links to embedded resources are listed in the Embedded resources text box. You can specify relative and absolute links.

5.2.1. Absolute links

For a description of the format of absolute links, see RFC: https://tools.ietf.org/html/rfc3986 .

Absolute links begin with the protocol:

Other protocols will not be processed by HTTP Request Tail . Examples of absolute links:



5.2.2. Relative links

Relative references are complemented by field values:

An example of relative links:



5.2.3. Parameterization using variables and functions

To parameterize GET queries, you can use JMeter variables and functions. Example:


5.2.4. Unicode and html entity handling features

When forming the html-page of the links used html-screening. Therefore, when writing links, you can use:


The structure of the HTML-page is not broken, the links will be processed correctly and in full.
For a complete list of html entities for html4, see here:


If an entity from the specification is not escaped, then make a note to the plugin.

No need to pre-screen special characters. So if there is a need to specify the URL of the form:


This is the way to write - just & , pre-screen on &amp; do not:



5.2.5. Unicode for java

It is noticed that if there is a unicode-character in the address, for example, ®, Ω, π, ≈:


And in the Implementation setting there is a Java value, then in the subquery, the unicode character will be replaced by a square:


When running JMeter from Windows using a bat-file, windows-1251 is assigned to java as an encoding, I guess this is the reason for replacing the unicode-character with a square. To set the encoding, you need to specify in the bat-file the argument for java : -Dfile.encoding=UTF-8 . When using HttpClient4 and HttpClient3.1 such an unwanted transformation does not occur.

5.3. Generated response



Figure 12. The response to the main request - the generated response

The response to the main request is generated. There is no request, only the response body.

The body of the response is a html-document, text encoded in UTF-8, where for each link to the embedded resource an iframe tag is generated.

Sample document:

  <!DOCTYPE HTML> <html> <head> <meta charset="utf-8"> <title>Embedded resources</title> </head> <body> <iframe src="http://www.google-analytics.com/analytics.js"></iframe> <iframe src="/sites/all/themes/pro/static/img/icons.png"></iframe> <iframe src="sites/all/themes/pro/static/img/main_3_block90-s.png"></iframe> <iframe src="http://pflb.ru/sites/all/themes/pro/static/img/footer-shadow.png"></iframe> <iframe src="http://staticxx.facebook.com/connect/xd_arbiter.php"></iframe> </body> </html>


Figure 13. Executed query execution statistics

5.3.1. Plans to change

When writing an article about JMeter parsers, JMeter developer Philippe M. philmdot appeared in the comments and asked to post a defect to the recursive processing account:


I plan to post a defect, and even fix it. Making it so that for responses to requests from img tags there is no recursive search for references to resources. For JMeter parsing to work as well as the browser does. But if the iframe , then it is necessary to perform the parsing, and if just img , then a single query is executed.

And then it will be necessary to change the TailSampler plugin so that when generating, img tags are used instead of iframe tags.

Today Philippe wrote that he would like to see this code in the JMeter core, that it would be a good way to work with AJAX . I will try to catch this year, it will be an interesting experience.


5.4. Known effects



5.4.1. +1 query in the log

Although the root request is not sent, it is logged:



5.4.2. Visual deception for answer 302

If there is a Follow Redirect setting, then for each embedded resource with the response code 302 (Found):



5.4.3. Recursion

A rare but possible effect is the loading of embedded resources for an embedded resource:



There are plans to limit the recursion depth to a parameter settable from the plug-in interface to 3 levels by default. When downloading a picture ( img tag), the browser does not load embedded resources for it. Immediately, all the resources are wrapped in the iframe tag, and Apache.JMeter does not currently distinguish which tag was used for the request - it always parses the answer to the subject of the subqueries, if the Content Type of the response is appropriate.


Figure 14. The first request, test_server.ru, is visible, which was not actually executed (“+1 request” effect). The redirection cascade is visible (the “Visual deception for response 302” effect) and the subqueries for subqueries (the “Recursion” effect).

5.5. Temporal characteristics


If you uncheck [] Retrieve All Embedded Resources or don’t specify any links to Embedded resources , then the logs will say that the request went instantly, and the answer came instantly.

Description of time characteristics:



6. Project on Github




7. Project structure


Source code in the directory:

/ src / ru / pflb / jmeter

Other directories are auxiliary, used for ease of debugging the project.

7.1. The form


The TailUrlConfigGui form is a modification of the main HTTP Request form from Apache.JMeter 2.13, from which the fields for editing the request body and setting the parameter list have been removed, but one large field has been added for entering the list of links.
And the appearance of TailHttpSamplerGui is also a copy of the HTTP Request , where the new main control TailUrlConfigGui is now used.

It was not without copy-paste. I took this step so that the HTTP Request Tail is almost the same as the HTTP Request .

But in JMeter 3.0, the look and feel of the HTTP Request has been changed, the controls have been swapped. Now HTTP Request Tail is different HTTP Request for JMeter 3.0. But this does not affect the work.

7.2. Wrappers


I wanted to use the existing code to the maximum, but I didn’t want to deal with copy-paste. Therefore, the Wrapper classes were created, thanks to which the methods sample , notifyFirstSampleAfterLoopRestart and threadFinished became available.

7.3. Shielding utility


The code for the class that performs the screening is taken from ibm.com: www.ibm.com/developerworks/ru/library/se-prevent/index.html
The html-entity table has been extended with an RFC with a description of HTML4.


8. Installation


  1. Download plugin ru.pflb.jmeter.samplers.TailSampler.jar :
  2. Copy plugin to lib / ext directory for JMeter .
  3. Restart JMeter .


Figure 15. HTTP Request Tail plugin is now available in JMeter

Catalog example:

  D: \ TOOLS \ apache-jmeter-2.13 \ lib \ ext \
 D: \ TOOLS \ apache-jmeter-2.13 \ lib \ ext \ ru.pflb.jmeter.samplers.TailSampler.jar
 D: \ TOOLS \ apache-jmeter-3.0 \ lib \ ext \
 D: \ TOOLS \ apache-jmeter-3.0 \ lib \ ext \ ru.pflb.jmeter.samplers.TailSampler.jar


9. System requirements



9.1. To work with Apache.JMeter 2.13



Apache.JMeter 2.13 is built using Java 6. And if you build the TailSampler plugin using, for example, OpenJDK 1.7.0_09-icedtea, and run Apache.JMeter 2.13 + the assembled plugin on a computer with only Java 6, then Apache.JMeter 2.13 it will start, but the plugin will not. As a result, their bundle will not work. The question of building projects and libraries for different versions of Java and their compatibility deserves a separate instruction.

9.2. To work with Apache.JMeter 3.0



Apache.JMeter 3.0 is built using Java 7. The TailSampler plugin must be built using Java 7 or the version of Java that will be used to run JMeter and the plugin.

Source: https://habr.com/ru/post/312352/

More articles:


All Articles