SWAT - DSL for rapid development of automated web application tests

Introduction

The task of automating testing is not new, but nevertheless the available tools in the field of testing web applications may have their limitations.

What if we have a hundred different web applications that need to be updated in a short period of time, but there are no tests to check their performance? Developing a UI test will take a lot of time, but simply making a curl request and checking that 200 OK returned is not enough.

We need a reasonable compromise, a simple, but at the same time sufficient universal means for the development of automatic tests. So SWAT was born.

Foundations idea

So, from the point of view of the end user, SWAT is a console client for running test scripts and some writing language ( DSL )
')
The user creates test scripts in a specific format, puts them in a separate directory, and runs. In general, it all looks like this:

$ swat /path/to/your/project/ $base_url 

Thus, a SWAT project is a folder with test scripts, as well as a certain base URL to which all http requests will be sent when testing a web application. OK, so far nothing new, many testing systems use similar layouts ... What is the essence of the SWAT system?

For a moment, let's forget that we are talking about SWAT and imagine that all we can do is perform a regular http request using the curl utility and analyze the answer using the grep utility:

 $ curl $base_url | grep foo-bar-baz 

Actually this is the quintessence of the SWAT framework.

The whole point is that the responses from the server being tested are perceived simply as text , in which you can perform a variety of searches, plus checking for successful http status is added. As shown by my practice of these two methods (roughly speaking, checking back 200 OK and see what came back) may be quite enough to write test scripts of various degrees of complexity, from superficial smoke tests to full-fledged functional

As my practical experience of application showed, SWAT is capable of much, we read further.

DSL Description and Data Structure

SWAT is based on a set of arrangements in terms of naming files and directories. It also provides a special syntax for writing rules for validating arbitrary text - in our case, the response from the server.

Let's start with a description of the file structure of a typical SWAT project.

So, the SWAT project, as mentioned earlier, is just a directory with files and subdirectories describing the test logic.

Well, all you have to do first is just create a project:

  $ mkdir swat-project 

I will introduce a couple of terms to simplify the understanding of the material.

Dealing with testing any web application, you can talk about the resources available in it or just the routes to which you can make a request using various http methods

So, let's say that the application we are testing has the following set of resources and methods for accessing them:

 GET / #   GET foo/bar # GET    foo/bar POST bar/baz # POST    bar/baz 

To describe this configuration with the help of SWAT you need quite a bit - to use the agreement adopted in SWAT projects, namely, resources are just directories, and methods are files. Actually it will look like this:

 $ cd swat-project $ mkdir -p foo/bar $ mkdir -p bar/baz $ touch get.txt $ touch foo/bar/get.txt $ touch bar/baz/post.txt #    ... 

I hope the principle is clear. Directory names correspond to http resource names, and file names refer to http method names. Why the names of the method files contain the * .txt extension will become apparent a little later, now we don’t pay attention to it.

Congratulations! We have just written a minimal set of tests that can be run (provided, of course, that we have a web service that will accept requests):

 $ cd swa-project $ swat ./ 127.0.0.1:3000 /home/vagrant/.swat/.cache/31999/prove/00.GET.t ........... ok 1 - GET 127.0.0.1:3000/ succeeded # response saved to /home/vagrant/.swat/.cache/31999/prove/KBoHRGrYRm 1..1 ok /home/vagrant/.swat/.cache/31999/prove/bar/baz/00.POST.t .. ok 1 - POST 127.0.0.1:3000/bar/baz succeeded # response saved to /home/vagrant/.swat/.cache/31999/prove/z5lXw_dCLa 1..1 ok /home/vagrant/.swat/.cache/31999/prove/foo/bar/00.GET.t ... ok 1 - GET 127.0.0.1:3000/foo/bar succeeded # response saved to /home/vagrant/.swat/.cache/31999/prove/_DnvkvcUBw 1..1 ok All tests successful. Files=3, Tests=3, 0 wallclock secs ( 0.04 usr 0.02 sys + 0.25 cusr 0.03 csys = 0.34 CPU) Result: PASS 

As we can see, SWAT successfully parsed the file structure, turn it into a set of http requests, and executed them. At the same time, the default responses from the server were validated for the presence of a successful http status, and in case of errors from the server, such tests would not have passed:

 $ cd swat-project #   : $ mkdir unknown $ touch unknown/get.txt #   SWAT : $ swat ./ 127.0.0.1:3000 /home/vagrant/.swat/.cache/32379/prove/bar/baz/00.POST.t .. ok 1 - POST 127.0.0.1:3000/bar/baz succeeded # response saved to /home/vagrant/.swat/.cache/32379/prove/Um9VB1zVyS 1..1 ok /home/vagrant/.swat/.cache/32379/prove/unknown/00.GET.t ... not ok 1 - GET 127.0.0.1:3000/unknown succeeded # Failed test 'GET 127.0.0.1:3000/unknown succeeded' # at /usr/local/share/perl/5.20.2/swat.pm line 81. # curl -X GET -k --connect-timeout 20 -m 20 -L -f -i -o /home/vagrant/.swat/.cache/32379/prove/TJO6JpsClL --stderr /home/vagrant/.swat/.cache/32379/prove/TJO6JpsClL.stderr '127.0.0.1:3000/unknown' # % Total % Received % Xferd Average Speed Time Time Time Current # Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (22) The requested URL returned error: 404 Not Found # can't continue here due to unsuccessfull http status code 1..1 # Looks like you failed 1 test of 1. # Looks like your test exited with 1 just after 1. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/1 subtests /home/vagrant/.swat/.cache/32379/prove/foo/bar/00.GET.t ... ok 1 - GET 127.0.0.1:3000/foo/bar succeeded # response saved to /home/vagrant/.swat/.cache/32379/prove/N0i8or4eCR 1..1 ok /home/vagrant/.swat/.cache/32379/prove/00.GET.t ........... ok 1 - GET 127.0.0.1:3000/ succeeded # response saved to /home/vagrant/.swat/.cache/32379/prove/eQpLp7zbAw 1..1 ok Test Summary Report ------------------- /home/vagrant/.swat/.cache/32379/prove/unknown/00.GET.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 Files=4, Tests=4, 1 wallclock secs ( 0.03 usr 0.02 sys + 0.28 cusr 0.02 csys = 0.35 CPU) Result: FAIL 

So, well, we see that SWAT works by successfully validating unknown resources, or rather by simply analyzing the http status of the responses that came from the server.

I want to say a few words about how SWAT makes requests. To generate http requests, use the curl utility. Why was curl chosen and not used, like some kind of Perl library? (for example, LWP ). IMHO is one of the important advantages in the case of curl - ease of use, good documentation and support (I recently started a non-critical ticket that I was answered in minutes). For all the time I use this utility, I have hardly come across types of http requests that cannot be implemented via the curl command line interface, except that curl web sockets cannot ...

So, SWAT uses curl to create http requests. This means that all the parameters required for the request you write in the terms of this utility. For this, the so-called request settings are used - swat.ini files that are created by the user inside the directories with resources and allow you to set additional parameters for each test script that determine its behavior, including the parameters of the curl call. I will give examples:

 $ cat bar/baz/swat.ini #     POST/GET  curl_params="-d name=Alexey -d age=39" $ cat foo/bar/swat.ini #  http  curl_params="-H 'Content-Type: application/json'" $ cat login/swat.ini #  http basic  curl_params="-ufoo-user:foo-password'" $ cat slow-route/swat.ini #  -         try_num=3 

More information on the list of parameters that can be set in swat.ini files can be found in the documentation.

It is important to note that the swat.ini files are regular bash scripts, in which, among other things, you can use standard bash constructs, there is a lot of room for creativity, to get acquainted with examples - I refer interested readers to the SWAT documentation pages.

Okay, everything is fine, but as you can see, we have never told to check the responses from the server, except for the truth, only the http status. All that we managed to understand by running the above set of tests is that a certain set of application resources is available and that the server returned a successful (200 OK) response when querying for these resources.

It's time to talk about SWAT DSL.

SWAT DSL

SWAT DSL is the second (after describing resources and setting requests) the main component of the framework. DSL allows you to check the answers that came from the server for compliance with certain rules, described in the form of single-line statements:

 RULE1 RULE2 #    ... 

If no match is found, a test passing error is generated. For each statement, the answer is checked again. This is a formal description of the validation process. In fact, everything is simpler, assertions are just plain lines of text or regular expressions that you would like to see in the server response.

I will give examples:

 #     #  ,   SWAT   , #        200 OK #      regexp: (red|green|blue) #      #   ,    ,     #    -    HELLO WORLD #   regexp: ^HELLO WORLD$ 

After these examples, I want to add the following:

• SWAT does not provide DSL directly, but uses a ready-made module for validating textual data - Outthentic-DSL . For a deeper understanding of the capabilities of SWAT DSL - use the documentation for this module.
• comments can be used when writing rules or (synonym for this tremin) verification statements
• when parsing a DSL script, lines containing only whitespace or empty lines are ignored
• SWAT uses a line-by-line check mode, which means that the response from the server is broken down into lines and each line is compared with the next rule. If at least one line corresponds to the rule - a test for this rule is considered passed, otherwise not passed, as reported by SWAT in the output from the console client.
• if you want multi-line checks, you can use SWAT blocks
• regular expressions syntax must match regular expressions in Perl , because SWAT DSL written in Perl
• check rules can be static, i.e. described in advance, as in the example given here, but SWAT provides the ability to set such rules dynamically, through the programming API through the generator mechanism
• In this article it is impossible to mention all the possibilities of DSL and the rules for writing verification statements - since they are quite diverse and numerous, again, I’m referring to the pages of the interested reader.

So, we have a DSL for writing test rules, the question remains, where will we create these rules? The answer comes up on its own - of course in the http method files! (We recall the remark about the extension * .txt in the names of the method files).

Let's rewrite, or rather update our previous example, by adding a few checks on the response from the server to when requesting various resources:

 $ cat get.txt HELLO USER! THIS IS INDEX PAGE $ cat foo/bar/get.txt I AM FOO-BAR $ cat bar/baz/post.txt POST TO BAR-BAZ OK 

Now run the tests again:

 $ cd swat-project $ swat ./ 127.0.0.1:3000 /home/vagrant/.swat/.cache/1422/prove/foo/bar/00.GET.t ... ok 1 - GET 127.0.0.1:3000/foo/bar succeeded # response saved to /home/vagrant/.swat/.cache/1422/prove/CmDiEY28iD ok 2 - output match 'I AM FOO-BAR' 1..2 ok /home/vagrant/.swat/.cache/1422/prove/bar/baz/00.POST.t .. ok 1 - POST 127.0.0.1:3000/bar/baz succeeded # response saved to /home/vagrant/.swat/.cache/1422/prove/rX8oenyA0j ok 2 - output match 'POST TO BAR-BAZ OK' 1..2 ok /home/vagrant/.swat/.cache/1422/prove/00.GET.t ........... ok 1 - GET 127.0.0.1:3000/ succeeded # response saved to /home/vagrant/.swat/.cache/1422/prove/PxDCnlbOA5 ok 2 - output match 'HELLO USER! THIS IS INDEX PAGE' 1..2 ok All tests successful. Files=3, Tests=6, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.24 cusr 0.00 csys = 0.27 CPU) Result: PASS 

As we can see, SWAT did its job and validated the responses from the server. This is the first introduction to SWAT DSL.

At the end of this article, I would like to mention one more interesting feature of SWAT, which allows you to write not only simple checks, but also full functional tests (remember at the very beginning I said that SWAT is capable of much? ...). So, let's talk about reused http requests.

Reusable http requests

Any more or less complex web application or service, being decomposed into many individual resources or routes, turns out to be strongly “bound” relative to these resources. What I want to say is that a request to a resource implies, in some cases, a preliminary request to another resource. Those. we are not talking about single and independent requests for individual resources, but about chains of such requests. Examples are obvious:

• access to resources requiring authentication / authorization
• creation of records in the database, requiring the prior removal of the old record
• conditional requests - delete the record from the database if it is there

And so on…

Obviously, with the “one test - one resource - one request” approach, such test scenarios are not feasible. It is necessary somehow before a request to one resource, to make requests to one or, possibly, to several other resources. How to be? And here we come to the aid of SWAT modules - reusable http requests.

In a general sense, SWAT modules are very similar to functions, having defined them once, you can call them arbitrarily once, in another place, if necessary, transferring some initial parameters to the module input and processing the result, SWAT implements this through the upstream / downstream history .

So, what you need to understand about SWAT modules:

• these are the same http resources as all the other ones created in the SWAT project, with the only difference that SWAT will never call (execute the request associated with the http resource) module
• another resource can invoke the SWAT module via the SWAT HOOKs API
• calling the SWAT module in another resource means that SWAT will first make the http request for the module resource and validate the answer (that is, it will run the entire chain of checks associated with this resource, starting from checking the http status to validation through the rules defined for the resource), and then it will make a request for the main resource in which the SWAT module was called.
• On the SWAT documentation page, a resource that calls a SWAT module is called an upstream story - the main story, and the SWAT module being called is a downstream story - a secondary story. The terminology is taken from jenkins-ci.org , although this model is inverted in SWAT, since in Jenkins downstream job is called after the main task
• SWAT HOOKs API - the ability to extend SWAT test scripts by writing Perl code, in this context the HOOKS API, also allows SWAT modules to be programmatically called

Let us explain all the above with a simple example. Suppose we have a web service that provides access to two resources:

POST login / - Resource for user authentication. If a valid username and password are sent to the server, the application returns the cookie session. For simplicity, we assume that the server accepts the login and password through the named fields of the POST request — user and password.

GET restricted / - Only authenticated users can have access to a protected resource.

We will write a SWAT test for such an application. An obvious test history here is to authenticate and gain access to the GET restricted protected resource.

Create a skeleton of the project, describing the resources and methods:

 $ mkdir swat-project $ cd swat-project $ mkdir login $ mkdir restricted $ touch login/post.txt $ touch restricted/get.txt 

Now we need to declare that the resource is a login / SWAT module, since we will want to call it before calling the GET restricted / resource.
Use the resource settings file for this - swat.ini file:

 $ cat login/swat.ini swat_module=1 

Setting the variable swat_module to one declares the resource as a SWAT module.

Add the parameters required for authentication. In this case, for the sake of simplicity of the example, the login and password are set explicitly, but SWAT will also allow you to take similar parameters from the project and set them elsewhere for security reasons.

 $ cat login/swat.ini swat_module=1 url_params="-d user=my-login -d password=my-password" 

And finally, the final touch - the server in case of successful authentication will return us cookies, we must save them somewhere, we will keep using the cookie-jar mechanism of the same curl utility, the final version of the resource request settings will be:

 $ cat login/swat.ini swat_module=1 # $test_root_dir -  ,    #      #        url_params="-d user=my-login -d password=my-password --cookie-jar $test_root_dir/cookie.txt" 

Ok, POST login / resource ready. If we know that, in addition to the successful status (200 OK), the server must return (and we know ;-)), we can add an additional check in the method file:

 $ cat login/post.txt hello user! 

Now we call POST login / before calling GET restricted /, this is done like this:

 $ cat restricted/hook.pm run_swat_module( POST => '/login' ); 

What have we done? We created a hook file for the GET restricted / resource and requested that the POST login / request be made at the very beginning .

What is left for us? Set up a GET restricted call / that he would use cookies to be created after a successful authentication using POST login /:

 $ cat restricted/swat.ini url_params="--cookie $test_root_dir/cookie.txt" 

Well, similarly to the POST login / resource, you can specify an additional check for the response from the server in the case of the GET restricted / request:

 $ cat login/get.txt restricted content 

What else would you like to mention in brief regarding the re-used SWAT requests (unfortunately the format of the article does not allow to fully disclose all the material)? I will list theses:

• cascading calls to SWAT modules — one module can call another, and so on, all as with normal functions
• data transfers (states) between the main resource and the SWAT modules it calls for. Yes, it is possible, because purely technically, the module and the code that caused it are executed in the same process, see the documentation
• passing parameters to the SWAT module input and accessing them within the code of the module implementing - this is called module variables, again see the documentation

Conclusion

I do not want to finish the article with a banal phrase - “now go to the project page , learn how to install the utility and start using it ...”, although it would not be bad :-)

But here is why you could start using SWAT in your work (IMHO):

• let me paraphrase freely Larry Wal - SWAT allows you to make simple things simple and complex workable. Notice how little code we had to write in our examples, and at the same time, SWAT is a powerful tool with the ability to extend to Perl. Whether you are a sysadmin with basic programming skills or an experienced developer - SWAT can be equally convenient and understandable to you
• SWAT is very pragmatic. It was coined on the basis of the realities of life for solving specific problems - namely, the rapid development of autotests for a large number of applications that are subject to frequent updates. All extra functionality has been removed from SWAT, it is as accurate as possible to solve the problem and does not provide anything to the load.
• SWAT is built on widely used and proven solutions, known to both system administrators and developers - namely, curl and bash. If you know curl, then you are already half-familiar with SWAT syntax, SWAT simply translates the settings in swat.ini files directly to curl (or almost so ;-)), if you know the basics of bash, it’s easy for you to describe the SWAT settings resources
• SWAT is supported by the satellite project Sparrow . SWAT - SWAT , SparrowHub , .

.

Thank.

PS web SWAT , , .

PPS !