Geekly Articles each Day
📜 ⬆️ ⬇️

Testing Bash Applications

Recently, I faced the task of testing an application written in Bash. Initially, I decided to use unit tests in Python, however, I did not want to add extra technologies to the project. And I had to choose a test framework, the native language of which is the long-suffering Bash.

Review of existing solutions


When I turned to Google with a query: what is already available to choose from, the answer was not so many options. Here I will review some of them.

What criteria will I pay attention to?

  1. Dependencies: if we take the test framework for Bash, we would like it not to pull along: Python, Lua and a couple more system packages (and there are such).
  2. Installation complexity: since one of the tasks was the deployment of continuous-development and continuous-integration in Travis , it was important for me that the installation could be done in sane time and number of steps. Ideal options: package managers, acceptable: git clone , wget .
  3. Documentation and support: the application should work on different unix distributions, respectively, and tests should work everywhere, taking into account the number of different platforms, shells, their combinations and the speed of updating them, without the community and experience of other users I would not want to.
  4. The presence of fixtures in any form and / or (at least!) teardown() and teardown() functions.
  5. Sane syntax for writing new tests. In the world of Bash - a very important requirement.
  6. The usual conclusion for me about the results of the tests: how much has passed, what and where it fell, in which line (preferably).

assert.sh


One of the first options that I noticed was the small framework assert.sh . Pretty good solution: easy to install, easy to use. In order to write the first tests you need to create the file tests.sh and write to it just something (an example from the documentation):
')
Expand
 . assert.sh # `echo test` is expected to write "test" on stdout assert "echo test" "test" # `seq 3` is expected to print "1", "2" and "3" on different lines assert "seq 3" "1\n2\n3" # exit code of `true` is expected to be 0 assert_raises "true" # exit code of `false` is expected to be 1 assert_raises "false" 1 # end of test suite assert_end examples

Then tests can be run and see the results:

 $ ./tests.sh all 4 examples tests passed in 0.014s.


Of the benefits, you can additionally highlight:

  1. Easy syntax and use.
  2. Good documentation, examples of use.
  3. Ability to do conditional or unconditional skip tests.
  4. Ability to fail-fast or run-all.
  5. It is possible to make error output detailed (if you use the -v flag), initially it does not say which tests fail.

There are some serious drawbacks:

  1. At the time of writing this article on github, a red "build failing" icon was burning, it looks scary.
  2. The framework positions itself as easy, it lacks the setup() and teardown() methods for me, so that you can prepare the necessary data for each test and delete them upon completion.
  3. It is not possible to run all the test files from a specific folder.

Conclusion: a good tool that I would recommend to use if you need to write a couple of simple tests for the script. For more serious tasks - not suitable.

shunit2


Cases with the installation of shunit2 are somewhat worse. I could not find an adequate repository: there is a project on Google.Code, there are several projects on github of various neglect (3 years and 5 years), there are even several svn repositories. Accordingly, it is impossible to understand which release is the last one and where to download it from. But then the little things. What do the tests themselves look like? Here is a somewhat simplified example from the documentation :

Expand
 testAdding() { result=`expr 1 + 2` assertEquals \ "the result of '${result}' was wrong" \ 3 "${result}" }

Performance:

 $ /bin/bash math_test.sh testAdding Ran 1 test. OK


This framework has a number of unique features in its class:

  1. The ability to create test suites inside the code, such a function can be useful, there are tests for specific platforms or shells. Then you can use your own namespaces, like zsh_ , debian_ , etc.
  2. There are setUp and tearDown that are performed for each test, and oneTimeSetUp and oneTimeTearDown that are performed at the beginning and at the end of testing.
  3. A rich selection of different assert , it is possible to display line numbers where the test falls, using the ${_ASSERT_EQUALS_} , but only in shells, where line numbering is supported. From the documentation: bash (> = 3.0), ksh , pdksh , and zsh .
  4. It is possible to skip tests.

But there are a number of significant drawbacks that ultimately pushed me away:

  1. There is no activity in the project, all recent errors in Google.Code have been hanging without a solution since 2012, there have not been commits in the repository for three years. In general, the trouble.
  2. It is not clear what and how to put, the last release was in 2011. Related to the last item.
  3. The number of functions, even slightly unnecessary, so there are two ways to check equality: assertEquals and assertSame . A trifle, but surprising.
  4. It is not possible to run all the files in the folder.

Conclusion: a serious tool that can be flexibly configured and turned into an indispensable part of the project, but the lack of a coherent project management system of the shunit2 itself is shunit2 . I decided to look further.

roundup


I was initially interested in this framework because it was written by the author of Sinatra for Ruby . And I also liked the syntax of the tests, which resembles the familiar and familiar Mocha . By default, all functions that start with it_ inside the file are run. Interestingly, all tests are run inside your own sandbox, which allows you to avoid unnecessary errors. And here are the tests themselves, an example from the documentation:

Expand
 describe "roundup(5)" before() { foo="bar" } after() { rm -f foo.txt } it_runs_before() { test "$foo" "=" "bar" }


There are no examples of output, to see - you need to put and check, bad. Here are the advantages:

  1. Each test runs inside its own sandbox, which is very convenient.
  2. Easy to use.
  3. Installation via git clone and ./configure && make , can be installed in a local directory with the addition of $PATH .

And there were enough minuses:

  1. There is no possibility to make a source some common functions for all tests, but for the sake of fairness, it’s worth saying that with the help of a hack it is possible.
  2. It is not possible to run all the test files from the folder.
  3. The documentation is replete with TODO , and the work has not been going on for a couple of years.
  4. You can not miss the test.

Conclusion: absolutely average such a thing, you can not say that bad. But you can't call her good either. The functionality is similar to assert.sh , only slightly more. Where to use? If there is enough assert.sh functionality, but you need a function before() or after() .

bats


I will say right away, I chose this framework. Liked a lot. First of all - excellent documentation: examples of use, semantic versioning, separately pleased with the list of projects that use bats .

bats uses the following approach: a test is considered passed if all commands inside it return a code of 0 (like set -e ). That is, each line is a test of truth. Here are the tests written in bats :

Expand
 #!/usr/bin/env bats @test "addition using bc" { result="$(echo 2+2 | bc)" [ "$result" -eq 4 ] } @test "addition using dc" { result="$(echo 2 2+p | dc)" [ "$result" -eq 4 ] }

And the conclusion:

 $ bats addition.bats ✓ addition using bc ✓ addition using dc 2 tests, 0 failures


The output of information about tests using the flag ( --tap ) can be represented as text that is compatible with Test Anything Protocol , for which there are plugins for more programs: Jenkins, Redmine and others.

In bats , in addition to the special syntax for writing a test, there are many interesting things:


Objectively, there are a lot of bats for bats , and I have already listed them, but I could notice only one minus:


Conclusion: a quality tool, with virtually no weak points. I advise to use.

PS


If it's interesting to see what happened in the end, then here is the link to the tests for my free-time project git-secret .

Source: https://habr.com/ru/post/278937/

More articles:


All Articles