Good time of day!

In this post, I will talk about how to quickly and easily write various scripts to check the state of the infrastructure using the tools Bash , Outthentic and Sparrow ...

Task - we have a server on which we install applications and do configuration settings. I want to write a script that will quickly give us the answer that everything is fine with the server, and the application is configured and working correctly. A sort of smoke test that will be useful to us when we search for problems or simply check that the next deployment has not broken anything. Anticipating possible questions, I know that there are already tools that do something similar ( inspec ), however, I want to talk about an alternative hike. (It will be interesting to compare).

Tool selection

So why bash ? Because it is quite simple to use and allows you to quickly and efficiently write all sorts of scripts, imho I would not use Bash for more complex tasks, but for this kind of problems it is quite suitable.

Then, what is Outthentic and how is it useful to us here? Outthentic is a scripting framework that allows you to quickly write, configure, and run your script (in this case, written in Bash, but also in other languages), just as importantly, Outthentic has a built-in DSL that is suitable for writing scripts in style automated tests that can be convenient when writing monitoring scripts.

And finally - why (or something like that) Sparrow and how will it help us? Sparrow is a platform and runtime environment for user scripts that allows you to distribute and customize ready-made scripts in the form of so-called Sparrow plugins. The main exhaust is that when our squeak is written and tested, you can pack it in the form of a plug-in, upload it to the Sparrow repository and transfer it further to the operations department and / or any other colleagues who want to use your script.

Practical example

The example is partly based on real practice, and partly deliberately simplified so as not to overload the article with unnecessary details. I have to do more or less of this kind of checking all the time, so I decided to write an automation script for them, so we’ll check what the target target server is on the target-server :

• tomcat is running, that is, it is visible in the process list

  
  

• http requests for some application resources (deployed on the server)
  return successful http code ( 200 ) - GET 127.0.0.1:8080/healthcheck

  
  
• a database server is available from the target server ( 192.168.0.2 ), at the access level on the tcp port (very often, due to incorrectly configured security policies, this may not be the case, which leads to an application being inoperative)

Yes, and it is important to note that all checks are performed directly on the target server:

 $ ssh target-server $ bash /path/to/check/script.bash 

Bash script

In this case, the script will be trivial:

 $ cat script.bash #!/bin/bash ps uax | grep tomcat | grep -v grep echo; echo timeout 5 curl -sL 127.0.0.1:8080/healthcheck -w "GET /healhcheck --- %{http_code}\n" -o /dev/null echo; echo timeout 5 bash -c "echo OK | telnet 192.168.0.2 3306" 

By running the script on the target server, we’ll get something like this in the output: (at this stage no checks are performed, just make sure the script is working):

 $ bash script.bash GET /healhcheck --- 200 tomcat 8264 0.0 32.1 2222884 326452 ? Sl Sep14 4:04 /usr/lib/jvm/java-1.8.0/bin/java -Djava.util.logging.config.file=/usr/share/tomcat8/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx128M -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/share/tomcat8/endorsed -classpath /usr/share/tomcat8/bin/bootstrap.jar:/usr/share/tomcat8/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/tomcat8 -Dcatalina.home=/usr/share/tomcat8 -Djava.io.tmpdir=/usr/share/tomcat8/temp org.apache.catalina.startup.Bootstrap start Trying 192.168.0.2 ... Connected to 192.168.0.2. Escape character is '^]'. Connection closed by foreign host. 

Script output check

The whole point of our monitoring is the possibility of consecutively launching several commands and then analyzing their output using a set of simple rules, here Outthentic comes into play.

First, install the package as a CPAN module:

 $ cpanm Outthentic 

Next, slightly modify our script so that it can be run through Outthentic:

• Rename the script to strory.bash - this is the agreement to have scripts in the Outthentic framework:

 $ mv script.bash story.bash 

• run the script through the console client strun , which comes with the Outthentic framework and actually runs the scripts:

 $ strun 

We'll get the output, just like when we ran the script directly. So far, the benefits of Outthentic are not obvious. We reach the use of DSL. Create some simple check rules for validating the output of the script and put the rules in the file story.check :

 $ cat story.check GET /healhcheck --- 200 tomcat8 Connected to 192.168.0.2 

Run the strun again:

 $ strun 2017-09-18 17:39:55 : [path] / GET /healhcheck --- 200 tomcat 8264 0.0 32.1 2222884 326452 ? Sl Sep14 4:04 /usr/lib/jvm/java-1.8.0/bin/java -Djava.util.logging.config.file=/usr/share/tomcat8/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx128M -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/share/tomcat8/endorsed -classpath /usr/share/tomcat8/bin/bootstrap.jar:/usr/share/tomcat8/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/tomcat8 -Dcatalina.home=/usr/share/tomcat8 -Djava.io.tmpdir=/usr/share/tomcat8/temp org.apache.catalina.startup.Bootstrap start Trying 192.168.0.2 ... Connected to 192.168.0.2. Escape character is '^]'. Connection closed by foreign host. ok scenario succeeded ok text has 'GET /healhcheck --- 200' ok text has 'tomcat8' ok text has 'Connected to 192.168.0.2' STATUS SUCCEED 

We see that the verification rules worked and that the output from the script successfully passed all the checks, and that’s all we need from our monitoring. The strun report strun customizable and has several options, for example, we can choose a more concise output that will give all the details only in case of an error:

 $ strun --format production 

Here is what the report will look like if, for some reason, we do not have a tomcat server running:

 $ strun --format production 2017-09-18 17:44:43 : [path] / not ok text has 'tomcat8' GET /healhcheck --- 200 Trying 192.168.0.2 ... Connected to 192.168.0.2. Escape character is '^]'. Connection closed by foreign host. STATUS FAILED (2) 

Parameterization of the monitoring script

Outthentic is also convenient because it allows you to simply add and customize input parameters for your scripts. Suppose, theoretically, we want to specify the host and port for the database server.

Add default values ​​of input parameters via suite.yaml - file for storing default settings in Outthentic terminology:

 $ cat suite.yaml --- db_server: ip_address: "192.168.0.2" port: 3306 

In this case, the hierarchy of the configuration file can be arbitrary, I just want to show how easy it is to transfer input parameters described by a hierarchical data structure using Outthentic and even use them inside Bash scripts (without additional parsing):

Slightly change the monitoring creak that he would take his input parameters from outside :

 $ cat script.bash #!/bin/bash db_server_address=$(config db_server.ip_address) db_server_port=$(config db_server.port) # ...      # ... timeout 5 bash -c "echo OK | telnet $db_server_address $db_server_port $db_server_port" 

Now we can run our script with default parameters:

 #       ip address 192.168.0.2   3306 $ strun 

Or override the parameters via the command line:

 $ strun --param db_server.ip_address=192.168.0.3 --param db_server.port=3307 

Go ahead, all we have to do is transfer the script to operation for all interested parties, for this we need Sparrow.

Sparrow plugin script distribution

Sparrow provides two basic scripting distribution options - through the SparrowHub public repository and through private repositories built on the use of remote Git repositories.

Most likely , when we write purely internal scripts, the second method is more suitable for us. It is also more simple, since it requires only that the script source code be in some remote Git repository, what will we do this:

 $ git init . $ git add . $ git commit -a -m "outthentic monitoring script" 

Having added the main project files (story.bash and story.check), it remains for us to set up a file with meta data (which actually makes it clear that this is not just a script, but a Sparrow plugin):

 $ cat sparrow.json { "name" : "server-check" "description" : "check server health" } 

In fact, a file with meta data may contain a much larger number of parameters, but for the sake of simplicity we will limit ourselves to the minimum set.

Ok, we actually made our first Sparrow plugin, it remains to send files to git remote:

 git add sparrow.json git commit -a -m "add sparrow meta file" git remote add origin $remote-git-repository git push -u origin master 

Using a ready-made monitoring script as a Sparrow plugin

This is the most interesting part in the sense that it shows how Sparrow facilitates the process of installing and integrating your scripts with third-party commands.

To begin with, in order to use the plugin created by us on some target server, we need to install the Sparrow client on this server:

 $ cpanm Sparrow 

Then everything is simple, because the plugin is private and will not be downloaded from the common repository, we notify Sparrow about it:

 $ echo "server-check $remote-git-repository" >> ~/sparrow.list 

The pair of values ​​that we have to put in the local index file ~/sparrow.list is the name of the plugin (it doesn’t have to be the same as what we used in the previous part and the URL of the remote repository, where the plugin source code lies)

Now we are updating the sparrow index so that the plugin added by us becomes available:

 $ sparrow index update 

And install the plugin itself:

 $ sparrow plg install server-check 

Now we can run the plugin as is:

 $ sparrow plg install server-check 

Or, passing it the parameters:

 $ sparrow plg install server-check --param db_server.ip_address=192.168.0.3 --param db_server.port=3307 

And finally, all the same can be run as a Sparrow task :

 $ sparrow project create monitoring $ sparrow task add monitoring app1 server-check $ sparrow task ini monitoring/app1 --- db_server: ip_address: "192.168.0.2" port: 3306 --- $ sparrow task run monitoring/app1 

The last launch option is convenient because you can create several tasks (with different configurations) to launch the same plug-in. In essence, a task is a named configuration for running a Sparrow plug-in, and a project is a group of tasks.

PS

That's all. If anyone is interested, this is what I have not said:

• Private Sparrow repositories can be configured not only through the local index files ~/sparrow.list (which is inconvenient with a large number of plug-ins), but also through Sparrow :: Nest - API for managing private Sparrow repositories.

  
  

• Sparrow plug-ins can be run remotely on servers (via ssh) with automatic pre-installation of the Sparrow client - welcome to the Sparrowdo project. There is another Perl6 API for Sparrow and much more (- :!

  
  
• Outthentic DSL allows you to create much more complex and interesting test rules than just matching a substring. Among them are the Perl5 regexp check, the range check and sequence of strings, as well as the dynamic rule generation using general purpose programming languages.

As always - questions, comments, suggestions, constructive criticism - is welcome.

Sincerely.

Alexey