Greetings

In a previous article I wrote about SparrowHub - a repository of ready-made utilities for system administration. Well, time has passed, and now I want to tell you about HOW to develop these same utilities and upload them to SparrowHub for reuse by someone.

Before showing a concrete example of writing a plugin, I want to reveal a little the ideology of the Sparrow framework, which actually develops the scripts.

Firstly, Sparrow does not impose any hard DSL to develop scripts. You have a choice of three languages ​​supported by Sparrow:

• Perl
• Ruby
• Bash

Thus, you simply write a regular script that does the work you need and "wrap" it into a plugin, and, voila - it is ready to be downloaded to SparrowHub and for reuse. Well, I simplified a little, not quite surely so. While developing a script, you still adhere to some conventions (for example, on naming script files, etc.) so that your utility can be packaged in a plugin in the "sparrow" format.

Another feature of Sparrow plug-ins is the built-in system for testing script operation. Built-in, because out of the box you have the opportunity to perform additional checks on the operation of scripts, in addition to the banal exit code. This system of checks in some cases can be very useful. Details will become clear on a specific example.

Well, the third, related to the previous one, feature of Sparrow plug-ins, is that the result of their work is displayed in TAP format. In a nutshell, TAP is a special protocol (plus output format) designed for testing software modules; it is standard for writing unit tests in Perl, but is not tied to a specific programming language and has support in many systems and languages.

Thus, Sparrow has attempted (as far as successful practice will show;)) to combine the writing of system administration scripts with the system of testing the work of the scripts themselves . Similar ideas can be seen in various configuration management systems, for example, chef is the minitest chef handler and the new development chef inspec .

So, let's write the simplest Sparrow plugin.

Suppose you want to install the nginx package and configure the nginx server to be started. We divide the task into two scenarios - the actual installation of the package itself and the configuration of the server. By setting, we mean adding a server to autoload, starting the server and the simplest test that the server is available on port 80. The operating system is Ubuntu . Immediately make a reservation that the example is quite speculative and trivial, any system administrator or programmer is able to perform a similar task, but it is well suited to explain how Sparrow plugins are written.

So, we have two scenarios:

• package installation
• server tincture

Installation script

As the language in which we will write the plugin, choose Bash because In this formulation of the problem (very simple actions at the operating system setup level), it is most suitable for our subject area. In principle, all the same can be implemented in Perl or Ruby.

In terms of Sparrow, a running script is a story, in the sense that there is a script that can be run and that outputs something to stdout (leaves a trace, figuratively speaking). After executing the script, Sparrow allows you to do two types of checks:

• check that the exit code is successful (== 0)
• check that the text output of the script contains certain data (check for matching strings and regular expressions)

Script naming rules are simple - the base name must be story with the extension appropriate for the script language:

• Bash - story.bash
• Perl - story.pl
• Ruby - story.rb

So, in the case of Bash, we have this story:

 $ cat story.bash sudo DEBIAN_FRONTEND=noninteractive apt-get -y install nginx >/dev/null dpkg -s nginx | grep Status: 

Having launched this script, let's see its output:

 $ bash story.bash Status: install ok installed 

Now it is easy to write a test for the correct operation of this script. In Sparrow, this is done using so-called check files. Each story should have its own verification file, it should be in the same directory as the script being launched and called story.check. The content of the test file must be created in the language of the test rules Outthentic :: DSL , which was developed specifically for analyzing the text output of arbitrary programs. DSL offers a large set of features, of which in practice it may be necessary only to check for the line-by-line inclusion of the specified strings and for matching the regular expression. So, story.check will be like this:

 $ cat story.check install ok installed 

Great, now you can run our stories through Sparrow. This will be the prototype of our plugin. To do this, install the Sparrow module from CPAN, which provides the tools for developing and running Sparrow plug-ins.

 $ cpanm Sparrow 

Great, going to the directory where our script is located, strun launch it using the strun utility - this is the client for running Sparrow scripts:

  $ strun /tmp/.outthentic/30382/home/melezhik/projects/nginx-example/story.t .. # Status: install ok installed ok 1 - output match 'Status: install ok installed' 1..1 ok All tests successful. Files=1, Tests=1, 1 wallclock secs ( 0.01 usr 0.00 sys + 0.51 cusr 0.03 csys = 0.55 CPU) Result: PASS 

As already mentioned, we received the result of the work of our script in the form of a TAP report, according to which it is clear that the script worked correctly. What happened here? strun ran the script and checked that:

• completion code successful
• the script output stdout with the line 'Status: install ok installed'

Customization script

Next, in the same way, create a script to configure the nginx server along with the verification file. Since each story must be located in its own directory, we will create a separate directory for the configuration script:

 $ mkdir nginx-setup $ cd nginx-setup 

The setup script will be very simple:

 $ cat nginx-setup/story.bash sudo update-rc.d nginx defaults sudo service nginx start sudo service nginx status curl -sf -o /dev/null 127.0.0.1 

For starters, you can simply run the script by hand and see the output

 System start/stop links for /etc/init.d/nginx already exist. * nginx is running 

Now you can create a verification file:

 $ cat nginx-setup/story.check nginx is running 

Great, we’ll run our scripts again via the strun client:

 $ strun /tmp/.outthentic/32332/home/melezhik/projects/nginx-example/nginx-setup/story.t .. # System start/stop links for /etc/init.d/nginx already exist. # * nginx is running ok 1 - output match 'nginx is running' 1..1 ok /tmp/.outthentic/32332/home/melezhik/projects/nginx-example/story.t .............. # Status: install ok installed ok 1 - output match 'Status: install ok installed' 1..1 ok All tests successful. Files=2, Tests=2, 1 wallclock secs ( 0.01 usr 0.00 sys + 0.58 cusr 0.08 csys = 0.67 CPU) Result: PASS 

We are convinced that we have achieved a given goal. Nginx is installed, started, added to startup and is available on port 80.

An attentive reader will see, however, one feature in the script we have written, namely, the nginx installation script runs after the configuration script. The fact is that by default, all strun client-launched strun are considered independent and unrelated , the order of their launch is generally not guaranteed, and even more - different stories can be run in parallel mode (read the documentation for the prove utility parameters, which strun dispatches run tests).

Obviously, with this behavior, the launch of the plug-in for the first time, when the nginx package is not yet installed on the system, will give us an error.
What do we do? How to ensure the launch of scripts in a given sequence. We need to install the nginx package first, and then configure the server. For this, there is the concept of Sparrow modules or secondary ( downstream ) stories.

Major and minor scenarios

Secondary stories are scenarios that you can call BEFORE executing some basic scenario (the main ( upstream ) story). In our case, the main scenario will be the configuration script, and the secondary scenario will be the installation script of the package. Secondary scripts are never run directly by the strun client, you must make them call explicitly in the main scripts. Here is how it is done:

In order for the script to become secondary it is necessary to place the corresponding story in the directory called ./modules , so strun will understand that this story is secondary:

 $ mkdir -p modules/nginx-install $ mv story.bash story.check modules/nginx-install 

To call a secondary history, we use the so-called hook file - the main mechanism in Sparrow, which allows us to expand the logic of the client's strun and add calls to third-party code before executing the main script:

 $ cat nginx-setup/hook.bash run_story nginx-install 

Without going into the details of the Hook API here, I’ll just say that in this example in the hook file we call the secondary history of nginx-install using the run_story function. Note that the function argument is the path to the directory where the background history is located, from which the modules fragment is excluded.

So run the reworked code:

 $ strun /tmp/.outthentic/367/home/melezhik/projects/nginx-example/nginx-setup/story.t .. # Status: install ok installed ok 1 - output match 'Status: install ok installed' # System start/stop links for /etc/init.d/nginx already exist. # * nginx is running ok 2 - output match 'nginx is running' 1..2 ok All tests successful. Files=1, Tests=2, 1 wallclock secs ( 0.01 usr 0.00 sys + 0.54 cusr 0.06 csys = 0.61 CPU) Result: PASS 

Well, judging by the conclusion, we are convinced that the installation script runs before the configuration script, which is what we wanted.

Read more about the main and secondary stories in the documentation for the client strun .

Download the plugin to the central repository SparrowHub

After testing the scripts and making sure that everything works as it should, we are ready to share our script with the rest of the world and upload it to SparrowHub - the central repository. To do this, you need to take two simple steps:

• Register at https://sparrowhub.org
• login and get plugin developer token

After that, you can download plugins under your authorship.

At the root of our project, where the stories lie, we will create a file with the plugin metadata:

 $ cat sparrow.json { "name" : "nginx-example", "version" : "0.1.0", "description" : "simple nginx server installer for Ubuntu", "url" : "https://github.com/melezhik/nginx-example.git" } 

The file format is described in detail here , we will not dwell on it in detail. It is only important to say that Sparrow supports versioning plug-ins, allowing you to download and install different versions of the same plug-in.

From the scripts directory, run the command sparrow manager, pointing your IDs to SparrowHub

 $ export sph_user=melezhik $ export sph_token=FOO-BAR-BAZ-FOO-BAR-BAZ $ sparrow plg upload sparrow.json file validated ... plugin nginx-example version 0.001000 upload OK 

All perfectly. The plugin is downloaded and located in the SparrowHub repository. We can go to another server and use this plugin:

 $ ssh some-other-host $ sparrow index update $ sparrow plg install nginx-example $ sparrow plg run nginx-example 

Conclusion

It is not possible to describe in this article all the features and subtleties of the Sparrow system and the Outthentic development environment. I will give the thesis that else could be revealed. (If the reader has an interest in this, it will be possible in the following articles).

• Outthenitc API for Perl and Ruby
• Parameters for secondary stories
• Outthentic :: DSL is a plain text validator (test rule generators, asert, captcha, text blocks, etc.)
• Configuration of Sparrow plugins (Tasks, runtime parameters, YAML configuration files and Config :: General format)

PS I hope for constructive criticism :) and of course for attracting new plugin developers to the SparrowHub project.

Thank!