Hello. During this year, I wrote a series of articles on Sparrow , a user script management system. At the end of the year, I would like to make a kind of resultant post, where I will once again try to describe the main essence of this tool.

So what is Sparrow? If you do not go into various options for using this system, Sparrow allows you to quickly develop, customize and run custom scripts.

I will give a specific example. In my work, I often have to go to many servers and do something there called hands. At the same time, I would like the environment for my account to be, if not the same, then very similar to what I am dealing with on my own machine. That's what I mean:

• list of installed packages that I often need to work with - for example tree, curl, nano, git, mc
• specific setting for nano editor
• specific setting for the git client

And so on. The list of this kind of little things everyone can have their own. All this, of course, can be configured and installed manually, as they say, not "rocket science", but first, you need to enter a bunch of commands each time and go into the documentation to properly set up the software. Secondly, you can trite something to forget or miss, because it's hard to keep everything in your head. All this takes time and effort. So, I’m not discovering America here for anyone, I need the usual automation.

Okay. Immediately I foresee here comments of the form, and why not use some chef or ansible for such tasks. On this I have a number of thoughts, here they are:

• Chef or Ansible are good for solving industrial-scale problems when you have a lot of servers and a typical, well-organized infrastructure that is easy to express through (cook / play) beeches.
• If you have custom, personal scripts that, in fact, suit you and you do not want to turn them into universal solutions, then solutions like chef or ansible may already be a sort of brute force.
• Also, we should not forget that the same chef in its standard workflow requires a server part and a correctly configured client chef (node ​​registration) - you will already agree a bit too much to solve a task like install the nano editor or configure the client git config.
• And finally, sometimes you already have a ready-made and bash script that suits you, which you just want to run on the target server, while not wanting to turn it into a chef cookbook or ansible playbook and perform additional "gestures" related to porting your code ( which, by the way, is not always smooth for ordinary bash / shell scripts - but this is a topic for a separate post in the ecosystem of these tools.
• All this suggests that you need a simple, but at the same time sufficient flexible solution that allows (to develop and) distribute user scripts to different machines. This is how the idea of ​​Sparrow and the SparrowHub user script repository appeared . I wrote about it in detail * including on Habré.

(*) The post is slightly outdated relative to the current implementation of Sparrow, but the main point is reflected there.

So, if you summarize the main features of Sparrow, you can say the following:

• Sparrow is an orchestrator of user scripts that allows you to develop, distribute and manage a variety of scripts, packaged and customized in a unified way.

  
  

• Thus, the scripts are loaded into a single repository, have the author, version and documentation, all this is similar to any package distribution system, such as apt / debian or CPAN / Perl.

  
  

• Sparrow provides a console client allowing you to search (repository index) and install and run scripts.

  
  

• Installing scripts can be on the latest or specified version. In general, versioning of scripts encourages their use in the team, when the developer can make the next changes to the script, putting it with the new version in the repository, while users in case of backward compatibility can always "roll back" to previous versions of the script. It also simplifies the process of tracking changes in the script, when, knowing the version, you can always see in the changelog file what has been updated recently.

  
  

• In the case of private scripts, when you would not want to share them in public access (central repository mode), Sparrow allows you to install scripts hosted on remote git repositories (so-called private sparrow plugins ).

  
  

• Sparrow to some extent - language agnostic system. Scripts uploaded to the SparrowHub repository can be written in one of three languages:

  
  
  • Perl
  • Ruby
  • Bash
  
  

Sparrow API is implemented for any of these languages. It allows a convenient way to customize scripts and organize multiscended programs (what is called modules in Sparrow), regardless of the scripting language, you get quite a powerful DSL for verifying the output stream (stdout) of the scripts being run, makes Sparrow very attractive for writing scripts testing, monitoring and auditing. (As one of the examples, see the project - minoca-pkg-test - testing assembled packages in the Minoca OS operating system). You can read about all this in detail in the documentation for the Outthentic module that implements the Sparrow scripts development environment.

• Sparrow itself is written in Perl and is installed as a CPAN module. It has a very small overhead dependency, which makes it quite easy to install.

These are the main, but not all characteristics of the Sparrow ecosystem, further, so that the post would not turn out to be too theoretical, I will give typical options for using this tool, it’s certainly not even an introduction, but it will help to grasp the essence of how Sparrow works.

Sparrow installation

We put as a normal CPAN module. We also need curl. And that is all.

$ cpanm --notest -q Sparrow $ yum install curl 

Pick up the SparrowHub repository index update.

What you usually do when executing apt-get update in Sparrow will look like this:

 $ sparrow index update 

Search scripts

Now, by updating the index, you can search for scripts that are called plugins in Sparrow. For example, I need everything related to the nano editor:

 $ sparrow plg search nano 

You can use regular expression search:

 $ sparrow plg search ssh.* 

Installing plugins

We put the plugin like this:

 $ sparrow plg install nano-setup 

By the way, if the script has dependencies, they will also be installed. Sparrow supports declaring and installing dependencies via cpanfile for Perl and Gemfile for Ruby.

As already mentioned, you can install the plugin for the required version:

 $ sparrow plg install nano-setup --version 0.1.2 

Running plugin

By installing the plugin, we can run it. Here, for example, how I configure the nano-rc file:

 $ sparrow plg run nano-setup 

Depending on the plugin, we can pass on various parameters to it:

 $ sparrow plg run nano-setup --param tabsize=2 --param speller='hunspell -x -c' 

Or, if we plan to run the plugin more than once, and there are many parameters, we can create a task in which we define the parameters of the plugin to be launched, for example, you can create a task for finding errors like 500 in the nginx log using the logdog plugin:

 $ sparrow plg install logdog #  -      $ sparrow project create nginx #         $ sparrow task add nginx 500-errors logdog #    : $ sparrow task ini nginx/500-errors <logdog> file /var/log/nginx/access.log time_pattern \[(\d+\/\S+\/\d+):(\S+) time_format %d/%b/%Y %T history 5 minutes timezone Europe/Moscow filter HTTP\/\S+?"\s+500\s key_field (\S+) density 3 check_mode report </logdog> 

Now we can start the task with the specified parameters:

 $ sparrow task run nginx/500-errors 

Or override some parameters at startup:

 $ sparrow task run nginx/500-errors --param logdog.density=10 --param logdog.history="'1 weeks'" 

If we forget how to use the plugin, you can always get its documentation:

 $ sparrow plg man nano-setup 

Or use the SparrowHub web interface by finding the required plugin:

https://sparrowhub.org/info/nano-setup

Publishing tasks

Sometimes it is convenient to save your own task in order to use it later on another server. I will give a simple example. I have a list of packages that I often use during my work. We can say that they are likely to be needed when working with a new server. Ok, let's use the package-generic plugin to install packages for different distributions.

First, install the plugin and create the task:

 $ sparrow plg install package-generic $ sparrow project create utils $ sparrow task add utils my-packages package-generic $ sparrow task ini utils/my-packages list tree mc nano hunspell git 

Now let's save the task on the SparrowHub server in my account:

  $ sparrow remote task upload utils/my-packages 'my useful packages' 

Now, having come to another server, without suffering and not remembering the list of packages I need, I simply install them from the task:

  $ ssh new-server $ sparrow remote task run utils/my-packages 

You can get a list of my remote tasks like this:

  $ sparrow remote task list 

You can also make my remote task public so that then any other person can use it:

 $ sparrow remote task share utils/my-packages 

Now users can run my task:

 $ sparrow remote task run melezhik@utils/my-packages 

To get a list of all available remote tasks on SparrowHub, you can:

 $ sparrow remote task public list 

Development and publication of your own plug-ins is a topic for a separate topic, for those who are interested can see my post on Habré about it.

(*) the publication is slightly outdated relative to the current version of Sparrow, but it can be useful as a starting point, then you can refer to the documentation on the Sparrow development environment for plugins - Outthentic or in the case of writing test scripts for web applications - the swat system.

This is probably all, although of course it was a cursory excursion into the system, you can read more about Sparrow ’s various features on the documentation pages.

Summary

So, how can Sparrow be useful? I will try to summarize:

• Quickly save your own scripts for the purpose of re-using them by you or others on different servers.
• A useful out-of-the-box API that allows you to customize your scripts with a large set of supported configuration formats — INI style, Config :: General, YAML, JSON, command line. Support for overriding individual configuration settings (while preserving default values).
• Built-in DSL to verify the output of scripts allows you to quickly and easily write various scripts for monitoring, testing and auditing, using the BlackBox testing model.

PS: Congratulations to everyone! Successes!

And at the end of the traditionally small questionnaire with a similar theme.