Hello! Much has been written about the development of the chef kukbuk and its associated infrastructure, and there are already plenty of tools in this area. Among them are such solutions as vagrant , test kitchen , food critic , chef spec , minitest-chef-handler , serverspec , inspec . All of them, in varying degrees, simplify and accelerate the industrial development and testing of chef books and their customized infrastructure.

If this area is close to you and you also have something to do with the Perl language (more precisely, with Perl6 ), then welcome to the topic.

So today I’ll tell you how I use Sparrowdo when developing and testing chef cookbooks.

First, a little introductory.

What is Sparrowdo?

Sparrowdo is an add-on to the sparrow plugin system that allows you to run these ssh plugins on remote servers. In turn, sparrow plugins are various automation scripts written mostly (but not limited to these languages) in Perl5 and Bash . I have already written a whole series of articles on habrahabr about all of this; anyone who is interested can type in a search query with the word sparrow and, accordingly, read. But, so that for the uninitiated reader it would be clear, all this is very similar to the ideology of ansible , when you have a centralized server from which you run different tasks on other remote servers.

So, you can run various sparrow plugins after or before running the chef client on the server, and perform various checks that determine that the deployment procedure is successful or satisfies certain conditions.

All this is very similar to writing scripts in the stylespec or inspec with the only difference that sparrow plugins are multi-purpose and are not limited to testing and auditing tasks. In principle, you can implement any other logic that you cannot use for any reason. would like to stuff kukbuk in chef.

In fact, sparrowdo so conveniently "fit" in the task of automatically setting up servers that I even run the client chef itself ... But, enough theory, let's consider a specific simple but illustrative example ...

Writing a kookbook to install nginx + tomcat and test it with sparrowdo

For the sake of simplicity, I will use ec2 instance with the chef client preinstalled and configured on the local chef server. Those. all questions related to the initial configuration of the system will be left out of the box. Next we have an ssh key to our instance and a login with sudo:

$ ssh remote.server 

Now let's step aside from the issues of access to the server and write our symbolic kukbook, we need:

• install nginx server
• install tomcat server
• if the tomcat_redirect attribute is set, configure proxying all requests that come to nginx to the tomcat server running on localhost port 8080

We write kukbuk

The code is trivial, again the focus of this article is not on learning to write chef cookbooks, but on using chef in conjunction with sparrowdo / sparrow

 # recipes/default.rb package 'nginx' package 'tomcat7' service 'nginx' do action [ :enable, :start ] end service 'tomcat7' do action [ :enable, :start ] end template '/etc/nginx/conf.d/default.conf' do source 'nginx.conf' mode '644' variables :tomcat_redirect => node[:tomcat_redirect] notifies :restart, 'service[nginx]', :delayed end # attributes/default.rb default[:tomcat_redirect] = false # cat templates/default/nginx.conf server { listen 80 default_server; server_name _; root /usr/share/nginx/html; index index.html index.htm; <% if @tomcat_redirect %> location / { proxy_pass http://127.0.0.1:8080; } <% else %> location / { try_files $uri $uri/ $uri.html =404; } <% end %> error_page 404 /404.html; location = /404.html { } # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { } } 

I think that comments on the code for the kookbook are superfluous, I will only note that by default nginx does not perform proxying on tomcat, since the tomcat_redirect attribute tomcat_redirect set to false. And finally, let's call the nginx-app kukbook and upload it to the chef server:

 # cat metadata.rb name 'app-nginx' version '0.0.1' $ knife cookbook upload app-nginx -o ../ 

Now, actually run the created kukbook on our server.

Writing a sparrowdo script

Sparrowdo script - code written in Perl6 and running tasks on a remote server. Tasks are nothing more than sparrow plugins with parameters. As I said, we will use sparrow, including to start the chef client

 $ cat sparrowfile task_run %( task => "set up chef run list an attributes", plugin => "file", parameters => %( target => "/tmp/chef.json", content => q:to/JSON/, { "run_list" : [ "recipe[nginx-app]", ], "tomcat_redirect" : false } JSON ), ); task_run %( task => "run chef-client", plugin => "bash", parameters => %( command => "chef-client --color --json /tmp/chef.json" ), ); 

A small explanation of the code.

• The script file should be called a sparrowfile and should be sparrowfile in the same directory where you will run the client's sparrowdo, it makes sense to keep it in the kukbook directory - clearly and always at hand.

  
  
• In this example, two sparrow plugins are used: file — to create files and bash — to run arbitrary commands on Bash, as already mentioned, they are launched with parameters using the task_run function, and the task parameter in calls to this function adds an arbitrary text description of the task to be performed.

Now you can run the script. Since On the given server, the sparrow client is not yet installed, we start with the --bootstrap option, asking sparrowdo to install the client first, which will perform all tasks:

 $ sparrowdo --http_proxy=$http_proxy --https_proxy=$https_proxy --host=remote.server --ssh_user=root --no_sudo --no_index_update running sparrow tasks on remote.server ... target OS is - centos6 push task <set up chef run list an attributes> plg <file> OK push task <run chef-client> plg <bash> OK set up task box file - /tmp/sparrowdo-box.json - OK public@file is uptodate (0.0.2) public@bash is uptodate (0.1.1) running task box from /tmp/sparrowdo-box.json ... <set-up-chef-run-list-an-attributes> / started set target content target created set target mode to 644 ok scenario succeeded ok output match /target (created|removed)/ ok output match 'set target content' STATUS SUCCEED <run-chef-client> / started [2016-10-19T10:02:48+00:00] INFO: Forking chef instance to converge... [2016-10-19T10:02:48+00:00] INFO: *** Chef 11.16.4 *** [2016-10-19T10:02:48+00:00] INFO: Chef-client pid: 5389 [2016-10-19T10:02:49+00:00] INFO: Setting the run_list to ["recipe[nginx-app]"] from CLI options [2016-10-19T10:02:49+00:00] INFO: Run List is [recipe[nginx-app]] [2016-10-19T10:02:49+00:00] INFO: Run List expands to [nginx-app] [2016-10-19T10:02:49+00:00] INFO: Starting Chef Run for sandbox-generic-ci-i-ef7cbedf [2016-10-19T10:02:49+00:00] INFO: Running start handlers [2016-10-19T10:02:49+00:00] INFO: Start handlers complete. [2016-10-19T10:02:49+00:00] INFO: HTTP Request Returned 404 Not Found: [2016-10-19T10:02:50+00:00] INFO: Loading cookbooks [nginx-app@0.0.1] [2016-10-19T10:02:50+00:00] INFO: Processing package[nginx] action install (nginx-app::default line 1) [2016-10-19T10:02:53+00:00] INFO: Processing package[tomcat7] action install (nginx-app::default line 3) [2016-10-19T10:02:53+00:00] INFO: Processing service[nginx] action enable (nginx-app::default line 5) [2016-10-19T10:02:53+00:00] INFO: Processing service[nginx] action start (nginx-app::default line 5) [2016-10-19T10:02:53+00:00] INFO: Processing service[tomcat7] action enable (nginx-app::default line 9) [2016-10-19T10:02:53+00:00] INFO: Processing service[tomcat7] action start (nginx-app::default line 9) [2016-10-19T10:02:53+00:00] INFO: Processing template[/etc/nginx/conf.d/default.conf] action create (nginx-app::default line 13) [2016-10-19T10:02:53+00:00] INFO: Chef Run complete in 3.604494597 seconds [2016-10-19T10:02:53+00:00] INFO: Running report handlers [2016-10-19T10:02:53+00:00] INFO: Report handlers complete bash-command-done ok scenario succeeded ok output match 'bash-command-done' STATUS SUCCEED 

The first launch of the script worked out well, we see that the chef client installed the required packages and configured them, following the logic set in the kukbook.

We will slightly change the sparrowdo script, minimizing the output from the client’s chef during the following launches:

 task_run %( task => "run chef-client", plugin => "bash", parameters => %( command => "chef-client --color --json /tmp/chef.json -l error" ), ); 

Now it's time to add a couple of checks that our deployment has ended in success, namely:

• nginx service is visible in the process list
• tomcat service is visible in the process list

This may seem redundant, since chef guarantees us the processing / output of all errors associated with the launch of services, however experience shows that this is not always enough , for example, the service may fall a little later, after the chef client has successfully completed. And then it never hurts to write a couple more "parallel" tests (which by the way can be run outside the context of the client chef), especially since with sparrowdo this is very easy to do. Add a couple of tasks to our sparrowdo script:

 task_run %( task => 'check nginx process', plugin => 'proc-validate', parameters => %( pid_file => '/var/run/nginx.pid', footprint => 'nginx.*master' ) ); task_run %( task => 'check tomcat7 process', plugin => 'proc-validate', parameters => %( pid_file => '/var/run/tomcat7.pid', footprint => 'java' ) ); 

Here we used the proc-validate plugin, which checks the existence of a process by the specified PID file, and also searches for it by following the “trace” in the list of running processes:

  $ sparrowdo --http_proxy=$http_proxy --https_proxy=$https_proxy --host=remote.server --ssh_user=root --no_sudo --no_index_update running sparrow tasks on remote.server ... target OS is - centos6 push task <set up chef run list an attributes> plg <file> OK push task <run chef-client> plg <bash> OK push task <check nginx process> plg <proc-validate> OK push task <check tomcat7 process> plg <proc-validate> OK set up task box file - /tmp/sparrowdo-box.json - OK public@file is uptodate (0.0.2) public@bash is uptodate (0.1.1) public@proc-validate is uptodate (0.0.5) running task box from /tmp/sparrowdo-box.json ... <set-up-chef-run-list-an-attributes> / started set target content target created set target mode to 644 ok scenario succeeded ok output match /target (created|removed)/ ok output match 'set target content' STATUS SUCCEED <run-chef-client> / started bash-command-done ok scenario succeeded ok output match 'bash-command-done' STATUS SUCCEED <check-nginx-process> / started pid_file - /var/run/nginx.pid pid file exists pid:4526 process footprint: 4526 nginx: master process /usr/ 31:47 ok scenario succeeded ok output match /pid_file - \S+/ ok output match 'pid file exists' ok output match /pid:\d+/ ok output match /process footprint:/ ok 'process footprint: 4526 nginx: master process /usr/ 31:47' match /nginx.*master/ STATUS SUCCEED <check-tomcat7-process> / started pid_file - /var/run/tomcat7.pid pid file exists pid:29458 process footprint: 29458 /usr/lib/jvm/java-1.6.0/bin 22:11:25 ok scenario succeeded ok output match /pid_file - \S+/ ok output match 'pid file exists' ok output match /pid:\d+/ ok output match /process footprint:/ ok 'process footprint: 29458 /usr/lib/jvm/java-1.6.0/bin 22:11:25' match /java/ STATUS SUCCEED 

Fine. We see that our services are really running. Another advantage of this approach is that you can run such checks on an already closed server without running the client chef itself.

OK, go ahead. Check the availability of our services on http . To do this, use the curl utility, this is what you need to add to our sparrowdo script:

 task_run %( task => 'install curl', plugin => 'package-generic', parameters => %( list => 'curl', ) ); task_run %( task => 'check nginx http port', plugin => 'bash', parameters => %( command => 'sleep 5; curl --retry 2 --noproxy 127.0.0.1 -f -o /dev/null -D - http://127.0.0.1', ) ); task_run %( task => 'check tomcat http port', plugin => 'bash', parameters => %( command => 'sleep 5; curl --retry 2 --noproxy 127.0.0.1 -f -o /dev/null -D - http://127.0.0.1:8080', ) ); 

...

Run the script:

$ sparrowdo --http_proxy=$http_proxy --https_proxy=$https_proxy --host=remote.server --ssh_user=root --no_sudo --no_index_update

 #    ... <install-curl> / started package installer /modules/yum/ started trying to install curl ... installer - yum Installed Packages curl.x86_64 7.19.7-37.el6_4 @base/$releasever ok scenario succeeded ok output match 'Installed' STATUS SUCCEED <check-nginx-http-port> / started % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 102 3698 102 3698 0 0 3218k 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Server: nginx/1.0.15 Date: Wed, 19 Oct 2016 11:30:51 GMT Content-Type: text/html Content-Length: 3698 Last-Modified: Fri, 26 Apr 2013 20:36:51 GMT Connection: keep-alive Accept-Ranges: bytes bash-command-done ok scenario succeeded ok output match 'bash-command-done' STATUS SUCCEED <check-tomcat-http-port> / started % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11197 0 11197 0 0 1195k 0 --:--:-- --:--:-- --:--:-- 1366k HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: text/html;charset=ISO-8859-1 Transfer-Encoding: chunked Date: Wed, 19 Oct 2016 11:30:56 GMT bash-command-done ok scenario succeeded ok output match 'bash-command-done' STATUS SUCCEED 

As we see, our services are available via http. I note, we previously installed the curl utility, using the package-generic plugin to install packages.

Well, finally, we set the tomcat_redirect attribute to true, forcing nginx to proxy all requests to tomcat and verify that this happens.

Set attribute:

 task_run %( task => "set up chef run list an attributes", plugin => "file", parameters => %( target => "/tmp/chef.json", content => q:to/JSON/, { "run_list" : [ "recipe[nginx-app]", ], "tomcat_redirect" : true } JSON ), ); 

Add the expect_stdout parameter to the bash plugin launch, which will force it to check the STDOUT of the command being executed by the specified regex:

 task_run %( task => 'check nginx http port', plugin => 'bash', parameters => %( command => 'sleep 5; curl --retry 2 --noproxy 127.0.0.1 -f http://127.0.0.1 | head -n 10', expect_stdout => 'Apache Tomcat' ) ); 

Run the script:

 $ sparrowdo --http_proxy=$http_proxy --https_proxy=$https_proxy --host=remote.server --ssh_user=root --no_sudo --no_index_update #    ... <check-nginx-http-port> / started <!DOCTYPE html> <html lang="en"> <head> <title>Apache Tomcat/7.0.57</title> <link href="favicon.ico" rel="icon" type="image/x-icon" /> <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" /> <link href="tomcat.css" rel="stylesheet" type="text/css" /> % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (23) Failed writing body (146 != 3151) bash-command-done ok scenario succeeded ok output match 'bash-command-done' ok output match /Apache Tomcat/ STATUS SUCCEED <check-tomcat-http-port> / started % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11197 0 11197 0 0 3364k 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: text/html;charset=ISO-8859-1 Transfer-Encoding: chunked Date: Wed, 19 Oct 2016 12:03:02 GMT bash-command-done ok scenario succeeded ok output match 'bash-command-done' STATUS SUCCEED 

Conclusion

In this simple example, we learned how to use sparrowdo / sparrow in the development and testing processes of chef cookbooks, namely:

• run the client chef with the specified parameters and attributes on the specified hosts
• check the existence of processes
• do the simplest checks on the availability of web services
• install the tools needed for testing.

In fact, you could write a lot more. Cases for using sparrowdo and chef are really a large number. Just take ready-made plugins here or write your own, because it’s really not difficult and start using sparrowdo in your development.

The whole feature of the sparrow system, which favorably distinguishes it from other similar tools, such as serverspec / inspec, is that the functionality is not hard-wired "in the kernel", but the user can easily and quickly write his plug-ins that fit his realities and immediately use them using sparrowdo, you can even "compile" more complex scripts based on sparrow plugins using the so-called sparrowdo modules , which are written in Perl6 (you can write a separate article about this, if you are interested).

If this article has caused your interest - write, comment, ask questions and ... constructively criticize :)) I would be grateful. As usual, the single center for documentation and the plugin repository of sparrow is https://sparrowhub.org

Thank.

PS At the end of the article, as usual, a simple question, indirectly related to the topic of the article.