In this article, I would like to tell how you can create image building scripts for Docker containers using the Sparrow * multi-purpose scripts system .

(*) Note - to understand some of the technical nuances of this article, it is desirable to have at least a superficial acquaintance with the Sparrow multipurpose scripting system, a brief introduction to which (besides the documentation pages ) can be found in my previous article on habrahabr.ru.

Docker Container Development

Initially, a little problematic. We have the task to describe the Docker image assembly using the Dockerfile . If the build script is non-trivial and contains many instructions, you need to somehow get out. Besides the fact that the Dockerfile cannot contain more than 120 layers (as far as I correctly understood from the Docker documentation), it’s not very pleasant to deal with the spreading Dockerfile. What can you do about it? The obvious options are to put the build code into separate Bash scripts in the working directory and install and configure the system directly from them. Another way is to fasten a configuration management tool like chef or ansible to the side. I leave it at the mercy of the reader to evaluate these alternatives (IMHO they have their pros and cons) and suggest a third way - using Sparrow.

Before giving the details of the implementation I want to say:

• The variant with Sparrow is very similar to the use of Bash scripts, with the only difference that the entire installation logic is placed in the Sparrow plugin, with its source code stored in a separate place (git repository or central repository).

  
  

• Thus, the basic configuration of the system in the context of Docker is described in the Dockerfile, and the more subtle and complex inside the Sparrow plugin.

  
  
• Plug-in development can be separated from the Dockerfile context - which is convenient, for example, you can debug the installation process on an existing Docker container, using the docker build command to build the image each time, and once the plugin is debugged and ready for work, you can run full system build cycle using the same docker build (while dropping the docker cache, of course).

Implementation example

So, we will show everything on a specific system. It is required to build an image with the CentOS distribution and install an application written in Ruby version equal to 2.3. After that, run the main application script from under the selected user. The source code of the application is downloaded from a certain archive server. The example is taken from real life, although some details are intentionally omitted so as not to overload the article with material.

Basic system configuration

Before writing the plugin code, create a Dockerfile. I took tutum / centos for a basic image because of its lightness. For the same reason, some packages have to be delivered, but in general this is not a problem.

 $ cat Dockerfile FROM tutum/centos MAINTAINER "melezhik" <melezhik@gmail.com> RUN yum clean all RUN yum -y install nano git-core RUN yum -y install make RUN yum -y install gcc RUN yum -y install perl perl-devel \ perl-Test-Simple perl-Digest-SHA perl-Digest-MD5 perl-CPAN-Meta \ perl-CPAN-Meta-Requirements perl-Getopt-Long \ perl-JSON perl-Module-CoreList perl-Module-Metadata perl-parent perl-Path-Tiny perl-Try-Tiny \ perl-App-cpanminus perl-JSON-PP perl-Algorithm-Diff perl-Text-Diff \ perl-Spiffy perl-Test-Base perl-YAML perl-File-ShareDir-Install perl-Class-Inspector \ perl-File-ShareDir perl-File-ShareDir-Install perl-Config-General RUN cd /bin/ && curl -L https://cpanmin.us/ -o cpanm && chmod +x cpanm RUN cpanm Sparrow -q 

A few comments on the Dockerfile.

• nano and git-core needed to develop the Sparrow plugin (see below) - we will edit the script code and commit the changes to the remote git repository.

  
  

• gcc , make will be required to build RubyGems and CPAN packages. The first will be required when installing Ruby via rvm , the last to install Sparrow.

  
  

• Installing multiple perl-* packages through yum necessary to optimize the build process for speed, one would not have to do this, because The following cpanm -q Sparrow instruction would install the required dependencies itself, but installing dependencies via cpanm generally takes much longer than setting the “native” rpm- centOS for CentOS.

  
  
• The cpanm Sparrow -q instruction cpanm Sparrow -q a multi-purpose scripting development environment, do not forget that we are going to develop Sparrow directly on a running Docker container.

So, let's try to create an image:

 $ docker build -t ruby_app . ... ... Successfully built 25e7cd784c99 

We are starting to develop a plugin

Great, we have an image with the basic infrastructure, you can run the Docker container and start developing the plugin right on it.

 $ docker run -t -i ruby_app /bin/bash $ mkdir ruby-app $ cd ruby-app $ git init . $ git remote add origin https://github.com/melezhik/ruby-app.git $ touch README.md $ git add README.md $ git config --global user.email "melezhik@gmail.com" $ git config --global user.name "Alexey Melezhik" $ git commit -a -m 'first commit' $ git push -u origin master 

With the above commands, we created a project template for our plugin and committed everything to a remote git repository. We will remember the repository URL, we will need it later when we will do a full-fledged image docker build with the docker build

Now make a small digression. Recall our task. For convenience, let's try to break it up into independent parts:

• Creating an application user account
• Installing Ruby with rvm
• Download the application archive, unpacking and installing dependencies

For logically separate tasks, Sparrow provides a mechanism for the modules; we use it. But first of all we will create the main story in which we will delegate the execution of tasks to different modules. So, all on the same container launched by Docker:

 $ nano hook.bash action=$(config action) for s in $action do run_story $s done set_stdout install-ok $ nano story.check install-ok 

A few comments on the code. We have three minor histories (modules) and one main history, given by a hook file (hook.bash), in order to show how all this works, we will create stubs for scripts in modules. Yes, and the default value for the action input parameter must be specified in the suite.ini file.

 $ nano suite.ini action create-user install-ruby install-app 

Create script stubs:

 $ mkdir -p modules/create-user $ mkdir -p modules/install-ruby $ mkdir -p modules/install-app $ nano modules/create-user/story.bash echo create-user-ok $ nano modules/install-ruby/story.bash echo install-ruby-ok $ nano modules/install-app/story.bash echo install-app-ok 

As well as verification files:

 $ nano modules/create-user/story.check create-user-ok $ nano modules/install-ruby/story.check install-ruby-ok $ nano modules/install-app/story.check install-app-ok 

Now we’ll run everything through strun , a console script for running Sparrow scripts:

 $ strun /tmp/.outthentic/93/ruby-app/story.t .. # [/ruby-app/modules/create-user] # create-user-ok ok 1 - output match 'create-user-ok' # [/ruby-app/modules/install-ruby] # install-ruby-ok ok 2 - output match 'install-ruby-ok' # [/ruby-app/modules/install-app] # install-app-ok ok 3 - output match 'install-app-ok' # [/ruby-app] # install-ok ok 4 - output match 'install-ok' 1..4 ok All tests successful. Files=1, Tests=4, 0 wallclock secs ( 0.00 usr 0.02 sys + 0.09 cusr 0.01 csys = 0.12 CPU) Result: PASS 

Fine. We see that all the scripts have worked successfully, this will be the skeleton of our future plug-in. It remains only to tighten the plugs of our modules.

User creation script

Let us assume that the username is customizable, we define the default value in the suite.ini file:

 $ cat suite.ini action create-user install-ruby install-app user_name app-user 

Now the script implementation:

 $ nano modules/create-user/story.bash user_id=$(config user_name) echo create user id: $user_id useradd -r -m -d /home/$user_id $user_id || exit 1 ls -d /home/$user_id || exit 1 id $user_id || exit 1 echo create-user-ok 

And start (note that here we used the opportunity to run a separate script using the action parameter):

 $ strun --param action=create-user /tmp/.outthentic/135/ruby-app/story.t .. # [/ruby-app/modules/create-user] # create user id: app-user # /home/app-user # uid=997(app-user) gid=995(app-user) groups=995(app-user) # create-user-ok ok 1 - output match 'create-user-ok' # [/ruby-app] # install-ok ok 2 - output match 'install-ok' 1..2 ok All tests successful. Files=1, Tests=2, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.11 cusr 0.04 csys = 0.18 CPU) Result: PASS 

We see that the script worked and the user was created. Note that most of the Bash commands inside the script end with the idiomatic construction cmd || exit 1 cmd || exit 1 , strun checks the script execution code and if it fails, the corresponding test fails, for example like this - try to create a user with an invalid name for the system:

 $ strun --param action=create-user --param user_name='/' /tmp/.outthentic/160/ruby-app/story.t .. # [/ruby-app/modules/create-user] # create user id: / # useradd: invalid user name '/' not ok 1 - scenario succeeded not ok 2 - output match 'create-user-ok' # [/ruby-app] # install-ok ok 3 - output match 'install-ok' 1..3 # Failed test 'scenario succeeded' # at /usr/local/share/perl5/Outthentic.pm line 167. # Failed test 'output match 'create-user-ok'' # at /usr/local/share/perl5/Outthentic.pm line 213. # Looks like you failed 2 tests of 3. Dubious, test returned 2 (wstat 512, 0x200) Failed 2/3 subtests Test Summary Report ------------------- /tmp/.outthentic/160/ruby-app/story.t (Wstat: 512 Tests: 3 Failed: 2) Failed tests: 1-2 Non-zero exit status: 2 Files=1, Tests=3, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.10 cusr 0.00 csys = 0.12 CPU) Result: FAIL 

I will make a small digression here. Let us ask ourselves why we need verification files if, in essence, verification of the script's completion code should be enough. Reasonable question. We can think of the verification rules of the Sparrow framework as an alternative way to monitor or verify the execution of our scripts. In the ideology of Sparrow, any executed script is history in the sense that it is a kind of script that runs and most often "reports" something about its work - figuratively speaking, "leaving a mark on history." This trace is the standard stdout output, the contents of which can be validated. Why this may be useful:

• Not always successful completion code means everything goes well

  
  
• Sometimes you want to leave the script in an emergency (using cmd || exit 1 ), allowing the script to do its work to the end and postpone verification by checking through the verification file.

As a concrete example, we can give the Ruby installation script via rvm, which is next in the list in our plan.

Ruby install script from rvm

Here is what the installation script will look like:

 $ nano modules/install-ruby/story.bash yum -y install which curl -sSL https://rvm.io/mpapis.asc | gpg2 --import - || exit 1 \curl -sSL https://get.rvm.io | bash -s stable --ruby || exit 1 source /usr/local/rvm/scripts/rvm gem install bundler --no-ri --no-rdoc echo ruby version: $(ruby --version) bundler --version echo install-ruby-ok 

And this is the verification file:

 $ nano modules/install-ruby/story.check regexp: ruby version: ruby 2\.3 install-ruby-ok 

Now run this script:

 $ strun --param action=install-ruby #    #   # ... # ... # ... # ruby version: ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux] # Bundler version 1.12.5 # install-ruby-ok ok 1 - output match /ruby version: ruby 2\.3/ ok 2 - output match 'install-ruby-ok' # [/ruby-app] # install-ok ok 3 - output match 'install-ok' 1..3 ok All tests successful. Files=1, Tests=3, 91 wallclock secs ( 0.03 usr 0.00 sys + 3.24 cusr 1.03 csys = 4.30 CPU) Result: PASS 

I would like to note that in order to verify the version of the installed Ruby, we used the checkout rule in the form of a regular expression:

 regexp: ruby version: ruby 2\.3 

Of course, rvm allows you to install the required version explicitly, just wanted to give an example here when the checks defined in the test files allow you to add additional verification of the script with minimal effort.

Now you can go to the application installation script.

Application Installation Scenario

I will remind you. We will need:

• download tarball for a given urla
• unpack archive
• go to the unpacked folder and execute the command bundle install --target ./local to install dependencies

That's all. Of course, in a real application, we would have to start some service or perform some additional operations, but this should be enough to demonstrate the work of the plugin.

Again, for the sake of simplicity, let us have a Ruby application consisting of:

• Gemfile - in which dependencies will be written
• hello.rb - a launch of a squeak that simply prints the Hello World line to the console

We pack all the archive and upload everything to the archive server on the local nginx, now the distribution will be available at the URL:

 127.0.0.1/app.tar.gz 

Update script code.

 $ cat suite.ini action create-user install-ruby install-app user_name app-user source_url 127.0.0.1/app.tar.gz $ cat modules/install-app/story.bash user_id=$(config user_name) source_url=$(config source_url) yum -y -q install sudo echo downloading $source_url ... sudo -u $user_id -E bash --login -c "curl -f -o ~/app.tar.gz $source_url -s" || exit 1 echo unpacking tarball ... sudo -u $user_id -E bash --login -c "cd ~/ && tar -xzf app.tar.gz" || exit 1 echo installing dependencies via bundler sudo -u $user_id -E bash --login -c "cd ~/app && bundle install --quiet --path vendor/bundle " || exit 1 sudo -u $user_id -E bash --login -c "cd ~/app && bundle exec ruby hello.rb " || exit 1 echo install-app-ok $ nano modules/install-app/story.check install-app-ok Hello World 

Small comments on the script:

• Installation is done from under the user specified in the configuration of the plugin suite.ini. For this we need the sudo package

  
  

• The last command runs the hello.rb application script hello.rb

  
  
• In the test file, we require that the stdout see the "trace" of the script - the line 'Hello World'

So, run the script:

 $ strun --param action=install-app /tmp/.outthentic/16462/ruby-app/story.t .. # [/ruby-app/modules/install-app] # Package sudo-1.8.6p7-17.el7_2.x86_64 already installed and latest version # downloading 127.0.0.1/app.tar.gz ... # unpacking app ... # installing dependencies via bundler # Hello World # install-app-ok ok 1 - output match 'install-app-ok' ok 2 - output match 'Hello World' # [/ruby-app] # install-ok ok 3 - output match 'install-ok' 1..3 ok All tests successful. Files=1, Tests=3, 2 wallclock secs ( 0.01 usr 0.00 sys + 1.61 cusr 0.50 csys = 2.12 CPU) Result: PASS 

As we see, the application is really installed and the hello.rb script hello.rb launched. Add another “paranoid” assort to the verification file to demonstrate the capabilities of the Sparrow verification system:

 $ nano modules/install-app/story.check install-app-ok Hello World generator: <<CODE !bash if test -d /home/$(config user_name)/app; then echo assert: 1 directory /home/$(config user_name)/app exists else echo assert: 0 directory /home/$(config user_name)/app exists fi CODE 

And run the script again.

 $ strun --param action=install-app 

In the output we get:

 $ ok 3 - directory /home/app-user/app exists 

Publishing Sparrow Plugin

This completes the creation of the plugin. Let's finish the changes and push the git repository:

 $ git add . $ git commit -a -m 'all done' $ git push $ exit 

We left the docker container, we no longer need it, you can delete it:

 $ docker rm 5e1037fa4aef 

Full Image Build Cycle for Docker Container

It remains to slightly change the Dockerfile, remembering that we will need a link to the remote git repository, where we placed the code of our Sparrow plugin, the final version will be like this:

 FROM tutum/centos MAINTAINER "melezhik" <melezhik@gmail.com> RUN yum clean all RUN yum -y install nano git-core RUN yum -y install make RUN yum -y install gcc RUN yum -y install perl perl-devel \ perl-Test-Simple perl-Digest-SHA perl-Digest-MD5 perl-CPAN-Meta \ perl-CPAN-Meta-Requirements perl-Getopt-Long \ perl-JSON perl-Module-CoreList perl-Module-Metadata perl-parent perl-Path-Tiny perl-Try-Tiny \ perl-App-cpanminus perl-JSON-PP perl-Algorithm-Diff perl-Text-Diff \ perl-Spiffy perl-Test-Base perl-YAML perl-File-ShareDir-Install perl-Class-Inspector \ perl-File-ShareDir perl-File-ShareDir-Install perl-Config-General RUN cd /bin/ && curl -L https://cpanmin.us/ -o cpanm && chmod +x cpanm RUN cpanm Sparrow -q RUN echo ruby-app https://github.com/melezhik/ruby-app.git > /root/sparrow.list RUN sparrow plg install ruby-app RUN sparrow plg run ruby-app 

Now we can carry out a complete image building cycle, “losing” all over again:

 $ docker build -t ruby_app --no-cache=true . 

As a result, we get a Docker image with the required system.

Conclusion

The use of Sparrow multipurpose scripting system can be an effective tool for building Docker images, since allows you to build complex configurations, leaving the main Dockerfile simple and concise, as well as simplifying the process of developing the configuration scripts themselves required system.

Thanks for attention.

As usual, I am waiting for questions and constructive criticism! :)

Alexey