Testing and continuous integration for Ansible roles with Molecule and Jenkins

After Ansible entered our practice, the amount of code on it and, in particular, the roles began to grow very quickly. Roles for backups, fronts, proxies, databases, monitoring, logging, etc., etc. - their number totals dozens. Some of the roles are specific to a particular project, but many solve common problems; they want to be divided between project teams in order not to create the same solution twice.

Together with a large amount of code, an old, familiar problem appears: the fear of change. People do not want to make changes to the "alien" role, fearing to spoil it, instead they create their own copy. Code refactoring is not performed if the code right now is not in the focus of development, due to the fear that the introduced problems may be discovered after too much time. The bottom line: bad code grows like a snowball.

A familiar problem has a familiar solution: automated testing and CI. But Ansible is too specific a technology. Is it possible to do anything other than manually rolling a role on a virtual machine and checking that nothing “has fallen off” to check the Ansible code? I thought it was impossible, until I learned about the existence of the project Molecule .
')
When using Molecule, all problems with the "fragility" of Ansible code are completely eliminated. If you are an adherent of test-driven development, then with the help of Molecule you can lead the development of Ansible in the cycle “wrote test - test failed - wrote code - test passed”. If not, then even simply “screwing Molecule” to the existing code and without writing a single test, you already get a system for checking your role for compliance with coding standards, for triggering and idempotency.

The first question I had when I found out about Molecule is where does he expand his role in the testing process? The answer is - options are possible. The easiest is that if Docker is available on the Molecule machine, the roles are deployed in Docker containers that are thrown out after the test run. But you can use other drivers in Molecule, such as Vagrant, Azure, EC2, GCE, etc.

Now - about everything in order.

Installation

Like Ansible, Molecule is written in python and set by the command

pip install molecule 

There are nuances: as of March 2018, I needed to manually update the cryptography and pyopenssl packages using pip, otherwise there were problems with the launch and operation of Molecule. If you want to use the Docker-driver (which I recommend), then you also need to install Docker and the docker-py pip package.

As a development machine, I use Windows and Cygwin, on which Ansible works fine for me. Unfortunately, I did not manage to make friends with Cygwin, Windows Docker and Molecule, so I work with Molecule only in Linux.

Project Initialization

If your role is called oldrole, then to add Molecule to it, you need to run the command

 molecule init scenario -r oldrole 

To create a completely new role you need to perform

 molecule init role -r newrole 

These commands create folders and files, all the most interesting - in the folder. In fact, I seem to have used this command only once. Then I copied the file structure to other roles manually to transfer my individual settings and tests.

Go?

After initialization, execute the command in the project root.

 molecule test 

From the first time it doesn’t take off, of course, but you will immediately see what Molecule is trying to do with your role, and where problems arise. The output to the console will contain the planned execution script:

 --> Test matrix └── default ├── lint ├── destroy ├── dependency ├── syntax ├── create ├── prepare ├── converge ├── idempotence ├── side_effect ├── verify └── destroy 

As you can see, Molecule offers a ready-made template script for comprehensive verification of your role, ranging from static code analysis to infrastructure tests based on the results of its execution. Of course, with the help of the settings you can change this scenario or perform only individual steps.

If the result of the execution is "crashed", and the information available in the console is not enough, try

 molecule --debug test 

- this will make the output much more detailed.

Sometimes in the process of debugging tests, you will want to log in to the test environment to see what went wrong. For this it is useful to run the command

 molecule test --destroy=never 

- so the test environment will not be deleted, and if, say, you are working with Docker, then you can log in as usual:

 docker exec -it instance /bin/bash 

Static analysis

“Ay-yay-yay, you forgot to put three minuses in the beginning of the YAML-file! And here you have spaces in the empty line! ”There will be the most such messages in the first step. Fortunately, many of them are warning and do not stop the entire check. But there are more substantial clues. For example, at this stage, the system may indicate that you incorrectly use the shell module when you could do with the command module, or use the wget command when you could use the special get_url module. In addition, if you have tests (and you need to write them on python), the system runs the flake8 static analyzer over the python code.

Launch role

If at the stage of static analysis there were no criticisms, the system creates a node and proceeds to launch the role using Ansible itself. How it will do this is defined in the file molecule/default/molecule.yml . In my case, it looks like this:

 --- dependency: name: galaxy options: role-file: requirements.yml driver: name: docker lint: name: yamllint platforms: - name: instance image: solita/ubuntu-systemd:latest command: /sbin/init privileged: True volumes: - "/sys/fs/cgroup:/sys/fs/cgroup:rw" provisioner: name: ansible lint: name: ansible-lint scenario: name: default verifier: name: testinfra lint: name: flake8 

If the role being tested cannot be installed without other roles being installed before, then the molecule system should be asked to download the required dependencies. This is done through the dependency setting and the requirements.yml file (details about this file are in the Galaxy help ).

In the driver section, you choose the driver with which you will work (possible options are described here ).

In the platforms section you choose the platforms that your role will roll into. In the case of the docker, you specify the basic docker containers, and here I ask you to pay special attention to the example above . If your role is created by systemd-based services, then to be able to run them in the docker-container, you will need to use a special solita / ubuntu-systemd image and customize it specifically, as shown in the example. If you want to test your role on different platforms (for example, on different Linux distributions), then you can specify several values ​​here.

The documentation about the purpose of assigning other sections of the configuration file (for example, here you can override the execution matrix and configure static analyzers).

The next file you probably need to edit is playbook.yml . It is the usual Ansible playbook, which is executed for rolling a role onto a node, and for one of our roles it looks like this:

 --- - name: Converge hosts: all roles: - role: ansiblebit.oracle-java - role: fluteansible tasks: - name: mkdir for score file: dest: /var/opt/flute/score state: directory mode: "0775" group: flute - name: copy flute config file copy: src: flute.xml dest: opt/flute/flute.xml mode: "0644" - name: start flute service service: name: flute state: started 

Here you prescribe

1. Anything that must precede the installation of your role (in our case, the role is ansiblebit.oracle-java, which is also specified in requirements.yml and therefore will be automatically installed).
2. Installation of the tested role ( fluteansible in our case).
3. Any additional steps needed for further checks (in our case, we copy configuration files and start the service so that the infrastructure tests can verify that the service is started and executed correctly).

Steps "converge" and "idempotence"

At the step, the converge molecule simply executes the playbook.yml on a clean test node.

Then, in the idempotence step, the molecule executes the dry run of the same playbook.yml with the --diff option (for more on this, see the Ansible documentation ) to make sure that when you restart, the same role will not try to perform any Some changes — the system is already in the desired state.

As a rule, the idempotency of the Ansible code is provided automatically, but in some cases (first of all, for shell commands) you need to specify the conditions under which the command should not be repeated.

“Verify” step

Here is the most interesting. Molecule uses the testinfra system by default in order to perform checks on the current status of the test node after the role has been rolled up. Tests are written in Python and are in file molecule/default/tests/test_default.py . Test procedure names must begin with " test_ ".

Practically everything you want to check regarding the state of the system, you can easily do in a couple of lines of code on testinfra. For example, if you want to make sure that some command can be run, check the result of its execution and output to the console, you can do it like this:

 def test_jython_installed(host): cmd = host.run('jython --version') assert cmd.rc == 0 assert cmd.stderr.find(u'Jython 2') > -1 

Verifying that the service is running looks like this:

 def test_service_is_running(host): assert host.service('flute').is_running 

The presence of files and their contents can be checked as follows:

 def test_log_files(host): err = host.file('/var/log/flute/std.err') assert err.exists assert err.contains('Flute started') 

The possibilities of testinfra are far from being exhausted; here is the detailed documentation on the built-in capabilities.

CI / CD pipeline

As soon as we have static checks and autotests, then the natural next step is to build a CI / CD pipeline.

The release for the Ansible role is getting into Ansible Galaxy, which considers the latest commit to the master branch of your Github repository as release. Thus, if you use GitHub to control versions with a protected master branch, then each merge into a master branch will be a release. If the necessary conditions for the merger put the code review and checking on the CI-server (in which the Molecule-test is performed) - we build a stable delivery pipeline.

We use Jenkins Multibranch Pipeline, and our Jenkinsfile consists of only two steps:

 node { stage ("Get Latest Code") { checkout scm } try { stage ("Molecule test") { /* Jenkins check out the role into a folder with arbitrary name, so we need to let Ansible know where to find 'fluteansible' role*/ sh 'mkdir -p molecule/default/roles' sh 'ln -sf `pwd` molecule/default/roles/fluteansible' sh 'molecule test' } } catch(all) { currentBuild.result = "FAILURE" throw err } } 

Actually, there would be nothing to talk about here if it were not for one particularity. Jenkins Multibranch downloads the project to a folder with a name that differs from the name of the project (it can be called something like this: fluteansible_PR-3-7YQKOAVNLO7P2S6Z4O6BLBAFYNYCTBDGTLQFWMXA35XGZZZMLQRA ), therefore, in order for Molecule to search for a project, you will have to search for the fluteansible_PR-3-7YQKOAVNLO7P2S6Z4O6BLBAFYNYCTBDGTLQFWMXA35XGZZZMLQRA have to search for it in the fluteansible_PR-3-7YQKOAVNLO7P2S6Z4O6BLBAFYNYCTBDGTLQFWMXA35XGZZZMLQRA , and then search for the fluteansible_PR-3-7YQKOAVNLO7P2S6Z4O6BLBAFYNYCTBDGTLQFWMXA35XGZZZMLQRA , and then search for the fluteansible_PR-3-7YQKOAVNLO7P2S6Z4O6BLBAFYNYCTBDGTLQFWMXA35XGZZZMLQRA , and then search for the fluteansible_PR-3-7YQKOAVNLO7P2S6Z4O6BLBAFYNYCTBDGTLQFWMXA35XGZZZMLQRA hint "in the form of a symlink to the root of the project in the folder of the molecule / default / roles.

In this article, you can find another example of the Jenkinsfile for Molecule, in which the author was not lazy and broke each of the steps of the molecule script on the stage, and if successfully executed, writes the tags in the Git repository.

Conclusion

Ansible is an excellent system in itself. But with the presence of Molecule, you can begin to work wonders with it, creating large, difficult-to-configure roles, without feeling at the same time afraid of spoiling the code. The ease of use of Molecule along with its capabilities makes Molecule a “must have” tool in the design of Ansible scripts.

An example of the role being developed with the help of Molecule and Jenkins can be found here .