Accelerate Ansible with Mitogen

Ansible has become one of the most popular Configuration Management systems . After Red Hat was bought in 2015, the number of project participants exceeded thousands and Ansible was probably the most used deployment and orchestration system. Its wide applications are very impressive.

Ansible works through SSH connections to remote hosts. It opens an SSH session, makes a login, copies the Python code over the network and writes it into a separate temporary file. After that, it runs this file on a remote machine. This whole sequence of operations is quite long and tedious, so there are various ways to optimize it.

One of these methods is SSH pipelines which allows you to use one SSH session to execute instructions, rather than opening a new session every time, which can save us a lot of time. (Just do not forget to disable the requiretty setting for sudo in your /etc/sudoers file on the remote machine)

A new way to "accelerate" Ansible is the python library called Mitogen . if someone has not heard about it - then briefly describe its functionality. It allows fast execution of python code on a remote machine and Ansible is just one example of use. Mitogen uses the UNIX pipe on the remote machine and transmits the python code compressed zlib and serialized using pickle. This helps to perform it faster and saves traffic. If you are interested in a more detailed explanation, it is best to read about it on the page "How it works" . But today we will focus only on the work of the library with Ansible.

Mitogen in certain circumstances can speed up your Ansible code several times and significantly reduce traffic consumption. Let's check the most popular examples of use and see how it helps us.

I mostly use Ansible for: creating configuration files on a remote machine, installing packages, copying files to and from a remote machine. Perhaps you have other examples - write in the comments.

Go!

The Mitogen configuration for Ansible is very simple:
Install the Mitogen library:

 pip install mitogen 

Now there are two equivalent ways - either to configure the options in the configuration file ansible.cfg, or to set the desired environment variables.

Suppose that the path to the installed Mitogen is /usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy . Then:

 export ANSIBLE_STRATEGY_PLUGINS=/usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy export ANSIBLE_STRATEGY=mitogen_linear 

or

 [defaults] strategy = mitogen_linear strategy_plugins = /usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy 

Install Ansible in virtualenv, with and without Mitogen:

 virtualenv mitogen_ansible ./mitogen_ansible/bin/pip install ansible==2.7.10 mitogen virtualenv pure_ansible ./pure_ansible/bin/pip install ansible==2.7.10 

Please note that Mitogen 0.2.7 does not work with Ansible 2.8 (as of May 2019)

Make aliases:

 alias pure-ansible-playbook='$(pwd)/pure_ansible/bin/ansible-playbook' alias mitogen-ansible-playbook='ANSIBLE_STRATEGY_PLUGINS=$(pwd)/mitogen_ansible/lib/python3.7/site-packages/ansible_mitogen/plugins/strategy ANSIBLE_STRATEGY=mitogen_linear $(pwd)/mitogen_ansible/bin/ansible-playbook' 

Now we will try to start the playbook creating files on the remote machine:

 --- - hosts: all gather_facts: false tasks: - name: Create files with copy content module copy: content: | test file {{ item }} dest: ~/file_{{ item }} with_sequence: start=1 end={{ n }} 

And run it with and without Mitogen to create 10 files:

 time mitogen-ansible-playbook file_creation.yml -i hosts -en=10 &>/dev/null real 0m2.603s user 0m1.152s sys 0m0.096s time pure-ansible-playbook file_creation.yml -i hosts -en=10 &>/dev/null real 0m5.908s user 0m1.745s sys 0m0.643s 

We see an improvement of 2 times. Let's check for 20, 30, ..., 100 files:

 time pure-ansible-playbook file_creation.yml -i hosts -en=100 &>/dev/null real 0m51.775s user 0m8.039s sys 0m6.305s time mitogen-ansible-playbook file_creation.yml -i hosts -en=100 &>/dev/null real 0m4.331s user 0m1.903s sys 0m0.197s 

As a result, we accelerated the execution of more than 10 times!
Now we will try different scenarios and see how much faster everything works with us:

• The script for copying files to a remote host from a local host (with a copy module):
  

  
  

• The script for creating files on a remote host with a copy module:
  

  
  

• Script loading files from a remote host to a local host:
  

  
  

Let's try a script with several (3) remote machines, for example a script copying files to a remote host:

As you can see, Mitogen saves us both time and traffic in these scenarios. But if the bottleneck is not in Ansible, but for example in an I / O disk or network, or anywhere else, then it’s hard to expect Mitogen to help us.

Let's try a script with installing packages with yum / dnf and python modules using pip. Packages were cached so as not to depend on glitches on the network:

 --- - hosts: all gather_facts: false tasks: - name: Install packages become: true package: name: - samba - httpd - nano - ruby state: present - name: Install pip modules become: true pip: name: - pytest-split-tests - bottle - pep8 - flask state: present 

With Mitogen, it took 12 seconds, as well as without it.
On the Mitogen for Ansible page you can see other benchmarks and tests. As stated on the page:

Mitogen cannot speed up a module when it is executed. It can only make the execution of this module as fast as possible.

Therefore, it is important to find your bottlenecks in the deployment and if they are due to Ansible, then Mitogen will help you solve them and significantly speed up the execution of your playbooks.