Expanding Ansible with plugins: part 2

Under the hood of the d2c.io service , we actively use Ansible — from creating virtual machines in the clouds of providers and installing the necessary software, to managing Docker containers with client applications.

In the first part, we looked at the types of plug-ins that Ansible supports and made some of their plug-ins: test, filter, action and callback. In this article we will try more complex modifications.

Callback with "mutation"

The most frequent use of callback plug-ins are logging and alerting systems. However, with their help, you can not only perform passive observation of events, but also actively influence the progress of the playbook.

In order to be able to perform only some tasks from some roles, we actively use tags in D2C. For example, when you start a role with the build tag, the service will be fully assembled from scratch, and when started with the update-configs tag, only the configuration files will be updated and applied. In the out-of-the-box version, Ansible can apply a single set of tags to the entire playbook.

Let us analyze the task of launching Master-Slave replication for the MySQL service:

• need to update primary server configuration
• make a copy of the base for the initial replica filling
• make a second server, set up replication
• restore original base on cue
• delete temporary data

Each task has its own tags. To combine this process into one playbook, we can describe three playa (play is a configuration unit of which a playbook consists): to prepare a wizard, to prepare a replica, to clear it. However, we cannot specify the tags for each part separately, as they are set through the tags parameter for the entire playbook. Let's fix this by using the callback plugin:

 from ansible.plugins.callback import CallbackBase from ansible.parsing.yaml.objects import AnsibleUnicode from ansible.compat.six import string_types import json import os class CallbackModule(CallbackBase): CALLBACK_VERSION = 2.0 CALLBACK_NAME = 'use_tags' def __init__(self): super(CallbackModule, self).__init__() self.tmp_context = None self.warn = False if os.environ.get('ANSIBLE_D2C_NO_WARN') else True def v2_playbook_on_play_start(self, play): vm = play.get_variable_manager() extra_vars = vm.extra_vars enable_use_tags = False if 'enable_use_tags' in extra_vars: if extra_vars['enable_use_tags']: enable_use_tags = True play_vars = vm.get_vars(play._loader, play=play) if enable_use_tags: tags = self.tmp_context.only_tags tags.clear() if 'use_tags' in play_vars: use_tags = play_vars['use_tags'] if isinstance(use_tags, (string_types, AnsibleUnicode)): use_tags = [t.strip() for t in use_tags.split(',')] if isinstance(use_tags, list): for t in use_tags: tags.add(t) else: tags.add('all') self._display.display(' [INFO]: "use_tags" variable is set, but unparsable (type "{}" is not a list or a string): {}'.format(type(use_tags),use_tags), color='cyan') else: self._display.display(' [INFO]: "use_tags" variable is not set, but "enable_use_tags" is set', color='cyan') tags.add('all') if self.warn: self._display.warning('Tags modified to: {}'.format(json.dumps(list(tags)))) def set_play_context(self, play_context): self.tmp_context = play_context 

In our plugin, the main character is the v2_playbook_on_play_start method. It is called after the initialization of the play (filling with variables, determining the list of hosts, etc.) and before starting the execution of the tasks themselves.

We use the extra variable (extra var) enable_use_tags as a sign that we will use the modification of the tags “on the fly” and the variable of the play level (play var) use_tags to generate a list of the necessary tags.

Everything is good, but the tags, along with many other runtime information, are copied to the PlayContext object, the link to which is not in the v2_playbook_on_play_start method. To combat this, we note that the queue manager in Ansible checks for the presence of the set_play_context method in the connected plugins and, if it exists, calls it, passing this same context.

Using the circumstances that PlayContext mutable and that Ansible only works with one play at a time, we implement the following algorithm in the plugin:

• upon initial initialization of the plugin, tmp_context inside the plugin
• every time set_play_context call set_play_context remember the current context in tmp_context
• on a subsequent call to v2_playbook_on_play_start analyze the enable_use_tags and use_tags variables and change the original PlayContext object (more precisely, we get the “link” to the mutable list of tags via self.tmp_context.only_tags and modify the list)
• display appropriate warnings that the list of tags has been changed (so that there are no surprises for the user)

Now we can run this playbook:

ansible-playbook -e enable_use_tags=1 make_mysql_slave.yml

 - hosts: master vars: use_tags: update-configs, replication-init, replication-sync roles: - mysql - hosts: slave vars: use_tags: build, replication-init, replication-sync roles: - mysql - hosts: all vars: use_tags: replication-sync-cleanup roles: - mysql 

In this case, Ansible will use its own set of tags for each play. This gives us the opportunity to compose orchestration of complex configurations with single playbooks.

Connection

Connection plugins are used to connect to target hosts. In short: the plugin should provide the ability to establish and terminate the connection, send a file, run a remote command. Examples of plug-ins out of the box are: local , ssh (used by default), winrm , docker .

If you have very special target hosts, for example, any proprietary virtualization system, then you will have to write your plugin from scratch. But if you need to add a bit of functionality to the existing, you can inherit from the plug-in "out of the box" and override the necessary methods.

Consider an example of an SSH connection using port knoking . Basically, these ssh sessions are no different from normal ones, but before trying to connect to a remote machine, you need to “knock” on certain ports in order for the server to open port 22 and accept an ssh connection.

Let's finish the basic ssh plugin (put in ./connection_plugins/ssh_pkn.py):

 from ansible.plugins.connection.ssh import Connection as ConnectionSSH from ansible.errors import AnsibleError from socket import create_connection from time import sleep try: from __main__ import display except ImportError: from ansible.utils.display import Display display = Display() class Connection(ConnectionSSH): def __init__(self, *args, **kwargs): super(Connection, self).__init__(*args, **kwargs) display.vvv("SSH_PKN (Port KNock) connection plugin is used for this host", host=self.host) def set_host_overrides(self, host, hostvars=None): if 'knock_ports' in hostvars: ports = hostvars['knock_ports'] if not isinstance(ports, list): raise AnsibleError("knock_ports parameter for host '{}' must be list!".format(host)) delay = 0.5 if 'knock_delay' in hostvars: delay = hostvars['knock_delay'] for p in ports: display.vvv("Knocking to port: {0}".format(p), host=self.host) try: create_connection((self.host, p), 0.5) except: pass display.vvv("Waiting for {0} seconds after knock".format(delay), host=self.host) sleep(delay) 

We use the set_host_overrides method, which allows plugins to change their behavior depending on host / group variables. This method is called when creating a new connection when not using reuse. In our case, it should not once again "knock" the ports.

An example of an inventory file to use this plugin:

 [pkn] myserver ansible_host=my.server.at.example.com [pkn:vars] ansible_connection=ssh_pkn knock_ports=[8000,9000] knock_delay=2 

We have indicated that the connection plug-in ssh_pkn will be used for all the hosts in the pkn group. When initializing our plugin inside the set_host_overrides method, the condition that the knock_ports variable is defined will work. Then, for each of the ports in the list, an attempt will be made to connect at knock_delay intervals of 2 seconds. We also intercept all exceptions from create_connection , since most likely the ports for “tapping” are closed and connection attempts will fail. However, for us it is not particularly important - the server will see attempts in any case.

Strategy

Strategy-type plug-ins define the order in which tasks are launched and do a lot of engine work: including dynamically adding facts, monitoring the status of hosts (healty / failed / unreachable), and invoking callbacks. I wrote about strategy out-of-box plugins in the first part .

Such custom plugins are extremely rare. In an unaccepted pull request , 18460 , for example, offered a plugin with the possibility of injecting tasks into an arbitrary place of a playbook, in order to increase the flexibility of distributed roles. We will make a more landed strategy-plugin.

Put in ./strategy_plugins/step_critical.py:

 from ansible.plugins.strategy.linear import StrategyModule as LinearStrategyModule import os try: from __main__ import display except ImportError: from ansible.utils.display import Display display = Display() class StrategyModule(LinearStrategyModule): def __init__(self, tqm): super(StrategyModule, self).__init__(tqm) display.vv('Safenet strategy: will give a prompt at critical tasks!') force_step = os.environ.get('ANSIBLE_FORCE_STEP', None) if force_step and force_step.lower() in ['1','y','yes','true','on']: display.vv('Safenet: "step" option is forced via environment!') self._step = True def _take_step(self, task, host=None): v = task.get_vars() ret = True if 'is_critical' in v: if v['is_critical']: display.vv('Safenet: critical task detected!') return super(StrategyModule, self)._take_step(task, host) return ret 

This plugin changes the behavior of the --step parameter --step such a way that Ansible asks permission only for tasks that have the is_critical variable and its value True , and not for everyone, as it happens out of the box.

We can also force the confirmation mode through the environment variable ANSIBLE_FORCE_STEP , and not only through the --step parameter. Otherwise, this plugin inherits the behavior of a linear plugin.

You can check the behavior of the plugin with the following playbook:

 --- - hosts: localhost strategy: step_critical gather_facts: no tasks: - name: Ensure user exists debug: msg: user_module - name: Drop database debug: msg: db_module vars: is_critical: yes - name: Ensure permissions debug: msg: permission_module 

Total

In two articles on the expansion capabilities of Ansible, we looked at all types of plug-ins that are supported in Ansible 2.3. And also I gave examples for most of them.

If any questions about the plugins are not disclosed, write in the comments - I will try to answer.

In the meantime, I'm starting to prepare an article about creating modules for Ansible. Stay tuned!