We are actively using Ansible in D2C . With it, we create virtual machines with cloud providers, install software, as well as manage Docker-containers with client applications. In the last article I talked about how to make Ansible work faster , now I will talk about how to extend its functionality.

Ansible is an extraordinarily flexible tool. It is written in Python and mainly consists of replaceable "cubes" - plug-ins and modules. Plug-ins affect the work flow of Ansible on the management machine, modules are executed on remote hosts and return the result to the management machine. Therefore, if the Ansible functionality “out of the box” is not enough for you, it is enough to write your own plugin or module, and then add it to the system. An additional convenience is that plug-ins and modules do not need to be specially installed on the control machine and can be distributed directly with their own playbooks.

Consider an example:

--- - hosts: localhost vars: foo: - a - b - c tasks: - copy: content: "{{ foo | shuffle }}" dest: /tmp/test 

In this case, copy is a module. It will be executed on the target machine; shuffle - Jinja2 filter loaded by plugin. Plugins in Ansible perform not only visible, but also hidden from the work.

Important: All plug-ins in Ansible run in the context of the local host (i.e., the management machine). One of the common mistakes is to try to read the environment variables on the target host using the env lookup plugin:

 - name: show current user shell: echo {{ lookup('env','USER') }} 

In this case, it does not matter on which server the task is executed, the USER variable will be equal to the value that is set by the ansible process on the management machine. Similar to all other plugins.

Types of plugins

I will list the types of plugins in alphabetical order, for Ansible 2.3.x:

Action

Action plugins are used as wrappers for modules. They are executed immediately before sending the modules for execution to the target hosts. Usually they are used for preliminary data preparation or for post-processing the results of module execution.

In general, the task execution for mymodule looks like this:

• The action plugin mymodule launched on the local computer;
• Inside the plugin, preparatory operations are performed;
• A command is given to launch the module.
• The mymodule module is started on the remote computer;
• The result is returned to the local computer.
• Control returns to the action plugin mymodule ;
• Post-processing the results

If the action plugin mymodule does not exist, the base class of the plugin is used.

Cache

Cache plugins are used to organize the repository (backends) of facts. The default is the memory backend, so the facts are saved only during the playlist execution. Alternative out-of-box jsonfile : jsonfile , memcached , pickle , redis , yaml .

The fact cache is used if you need to work with many remote hosts and it takes a long time to collect facts. In such a situation, it is possible to update the cache on a schedule, and in the playbooks themselves, the collection of facts cannot be performed or, in rare cases, by a forced command.

Callback

Callback plugins provide the ability to respond to events that Ansible generates during the playlist execution. For example, the output of the Ansible job log to the screen is done by the default callback plugin, which responds to many events and displays what is happening on the screen. You can turn on the slack callback plugin and get information about the progress of the playbook in the channel in Slack.

Connection

Connection plugins provide various ways to connect to target hosts — for example, ssh — for Unix, winrm — for Windows, docker — to run modules inside containers. The most common are ssh (default) and local , which is used to run commands locally on the management machine.

Filter

Filter plugins add new Jinja2 filters. Since for working with variables in Ansible Jinja2 template engine is used, almost all of its features are available in playbooks, including built-in and additional filters. If non-standard filters are required, you can add them with your own plugins.

Lookup

Lookup plugins are used to search or download data from external sources, as well as to create loops.

For example, you can use {{ lookup('etcd', 'foo') }} to load values ​​from etcd .

To make a loop on the lines of the command output, you can use the lines plugin:

 - debug: msg: "{{ item }}" with_lines: cat /etc/passwd 

In this task, the cat /etc/passwd command will be executed (on the local computer) and a debug pop up for each of the output lines.

To create a loop, you can use any lookup plugin in the with_<plugin-name>: construct. When you do the most primitive with_items , the items lookup plugin is called.

The list of available plug-ins is most convenient to look in the repository (pay attention to the version - this link is for the 2.3.x branch).

Shell

Shell plug-ins allow you to take into account the nuances of the different behavior of shells on target devices. For example, bash or csh . For Windows, the powershell plugin is used.

Strategy

Strategy plug-ins determine the progress of tasks on target hosts. Out of the box there are three plugins available:

• linear (enabled by default) - Ansible performs the current task on all hosts in turn, only after its execution on all hosts moves to the next task
• free - tasks are performed on each of the hosts as quickly as possible, and not waiting for all hosts
• debug - modification of linear - in case of an error, an interactive shell is enabled, which allows you to view the current variables, make changes to the parameters of the task and rerun it ( documentation )

Terminal

Terminal-plugins allow you to take into account varieties of interactive environments. These plugins are used for network devices such as switches and routers, since working with the shell on these devices is significantly different from the operation of a full-featured shell on a computer.

Test

Test plugins add Jinja2 tests that are used in conditional constructs. Similar to filters, there are built-in and additional tests.

Vars

Vars-plugins are used to manipulate host variables (host vars, group vars) - extremely rare.

We write our plugins

I will give a few examples of plugins in order of increasing complexity.

Test

For example, we often work with lists of EC2 servers and you need to select those that work from the list of instances. You can use the expression:

{{ ec2.instances | selectattr('state','equalto','running') | list }}

Or write your test plugin (put in ./test_plugins/ec2.py):

 class TestModule: def tests(self): return { 'ec2_running': lambda i: i['state'] == 'running' } 

And already use:

{{ ec2.instances | select('ec2_running') | list }}

Obviously, I have simplified the example a little, and if we need to check several statuses at the same time, then we would have to make a chain from the set selectattr . In our own test, we can describe any logic and at the same time keep the playbook code concise and well readable.

Similarly, you can use your tests inside when :

 when: my_instance | ec2_running 

The task will be executed if my_instance in the running state.

You can create tests with a parameter. An example is the standard test divisibleby , which checks whether it is divisible into something else.

Filter

Filters are used to modify variables. For example, in Ansible for a very long time there were no mechanisms for working with a date. If you need to make decisions based on time values ​​in playbooks, you can use this filter (put in ./filter_plugins/add_date.py):

 import datetime class FilterModule(object): def filters(self): return { 'add_time': lambda dt, **kwargs: dt + datetime.timedelta(**kwargs) } 

Now in playbooks you can “look into” the future:

 - debug: msg: "Current time +20 mins {{ ansible_date_time.iso8601[:19] | to_datetime(fmt) | add_time(minutes=20) }}" vars: fmt: "%Y-%m-%dT%H:%M:%S" 

Action

Action plug-ins are useful when you need to slightly modify the data coming into or out of the modules, or when you need to perform some task always locally on the management server. An example is the debug module for displaying information, in fact it is not a module since it is never copied to a remote host, but exists only in the form of an action plugin.

To show how action-plugins work, we modify the behavior of the setup module, which is used to collect facts. It is convenient to use it as an ad-hoc command to view information about servers:

ansible all -i myinventory -m setup

This module has a filter parameter that can filter the result. But he has one feature - it applies only to top-level keys. If we only need to check the time zone on the servers, we cannot specify tz . Or if we need to see all ipv4 addresses, we cannot make a filter for all such fields.

Add a wrapper in the form of an action plugin (put in ./action_plugins/setup.py):

 from ansible.plugins.action import ActionBase class ActionModule(ActionBase): def run(self, tmp=None, task_vars=None): def filter_dict(obj, filter): res = dict() for k, v in obj.items(): if filter in k: res[k] = v elif isinstance(v, dict): val = filter_dict(v, filter) if val is not None and val != dict(): res[k] = val return res result = super(ActionModule, self).run(tmp, task_vars) query = self._task.args.get('query', None) module_args = self._task.args.copy() if query: module_args.pop('query') module_return = self._execute_module(module_name='setup', module_args=module_args, task_vars=task_vars, tmp=tmp) if not module_return.get('failed') and query: return dict(ansible_facts=filter_dict(module_return['ansible_facts'], query)) else: return module_return 

The minimum necessary implementation of the plugin is the inheritance from ActionBase and the description of the run method.

In our example, we:

• We defined the auxiliary function filter_dict , which takes an object (dictionary) as input, looks for keys in it according to our filter and returns the object only with those keys that satisfy the filter (no matter what level of nesting they met);
• run parent class's run method;
• We tried to get the value of our new query parameter: if it exists, remember it and remove it from the list of parameters that we will pass on to the module - otherwise Ansible will say that the setup module does not know anything about the query parameter and will generate an error;
• Transferred control to the setup module;
• If the module has successfully completed its work and the query parameter has been set - we filter the result with our filter_dict function, otherwise we return the result without names;

Now we have added a new functionality to the existing module and at the same time did not touch its code. With the release of new versions of Ansible, the setup module can learn to do something else, but our plugin will still work on top of these features.

Callback

Callback plugins are used to monitor the events that occur inside Ansible during the playlist and somehow respond to them. One of the most frequent uses of such plugins is logging, logging and alerting.

A list of callback plugins available out of the box can be viewed in the repository .

For notifications, for example, are available: mail , slack , hipchat .

To modify the logging, for example: minimal , json . To set the standard output plug-in, you can use the setting:

 [defaults] stdout_callback = json 

Now Ansible will not display a human-readable protocol in the course of execution, but at the very end of the execution, the playbook will issue a huge JSON with all the information. It can be used to automatically analyze the results in cron tasks or on your CI / CD server.

For example, you can start the playbook and count the number of hosts in which there were changes:

ANSIBLE_STDOUT_CALLBACK=json ansible-playbook myplaybook.yml | jq '.stats | map(select(.changed > 0)) | length'

As an example of a callback plug-in, I’ll give a notification about the playbook execution, which displays a notification in your graphic shell about the playlist execution results (put in ./callback_plugins/notify_me.py):

 from ansible.plugins.callback import CallbackBase from subprocess import call from platform import system as get_system_name class CallbackModule(CallbackBase): CALLBACK_VERSION = 2.0 CALLBACK_TYPE = 'notification' CALLBACK_NAME = 'notify_me' CALLBACK_NEEDS_WHITELIST = True def v2_playbook_on_stats(self, stats): def notify(msg,is_error=False): sys_name = get_system_name() if sys_name == 'Darwin': sound = "Basso" if is_error else "default" call(["osascript", "-e", 'display notification "{}" with title "Ansible" sound name "{}"'. format(msg,sound)]) elif sys_name == 'Linux': icon = "dialog-warning" if is_error else "dialog-info" rc = call(["notify-send", "-i", icon, "Ansible", msg]) print "error code {}".format(rc) hosts = stats.processed.keys() failed_hosts = [] for h in hosts: t = stats.summarize(h) if t['unreachable'] + t['failures'] > 0: failed_hosts.append(h) if len(failed_hosts) > 0: notify("Failed hosts: {}".format(" ".join(failed_hosts)),True) else: notify("Job's done!") 

The plugin provides some kind of cross-platform attempt :)

We inherit from the CallbackBase class and override the v2_playbook_on_stats method, which is called when the final performance report is ready. A standard logging plugin using this method forms the PLAY RECAP table.

We need the auxiliary function notify , which, depending on the platform, tries to send an alert to the user.

In the main body of our method, we check if there are any hosts with errors: if there is, send a bad notification with the list of hosts; if not, send a good notification. Job's done! .

Pay attention to CALLBACK_NEEDS_WHITELIST = True . This parameter tells Ansible that this plugin requires forced inclusion. That is, despite the willingness of the plug-in to work, it will be included only when it is added to the whitelist. This is done so that when working with playbooks, the screen does not litter, but you can easily put such a notification for “long-playing” playbooks that you run in the background and go to do another thing. You can check the work like this:

ANSIBLE_CALLBACK_WHITELIST=notify_me ansible-playbook test.yml

The full list of methods (events) that can be overridden in callback plugins is best seen in the source code .

-

Feel free to experiment with examples from the article. A few more types of plug-ins will be discussed in the next section. Stay tuned!