Comprehensive Automation with Ansible and OpenStack
Ansible solutions guarantee maximum flexibility. This allows the community to find new ways to use Ansible modules to automate frequently performed operations at many levels, including in combination with OpenStack technologies.
In this blog, we will discuss the many options for using Ansible, the most popular automation software (software), in conjunction with OpenStack, the most popular software for cloud infrastructure. We will help you understand how and why you should use Ansible to make your life easier with the help of integrated automation (Full-Stack Automation), as we like to call it.
First, let's discuss the levels of complex automation shown in the diagram above. At the bottom of our equipment (servers, storage systems and network equipment). Next comes the operating system (Linux or Windows). On Linux, you can install OpenStack to abstract away from the physical resources of the data center and deploy the software version of the computing resources, network, and storage. On top of OpenStack, tenant-specific services are needed to create the virtual machines on which applications will run. Finally, you must manage the operating system (Linux or Windows) to deploy the applications and workloads that you really need (databases, web servers, mobile application servers, etc.). If you use containers (for example, Docker or Rkt), you will package these applications in images that will then be deployed on top of the guest OS. In addition, in some languages, the concept of application servers has appeared, which adds another layer (for example, J2EE).
')
Ansible provides modules for managing each layer. Even for network equipment, more precisely, if you approach this from a technical point of view, for a network operating system such as IOS or NXOS (see the full list of Ansible network modules ).
1. Standard interaction with the operating system: installing packages, changing or forcing the use of content or file access rights, managing services, creating and deleting users and groups of users, etc.
2. Software for implementing the infrastructure as a service (IaaS) concept: install the IaaS software and related components (databases, load balancers, configuration files, services, and other supporting tools).
3. Virtual resources: configure resources, such as a virtual machine or an instance, by defining the size, access rights, content, security profile, network connection settings, etc.
4. Guest OS: components are similar to those of the host OS. But how do you know how many guests you have?
5. Container management module (optional).
6. Rented software: databases, web servers, load balancers, data processing modules, etc.
As you already understood, some functions are available only in the newest Ansible versions, for example 2.2. However, your OS version may have an older version. For example, in RHEL 7 or CentOS 7 you will find only Ansible 1.9.
Given that Ansible is a command-line tool written in Python that does not prohibit the presence of multiple versions on the system, you may not need to strengthen the security in Ansible that your build offers, and you want to try out the latest version.
However, as is the case with any other Python software, there are many dependencies; therefore, it is not recommended to use untested new libraries together with system libraries. Libraries can be shared by other components of your system, and untested newer versions can disrupt other applications. The easiest way is to install the latest version of Ansible with all dependencies in an isolated folder under your unprivileged account. This will be a virtual Python environment ( Python Virtual Environment , virtualenv), and if you do it right, you can safely try out the latest Ansible modules for complex orchestration. Of course, it is not recommended to apply this practice in a production environment. This is nothing more than an exercise to help you improve your DevOps skills.
1. Install the required components (pip, virtualenv)
In this case, we only need one “system-wide” Python library - virtualenvwrapper. In addition, you should not execute the sudo pip install command, as this will replace the Python system libraries with newer unverified versions. Trust in this case, we can only library virtualenvwrapper. Virtual environments are an effective mechanism for installing and testing new Python modules in your unprivileged user account .
In addition, if your account is different from sudoer, you can use the following commands.
2. Install a new virtual environment where we will deploy the latest version of Ansible
First, create a directory for storing virtual environments.
Then add the following lines to your .bashrc file:
export WORKON_HOME = $ HOME / .virtualenvs
source /usr/bin/virtualenvwrapper.sh
Now run the source command for this file.
At this stage, shell links are created, but only on the first launch. Run the workon command to view the list of environments, it should not be empty. Next, we will create a new virtualenv environment named ansible2 , it will be activated automatically and get access to the default packages using RPM.
Enter deactivate to exit virtualenv, and workon to re-enter.
3. Log in to the new virtualenv environment and install Ansible2 via PIP (as a regular user, without root-rights)
You will see that the prompt for entering the command in the shell has changed and the name virtualenv is shown in brackets.
This command will install only the dependencies for Ansible 2 using your system-wide Python packages available in RPM (thanks to the system-site-packages flag we used earlier). Alternatively, if you want to try out the development branch.
If you need to remove virtualenv and all its dependencies, just enter the command rmvirtualenv ansible2.
NOTE. If the commands above did not work, make sure that the GCC and OpenSSL libraries are installed on your systems, since some Python dependencies use the latest cryptographic modules.
4. Install OpenStack Client Dependencies
The first command in the code snippet below ensures that you have the latest stable versions of the OpenStack API, however you can also use the pip install command to get the latest version of the command line tool (CLI). The second team provides the latest version of the Python Shade library to connect to the latest versions of the OpenStack API using ansible, regardless of the specific CLI tool.
5. Test your deployment
NOTE. You cannot run this version of ansible outside the virtual environment of virtualenv, so always run the workon ansible2 command before usi.
Attentive readers will notice that using Ansible to orchestrate OpenStack, we seem to be ignoring the fact that Heat is the official orchestration module for this platform. In fact, Ansible Playbook provides almost the same features as the HOT template (HOT is the YAML-based syntax for Heat, AWS CloudFormation reincarnation). However, many DevOps professionals prefer not to waste time learning the new syntax, and they are already consolidating the entire process for their hybrid infrastructure.
The Ansible team understands this, so it uses Shade , the official library from the OpenStack project, to create interfaces for calling the OpenStack API. At the time of this writing, Ansible 2.2 included modules for calling the following APIs.
From the Ansible point of view, it is necessary to organize interaction with the server, where the system can load OpenStack credentials and create HTTP connections with various OpenStack APIs. If this server is your machine (localhost), then Ansible will work locally, load Keystone credentials and initiate interaction with OpenStack.
Let's take an example. We will use Ansible OpenStack modules to connect to Nova and launch a small instance with the Cirros image. First we’ll upload the latest Cirros image, if you haven’t done so before. We will use the existing SSH key of the current user. You can download this playbook from this link on Github.
After launch, we see the instance IP address. We write it down. Now you can use Ansible to connect to this instance via SSH. It is assumed that the default Nova network supports connections from our workstation (in our case, through the network of the service provider).
Using Ansible instead of Heat has its advantages and disadvantages. For example, when working with Ansible, you must track the resources you create and manually delete them (in reverse order) when they are no longer needed. Especially difficult you will have with Neutron ports, floating IP addresses and routers. But if it is Heat, then you simply remove the stack, and all created resources will be deleted automatically.
Compare the above example with a similar (but not similar) Heat Template template, which can be downloaded from the following Github gist repository.
Now let's see what happens if we create many instances, but forget to write down their IP addresses. A great example of using Dynamic Inventory for OpenStack is analyzing the current state of our leased virtualized resources and getting all the IP addresses of servers so that we can, for example, check their kernel version. For example, this is done transparently with the Ansible Tower solution, which will periodically take inventory and generate an updated list of OpenStack servers to be managed.
Before performing these operations, make sure that you do not have outdated cloud.yaml files in the ~ / .config / openstack, / etc / openstack or / etc / ansible directory. The Dynamic Inventory script will first search for environment variables (OS_ *), and then the listed files.
For interest, you can view all the information that the above inventory script returns, to do this, follow these steps.
$ ./openstack.py –list
Although Heat is very useful, some may prefer to learn Ansible to use this orchestration solution, as it suggests using a single language to define and automate the entire set of IT resources. I hope this article was useful to you from a practical point of view. We looked at the basic use of Ansible for running OpenStack resources. If you are interested and want to evaluate the capabilities of Ansible and Ansible Tower, visit the resource . A good starting point would be to implement Heat to communicate with Ansible Tower via callbacks, as described in another blog post.
In addition, if you want to learn more about the Red Hat OpenStack platform, then on our website you will find many valuable resources (including videos and official documents ).
In this blog, we will discuss the many options for using Ansible, the most popular automation software (software), in conjunction with OpenStack, the most popular software for cloud infrastructure. We will help you understand how and why you should use Ansible to make your life easier with the help of integrated automation (Full-Stack Automation), as we like to call it.
First, let's discuss the levels of complex automation shown in the diagram above. At the bottom of our equipment (servers, storage systems and network equipment). Next comes the operating system (Linux or Windows). On Linux, you can install OpenStack to abstract away from the physical resources of the data center and deploy the software version of the computing resources, network, and storage. On top of OpenStack, tenant-specific services are needed to create the virtual machines on which applications will run. Finally, you must manage the operating system (Linux or Windows) to deploy the applications and workloads that you really need (databases, web servers, mobile application servers, etc.). If you use containers (for example, Docker or Rkt), you will package these applications in images that will then be deployed on top of the guest OS. In addition, in some languages, the concept of application servers has appeared, which adds another layer (for example, J2EE).
')
Management capabilities provided by Ansible solutions
Ansible provides modules for managing each layer. Even for network equipment, more precisely, if you approach this from a technical point of view, for a network operating system such as IOS or NXOS (see the full list of Ansible network modules ).
1. Standard interaction with the operating system: installing packages, changing or forcing the use of content or file access rights, managing services, creating and deleting users and groups of users, etc.
- Linux and BSD via SSH (the first and most popular use case).
- Windows via PowerShell (since version 1.7).
2. Software for implementing the infrastructure as a service (IaaS) concept: install the IaaS software and related components (databases, load balancers, configuration files, services, and other supporting tools).
- The OpenStack-ansible installer used in some OpenStack builds from other vendors. Do not forget that the Red Hat OpenStack platform does not use Ansible, but Heat and Puppet. In future releases, Ansible will be used to perform specific checks and assist operators in the process of installing updates and new versions.
- The CloudStack installer is also Ansible based project.
3. Virtual resources: configure resources, such as a virtual machine or an instance, by defining the size, access rights, content, security profile, network connection settings, etc.
- OpenStack Ansible modules (starting with Ansible 2.0), such as Nova or Neutron . They are based on the OpenStack Shade library, which is used by all CLI tools in OpenStack.
- These modules can also manage not so virtual network resources through netconf (starting with version 2.2).
- VMware vSphere Ansible Modules .
- RHV, or oVirt, or libvirt only for KVM.
- There are also modules for cloud service providers such as Amazon , Google Cloud , Azure and Digital Ocean .
4. Guest OS: components are similar to those of the host OS. But how do you know how many guests you have?
- The Ansible Dynamic Inventory tool will dynamically query the IaaS level and VM in order to detect currently available instances. This determines the hostname, IP address and security settings, that is, asset management becomes dynamic. This feature will be especially useful for those who use Auto Scaling Groups in their cloud infrastructure, as the list of instances changes dramatically over time.
5. Container management module (optional).
- Docker : Please note that the old Docker module in Ansible 2.1 will be replaced with a new own component.
- Kubernetes.
- Atomic Host.
6. Rented software: databases, web servers, load balancers, data processing modules, etc.
- Ansible Galaxy - a recipe repository (playbook) for deploying the most popular software, and this is the result of the joint work of thousands of community members.
- You can also manage a web infrastructure like JBoss , allowing Ansible to determine how the application will be deployed to the application server.
Recommendations for installing the latest version of Ansible in a virtual Python environment
As you already understood, some functions are available only in the newest Ansible versions, for example 2.2. However, your OS version may have an older version. For example, in RHEL 7 or CentOS 7 you will find only Ansible 1.9.
Given that Ansible is a command-line tool written in Python that does not prohibit the presence of multiple versions on the system, you may not need to strengthen the security in Ansible that your build offers, and you want to try out the latest version.
However, as is the case with any other Python software, there are many dependencies; therefore, it is not recommended to use untested new libraries together with system libraries. Libraries can be shared by other components of your system, and untested newer versions can disrupt other applications. The easiest way is to install the latest version of Ansible with all dependencies in an isolated folder under your unprivileged account. This will be a virtual Python environment ( Python Virtual Environment , virtualenv), and if you do it right, you can safely try out the latest Ansible modules for complex orchestration. Of course, it is not recommended to apply this practice in a production environment. This is nothing more than an exercise to help you improve your DevOps skills.
1. Install the required components (pip, virtualenv)
In this case, we only need one “system-wide” Python library - virtualenvwrapper. In addition, you should not execute the sudo pip install command, as this will replace the Python system libraries with newer unverified versions. Trust in this case, we can only library virtualenvwrapper. Virtual environments are an effective mechanism for installing and testing new Python modules in your unprivileged user account .
$ sudo yum install python-pip $ sudo pip install virtualenvwrapper $ sudo yum install python-heatclient python-openstackclient python2-shade In addition, if your account is different from sudoer, you can use the following commands.
wget https://bootstrap.pypa.io/get-pip.py python get-pip.py --user .local/bin/pip install virtualenvwrapper --user export PATH=$PATH:~/.local/bin #Remember to include this in your .bashrc 2. Install a new virtual environment where we will deploy the latest version of Ansible
First, create a directory for storing virtual environments.
$ mkdir $HOME/.virtualenvs Then add the following lines to your .bashrc file:
export WORKON_HOME = $ HOME / .virtualenvs
source /usr/bin/virtualenvwrapper.sh
Now run the source command for this file.
$ source ~/.bashrc At this stage, shell links are created, but only on the first launch. Run the workon command to view the list of environments, it should not be empty. Next, we will create a new virtualenv environment named ansible2 , it will be activated automatically and get access to the default packages using RPM.
$ mkvirtualenv ansible2 --system-site-packages Enter deactivate to exit virtualenv, and workon to re-enter.
$ deactivate $ workon ansible2 3. Log in to the new virtualenv environment and install Ansible2 via PIP (as a regular user, without root-rights)
You will see that the prompt for entering the command in the shell has changed and the name virtualenv is shown in brackets.
(ansible2) $ pip install ansible This command will install only the dependencies for Ansible 2 using your system-wide Python packages available in RPM (thanks to the system-site-packages flag we used earlier). Alternatively, if you want to try out the development branch.
(ansible2) $ pip install git+git://github.com/ansible/ansible.git@devel (ansible2) $ ansible --version If you need to remove virtualenv and all its dependencies, just enter the command rmvirtualenv ansible2.
NOTE. If the commands above did not work, make sure that the GCC and OpenSSL libraries are installed on your systems, since some Python dependencies use the latest cryptographic modules.
4. Install OpenStack Client Dependencies
The first command in the code snippet below ensures that you have the latest stable versions of the OpenStack API, however you can also use the pip install command to get the latest version of the command line tool (CLI). The second team provides the latest version of the Python Shade library to connect to the latest versions of the OpenStack API using ansible, regardless of the specific CLI tool.
(ansible2) $ yum install python-openstackclient python-heatclient (ansible2) $ pip install shade --upgrade 5. Test your deployment
(ansible2) $ ansible -m ping localhost localhost | SUCCESS => { "changed": false, "ping": "pong" } NOTE. You cannot run this version of ansible outside the virtual environment of virtualenv, so always run the workon ansible2 command before usi.
OpenStack Orchestration Using Ansible
Attentive readers will notice that using Ansible to orchestrate OpenStack, we seem to be ignoring the fact that Heat is the official orchestration module for this platform. In fact, Ansible Playbook provides almost the same features as the HOT template (HOT is the YAML-based syntax for Heat, AWS CloudFormation reincarnation). However, many DevOps professionals prefer not to waste time learning the new syntax, and they are already consolidating the entire process for their hybrid infrastructure.
The Ansible team understands this, so it uses Shade , the official library from the OpenStack project, to create interfaces for calling the OpenStack API. At the time of this writing, Ansible 2.2 included modules for calling the following APIs.
- Keystone: users, groups, roles, projects.
- Nova: servers, key pairs, groups, and security flags.
- Neutron: ports, network, subnets, routers, floating IP addresses.
- Ironic: knots, introspectives.
- Swift objects.
- Tom Cinder.
- Images Glance.
From the Ansible point of view, it is necessary to organize interaction with the server, where the system can load OpenStack credentials and create HTTP connections with various OpenStack APIs. If this server is your machine (localhost), then Ansible will work locally, load Keystone credentials and initiate interaction with OpenStack.
Let's take an example. We will use Ansible OpenStack modules to connect to Nova and launch a small instance with the Cirros image. First we’ll upload the latest Cirros image, if you haven’t done so before. We will use the existing SSH key of the current user. You can download this playbook from this link on Github.
--- # Setup according to Blogpost "Full Stack automation with Ansible and OpenStack". Execute with "ansible-playbook ansible-openstack-blogpost.yml -c local -vv" # # # # # # - name: Execute the Blogpost demo tasks hosts: localhost tasks: - name: Download cirros image get_url: url: http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img dest: /tmp/cirros-0.3.4-x86_64-disk.img - name: Upload cirros image to openstack os_image: name: cirros container_format: bare disk_format: qcow2 state: present filename: /tmp/cirros-0.3.4-x86_64-disk.img - name: Create new keypair from current user's default SSH key os_keypair: state: present name: ansible_key public_key_file: "{{ '~' | expanduser }}/.ssh/id_rsa.pub" - name: Create the test network os_network: state: present name: testnet external: False shared: False #provider_network_type: vlan #provider_physical_network: datacentre register: testnet_network - name: Create the test subnet os_subnet: state: present network_name: "{{ testnet_network.id }}" name: testnet_sub ip_version: 4 cidr: 192.168.0.0/24 gateway_ip: 192.168.0.1 enable_dhcp: yes dns_nameservers: - 8.8.8.8 register: testnet_sub - name: Create the test router ignore_errors: yes #for some reasons, re-running this task gives errors os_router: state: present name: testnet_router network: nova external_fixed_ips: - subnet: nova interfaces: - testnet_sub - name: Create a new security group os_security_group: state: present name: secgr - name: Create a new security group allowing any ICMP os_security_group_rule: security_group: secgr protocol: icmp remote_ip_prefix: 0.0.0.0/0 - name: Create a new security group allowing any SSH connection os_security_group_rule: security_group: secgr protocol: tcp port_range_min: 22 port_range_max: 22 remote_ip_prefix: 0.0.0.0/0 - name: Create server instance os_server: state: present name: testServer image: cirros flavor: m1.small security_groups: secgr key_name: ansible_key nics: - net-id: "{{ testnet_network.id }}" register: testServer - name: Show Server's IP debug: var=testServer.openstack.public_v4 After launch, we see the instance IP address. We write it down. Now you can use Ansible to connect to this instance via SSH. It is assumed that the default Nova network supports connections from our workstation (in our case, through the network of the service provider).
Comparison with OpenStack Heat
Using Ansible instead of Heat has its advantages and disadvantages. For example, when working with Ansible, you must track the resources you create and manually delete them (in reverse order) when they are no longer needed. Especially difficult you will have with Neutron ports, floating IP addresses and routers. But if it is Heat, then you simply remove the stack, and all created resources will be deleted automatically.
Compare the above example with a similar (but not similar) Heat Template template, which can be downloaded from the following Github gist repository.
heat_template_version: 2015-04-30 description: > Node template. Launch with "openstack stack create --parameter public_network=nova --parameter ctrl_network=default --parameter secgroups=default --parameter image=cirros --parameter key=ansible_key --parameter flavor=m1.small --parameter name=myserver -t openstack-blogpost-heat.yaml testStack" parameters: name: type: string description: Name of node key: type: string description: Name of keypair to assign to server secgroups: type: comma_delimited_list description: List of security group to assign to server image: type: string description: Name of image to use for servers flavor: type: string description: Flavor to use for server availability_zone: type: string description: Availability zone for server default: nova ctrl_network: type: string label: Private network name or ID description: Network to attach instance to. public_network: type: string label: Public network name or ID description: Network to attach instance to. resources: ctrl_port: type: OS::Neutron::Port properties: network: { get_param: ctrl_network } security_groups: { get_param: secgroups } floating_ip: type: OS::Neutron::FloatingIP properties: floating_network: { get_param: public_network } port_id: { get_resource: ctrl_port } instance: type: OS::Nova::Server properties: name: { get_param: name } image: { get_param: image } flavor: { get_param: flavor } availability_zone: { get_param: availability_zone } key_name: { get_param: key } networks: - port: { get_resource: ctrl_port } Using dynamic inventory in conjunction with OpenStack modules
Now let's see what happens if we create many instances, but forget to write down their IP addresses. A great example of using Dynamic Inventory for OpenStack is analyzing the current state of our leased virtualized resources and getting all the IP addresses of servers so that we can, for example, check their kernel version. For example, this is done transparently with the Ansible Tower solution, which will periodically take inventory and generate an updated list of OpenStack servers to be managed.
Before performing these operations, make sure that you do not have outdated cloud.yaml files in the ~ / .config / openstack, / etc / openstack or / etc / ansible directory. The Dynamic Inventory script will first search for environment variables (OS_ *), and then the listed files.
#ensure you are using latest ansible version $ workon ansible2 $ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/openstack.py $ chmod +x openstack.py $ ansible -i openstack.py all -m ping bdef428a-10fe-4af7-ae70-c78a0aba7a42 | SUCCESS => { "changed": false, "ping": "pong" } 343c6e76-b3f6-4e78-ae59-a7cf31f8cc44 | SUCCESS => { "changed": false, "ping": "pong" } For interest, you can view all the information that the above inventory script returns, to do this, follow these steps.
$ ./openstack.py –list
{ "": [ "777a3e02-a7e1-4bec-86b7-47ae7679d214", "bdef428a-10fe-4af7-ae70-c78a0aba7a42", "0a0c2f0e-4ac6-422d-8d9b-12b7a87daa72", "9d4ee5c0-b53d-4cdb-be0f-c77fece0a8b9", "343c6e76-b3f6-4e78-ae59-a7cf31f8cc44" ], "_meta": { "hostvars": { "0a0c2f0e-4ac6-422d-8d9b-12b7a87daa72": { "ansible_ssh_host": "172.31.1.42", "openstack": { "HUMAN_ID": true, "NAME_ATTR": "name", "OS-DCF:diskConfig": "MANUAL", "OS-EXT-AZ:availability_zone": "nova", "OS-EXT-SRV-ATTR:host": "compute-0.localdomain", "OS-EXT-SRV-ATTR:hypervisor_hostname": "compute-0.localdomain", "OS-EXT-SRV-ATTR:instance_name": "instance-000003e7", "OS-EXT-STS:power_state": 1, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-10-10T21:13:24.000000", "OS-SRV-USG:terminated_at": null, "accessIPv4": "172.31.1.42", "accessIPv6": "", (....) Conclusion
Although Heat is very useful, some may prefer to learn Ansible to use this orchestration solution, as it suggests using a single language to define and automate the entire set of IT resources. I hope this article was useful to you from a practical point of view. We looked at the basic use of Ansible for running OpenStack resources. If you are interested and want to evaluate the capabilities of Ansible and Ansible Tower, visit the resource . A good starting point would be to implement Heat to communicate with Ansible Tower via callbacks, as described in another blog post.
In addition, if you want to learn more about the Red Hat OpenStack platform, then on our website you will find many valuable resources (including videos and official documents ).
Source: https://habr.com/ru/post/329556/
All Articles