vSphereTools: How we created an open source automation tool for vSphere

In our blog on Habré, we talk a lot about the implementation of DevOps approaches and the tools for automating development and testing that we developed. Today we will talk about how we solved the problem of interaction with VMware vSphere.

A bit of history

All our virtual machines run in vSphere, both build and test servers.
')
A couple of years ago, before the advent of our own Continuous Integration system, most of our builds were moved to the TeamCity tool. At that time, we didn’t have test and deployment configurations, but the need for their development was becoming increasingly apparent.

In mid-2014, we knew two solutions for automating virtual machines: using our own VMware API for working with vSphere (the VIX API library) and using the pysphere library. Each of these tools had flaws.

The VIX API was too high for us to enter:

• The tool was a set of complex and poorly documented C-libraries.
• All integration with our CI systems would have to be written independently in C.

The pysphere library in general was actually undocumented and with a number of problems:

• The tool is written under Python 2.7;
• there was no command line interface;
• the product was a collection of disparate classes randomly scattered around the package scripts;
• The project only developed and contained many bugs.

However, if you compare pysphere with the VIX API, then the first one also had its advantages:

• It works much faster by using http rest api to access vSphere features.
• Our company's expertise in Python is much higher, so the threshold for entering this tool was lower.
• Reviews of our samopisny library PT.VIX over the VIX API did not withstand any criticism in terms of support and reliability of its work.

Therefore, it was decided to use this library and create its own solution based on it.

VSphereTools project

Before starting to program, we made up the requirements for a set of automation tools.

• the product had to support work from the console with a wide set of commands;
• have a clear API for the ability to import it as a normal python library;
• be built into TeamCity as metaranners available for selection in configuration steps.

The conceptual model of the interaction of vSphereTools with the Sphere itself is presented below:



Everything is quite simple: the initiator, which can be a user or a script, sends commands received via the command line to the vSphereTools tool, which forwards the data to the pysphere library. Then the request goes to the vSphere, where it is processed and sent to the virtual machine, which gives us what we need ("steits", attributes, etc.)

The environmental requirements for vSphereTools were minimal:

• need a service user to access vSphere;
• The vcenter service must be available from the machine running vSphereTools, as well as the ESX running the target virtual machine.
• VMware Tools should also be installed on the target virtual machine;
• the machine on which vSphereTools is running must have Python 2 * version 2.7 or later installed.

Full documentation with examples is available in the open community DevOpsHQ . As a result, vSphereTools scripts implement many functions from starting and stopping virtual machines to copying a local file into the desired virtual machine or running a specific program on it.

Currently, vSphereTools metranners and scripts are used in the development and testing of practically all major Positive Technologies products — for example, MaxPatrol SIEM , PT Application Firewall and Application Inspector, and in many other projects.

Restrictions and possible improvements

Like any tool, our vSphereTools product has its limitations:

• The latest version available in the index is pysphere 0.1.8, namely, it is the basis of vSphereTools.
• New methods for vSphereTools have to be implemented only by the DevOps department of the company.
• Newer versions of VMware vSphere may not support the old API.
• At the moment, our scripts work only with Python 2 * (2.7 and higher).

Therefore, we are planning a number of improvements. For example, we want to rewrite vSphereTools on the VMware vCloud Suite SDK for Python for vSphere 6.0. The documentation for this tool is already available, there are also python-api for it - the pyvmomi library.

PS The story about the vSphereTools project was presented within the DevOps-mitap, which took place in the fall in Moscow.

Video:



Slides



The link presents presentations of 16 reports presented during the event. All presentations and video presentations will be added to the table at the end of this topic-announcement .

Author : Timur Gilmullin

PPS We remind you that very soon, with the support of Positive Technologies, Moscow will take a course on asyncio + aiohttp from Core developer Python Andrei Svetlov.

We want to offer one free ticket to the seminar to the author of the best question for Andrei - the question he chooses himself and will answer it during the course. Send your questions to: asyncio2016@ptsecurity.com .