Geekly Articles each Day
📜 ⬆️ ⬇️

Automatic configuration of virtual machines in the clouds using metadata


In God we trust, the rest we automate
- unknown DevOps Engineer

The use of virtualization and cloud platforms allows to reduce the time spent on launching and maintaining the IT infrastructure tenfold. One person can manipulate tens, hundreds and even thousands of virtual servers, easily start, stop, clone them, change the hardware configuration and create ready-made images of systems based on them. If all your servers have the same configuration, then there are no special problems; you can manually configure the server once, make an image based on it and run as many machines as you need. If you have a large number of different operating systems with a different set of software, or if you need to quickly start and stop complex cluster configurations, then servicing even a few dozen of such servers will take a very long time. You can, of course, have a set of different scripts and images for all occasions that need to be maintained and updated, but it is more efficient to use one script and several images, and transfer all the necessary parameters at system start. Many cloud computing platforms offer a so-called metadata mechanism (metadata) or user data (user-data), using this mechanism, you can transfer to the script all the necessary data on setting up a specific virtual machine, or even transfer the script itself so that it runs when start up



To some extent, the following cloud platforms will be considered in this article:


1. Overview of how user data works for different platforms and examples of their use through CLI or simple scripts

1.1 Amazon EC2
')
In Amzon, the user-data parameter can be set in free form when the virtual machine is started and then it can be obtained via a specific link:

curl 169.254.169.254/latest/user-data

The IP address 169.254.169.254 is virtual and all requests to it are redirected to the internal API EC2 of the service in accordance with the IP address of the source.

Standard system images provided by Amazon have a built-in ability to execute Bash and Power Shell scripts transmitted via user-data. If user-data starts with shebang (#!), Then the system will try to execute the script using the interpreter specified in it. Initially, this feature was implemented in a separate package of cloud init for Ubuntu, but now it is included in all standard system images, including Windows.

For Windows systems, you can specify how to execute normal console commands,

<script> netsh advfirewall set allprofiles state off </script>


and the code on the Power Shell:

 <powershell> $source = "http://www.example.com/myserverconfig.xml" $destination = "c:\myapp\myserverconfig.xml" $wc = New-Object System.Net.WebClient $wc.DownloadFile($source, $destination) </powershell>


This functionality can be used in conjunction with Cloud Formation templates and run entire server stacks by specifying the necessary user-data:

 { "AWSTemplateFormatVersion" : "2010-09-09", "Parameters" : { "AvailabilityZone" : { "Description" : "Name of an availability zone to create instance", "Default" : "us-east-1c", "Type" : "String" }, "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Default" : "test", "Type" : "String" }, "InstanceSecurityGroup" : { "Description" : "Name of an existing security group", "Default" : "default", "Type" : "String" } }, "Resources" : { "autoconftest" : { "Type" : "AWS::EC2::Instance", "Properties" : { "AvailabilityZone" : { "Ref" : "AvailabilityZone" }, "KeyName" : { "Ref" : "KeyName" }, "SecurityGroups" : [{ "Ref" : "InstanceSecurityGroup" }], "ImageId" : "ami-31308xxx", "InstanceType" : "t1.micro", "UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[ "#!/bin/bash","\n", "instanceTag=WebServer","\n", "confDir=/etc/myconfig","\n", "mkdir $confDir","\n", "touch $confDir/$instanceTag","\n", "IPADDR=$(ifconfig eth0 | grep inet | awk '{print $2}' | cut -d ':' -f 2)","\n", "echo $IPADDR myhostname","\n", "hostname myhostname","\n" ]] } } }


If the launch of the script at the start of the machine does not suit you, for example, you want other people to use your images and they do not want to understand your code, then you can install your own script in the system, add it to the autoload, and create an image of the system. And provide image users with a description of possible parameters that can be set in user-data. For example, the parameter list key = value separated by semicolons:

graylogserver = "192.168.1.1"; chefnodename = "chef_node_name1"; chefattributes = "recipe1.attribute1 = value1, recipe1.attribute2 = value2, customparameter1 = value1"; chefserver = "192.168.1.38 Com000"; chefrole = "apache, mysq , php ";

You can get the whole Bash line like this:
 function get_userdata { user_data=$(curl -w "%{http_code}" -s http://169.254.169.254/latest/user-data) result_code=${user_data:(-3)} if [ -z "$user_data" ] || [ $result_code != "200" ] then echo "$CurrentDate: Couldn't receive user-data. Result code: $result_code" return 1 else export user_data=${user_data%%$result_code} return 0 fi }


and then, from the resulting list, get the desired value:
 function get_userdata_value { IFS=';' for user_data_list in $user_data do user_data_name=${user_data_list%%=*} if [ $user_data_name = $1 ] then user_data_value=${user_data_list#*=} user_data_value=$(echo $user_data_value | tr -d '\"') return 0 fi done return 1 }


After that, you can continue to configure the system in accordance with the data. It is not necessary to store all the scripts inside the image, it is enough to have a simple startup script that reads user-data and then downloads and runs all the necessary or transfers control to Chef or Puppet.

Similar functionality can be implemented on the Power Shell.

1.2 Eucaliptus
This product is compatible with Amazon AWS, and the user-data engine is implemented in the same way.

1.3 Nimbula
This product is relatively young, but fast-growing, it is designed to create private cloud systems and uses KVM virtualization. Its founders come from Amazon and have declared compatibility with Amazon, but despite this, compatibility is not complete. They have support for the user-data mechanism via virtual IP, but they are specified as key = value.
A list of all keys can be obtained by reference:
192.0.0.192/latest/attributes or 169.254.169.254/latest/attributes

example:
curl 169.254.169.254/latest/attributes
nimbula_compressed_size
nimbula_decompressed_size
chefserver
hostname


Get the value of a concreting key:
curl 169.254.169.254/latest/attributes/chefserver
192.168.1.45:4000

Thus, it is impossible to transfer the whole script for execution via user-data, it is necessary to create your own image of the system with a built-in start script.

Bash code example:
 function get_value { user_data_value=$(curl curl -w "%{http_code}" -s http://169.254.169.254/latest/attributes/"$1") result_code=${user_data_value:(-3)} if [ -z "$user_data_value" ] || [ $result_code != "200" ] then echo "$CurrentDate: $1 variable is not set, skip it, return code: $result_code" >> $LogFile return 1 else user_data_value=${user_data_value%%$result_code} return 0 fi }


1.4 VMWare vCloud Director
Starting with version 1.5, vCloud Director introduced a mechanism for using metadata within the framework of vApp (container for virtual machines). The data is given in the key = value format. To set metadata, you need to create XML with their description:

 <Metadata xmlns="http://www.vmware.com/vcloud/v1.5"> <MetadataEntry> <Key>app-owner</Key> <Value>Foo Bar</Value> </MetadataEntry> <MetadataEntry> <Key>app-owner-contact</Key> <Value>415-123-4567</Value> </MetadataEntry> <MetadataEntry> <Key>system-owner</Key> <Value>John Doe</Value> </MetadataEntry> </Metadata>


And then perform a POST request on the URL for the corresponding vApp:
$ curl -i -k -H "Accept: application / * + xml; version = 1.5" -H "x-vcloud-authorization: jmw43CwPAKdQS7t / EWd0HsP0 + 9 / QFyd / 2k / USs8uZtY =" -H "Content-Type: application / vnd.vmware.vcloud.metadata + xml ”-X POST 10.20.181.101/api/vApp/vapp-1468a37d-4ede-4cac-9385-627678b0b59f/metadata -d @ metadata-request.

Read all the metadata can GET request:
$ curl -i -k -H "application / * + xml; version = 1.5" -H "x-vcloud-authorization: jmw43CwPAKdQS7t / EWd0HsP0 + 9 / QFyd / 2k / USs8uZtY =" -X GET 10.20.181.101/api/ vApp / vapp-1468a37d-4ede-4cac-9385-627678b0b59f / metadata .

In order to read the value of a specific key, the request must be of the form:
$ curl -i -k -H "application / * + xml; version = 1.5" -H "x-vcloud-authorization: jmw43CwPAKdQS7t / EWd0HsP0 + 9 / QFyd / 2k / USs8uZtY =" -X GET 10.20.181.101/api/ vApp / vapp-1468a37d-4ede-4cac-9385-627678b0b59f / metadata / asset-tag

The answer is in the form of XML.

Learn more about how metadata works in vCloud here : blogs vmware

2. Work with user data using control systems such as: Chef and Puppet

2.1 Chef
The choice of how the Chef client gets to the machine is yours, you can install it manually and then create your own system images, or you can install it automatically when the system starts. Both methods have their advantages and disadvantages: the first method reduces the time spent on the configuration of the machine during the start, the second method allows you to always install the latest version of the client or install the version of the client you absolutely need, depending on your needs. In any case, we must transfer the list of roles and recipes that must be executed on the machine; we can get this list via used-data and configure the Chef client at system startup. Also, if we install and configure the client at system startup, we need to download the validation.pem key for the corresponding Chef server (which data can also be transferred via user-data)

An example of a bash script that gets a list of roles:

 rolefile="/etc/chef/role.json" function get_role { get_value "chefrole" if [ $? = 0 ] then chefrole=$user_data_value else echo "$CurrentDate: Couldn't get any Chef role, use base role only." chefrole="base" fi commas_string=${chefrole//[!,]/} commas_count=${#commas_string} echo '{ "run_list": [ ' > $rolefile IFS="," for line in $ep_chefrole do if [ $commas_count = 0 ] then echo "\"role[$line]\" " >> $rolefile else echo "\"role[$line]\", " >> $rolefile fi commas_count=$(($commas_count-1)) done echo ' ] }' >> $rolefile }


And then creates a configuration file for the client:
 function set_chef { if [ -d $chef_dir ] && [ -e $chef_bin ] then service $chef_service stop sleep 10 echo -e "chef_server_url \"http://$1\"" > $chef_dir/client.rb echo -e "log_location \"$chef_log\"" >> $chef_dir/client.rb echo -e "json_attribs \"$rolefile\"" >> $chef_dir/client.rb echo -e "interval $chef_interval" >> $chef_dir/client.rb echo "$CurrentDate: Writing $chef_dir/client.rb" service $chef_service start else echo "$CurrentDate: Chef directory $chef_dir or chef binary $chef_bin does not exist. Exit." exit 1 fi }


The json_attributes parameter sets the path to the JSON file with a list of roles and recipes.

After we have handed over the control to Chef to the client, he will register on the server, download the list of recipes and start their execution, but there are a few nuances :


To solve the first problem, there is an Opscode cookbook called chef_handler. It provides a mechanism called Exception and Report Handlers, which is called after the Chef client has finished executing the recipes. Using this cookbook, we can check the result of the last client run and perform any actions. You can send an email about the execution results ( an example described in the Opscode documentation ) or write the execution result to the Chef server to check this value with your applications and display the execution status.

Sample recipe:

set attribute values ​​to default
 default['lastrun']['state'] = "unknown" default['lastrun']['backtrace'] = "none"


indicate that you need to perform
 include_recipe "chef_handler" chef_handler "NodeReportHandler::LastRun" do source "#{node.chef_handler.handler_path}/nodereport.rb" action :nothing end.run_action(:enable)


what we are doing
 module NodeReportHandler class LastRun < Chef::Handler def report if success? then node.override[:lastrun][:state] = "successful" node.override[:lastrun][:backtrace] = "none" else node.override[:lastrun][:state] = "failed" node.override[:lastrun][:backtrace] = "#{run_status.formatted_exception}" end node.save end end end


As a result, we initially set the values ​​of the lastrun.state and lastrun.backtrace attributes to both 'unknown' and 'none' and then, according to the results of the client’s execution, we get either the record 'successfull' or 'failed' with a description of the error in lastrun. backtrace.
This recipe should be on the run list first to cover errors when executing any recipes.

In order to change the default attributes, we have to get them somehow, then save them and then start performing recipes. We can get them again, through user-data.

The recipe for getting user-data, on the example of Amazon:
we get the whole line
 # Get whole user-data string def GetUserData(url) uri = URI.parse(url) http = Net::HTTP.new(uri.host, uri.port) http.open_timeout = 5 http.read_timeout = 5 proto = url.split(":", 2) if proto[0] == "https" http.use_ssl = true http.verify_mode = OpenSSL::SSL::VERIFY_NONE end request = Net::HTTP::Get.new(uri.request_uri) begin http.open_timeout = 5 http.read_timeout = 5 request = Net::HTTP::Get.new(uri.request_uri) response = http.request(request) result = response.body.to_s if response.is_a?(Net::HTTPSuccess) Chef::Log.info("Successfuly get user-data.") return result else return false end rescue Exception => e Chef::Log.info("HTTP request failed.") return false end end


we get the value of a specific parameter
 # Get specified user-data value def GetValue(user_data,attribute) user_data.split(";").each do |i| attribute_name=i.split("=", 2) if attribute_name[0] == attribute return attribute_name[1].strip end end return false end

Now, when we can get the value for a specific parameter from the transmitted data, we can set them:
hefnodename = "chef_node_name1"; chefattributes = "recipe1.attribute1 = value1, recipe1.attribute2 = value2, customparameter1 = value1"; chefserver = "192.168.1.38 Tet000"; chefrole = "apache, mysql, php"

In the chefattributes parameter we passed a list of attributes that we want to change, they are specified in the format “cookbookname.attributename = value”. If we want to change the default port for Apache, we need to set chefattributes = apache.port = 8080.

A recipe that reads this value and saves it:
 chefattributes = GetValue("#{node[:user_data]}","chefattributes") if chefattributes != false hefattributes.split(",").each do |i| attribute_name=i.split("=") recipe_name=attribute_name[0].split(".", 2) node.override[:"#{recipe_name[0]}"][:"#{recipe_name[1].strip}"]="#{attribute_name[1].strip}" Chef::Log.info("Save node attributes.") node.save else Chef::Log.info("Couldn't get Chef attributes. Skip.") end

This recipe must be performed before performing other recipes.

The disadvantages of the above recipes. The node.save operation sends to the server to save the entire JSON array for a particular node, including information collected by Ohai. If you have thousands of machines and all of them will constantly try to overwrite their attributes on the server, this may have a bad effect on its performance. The same applies to the use of flexible and powerful search provided by Chef, the search operation is very time consuming and in the case of servicing thousands of machines, this will create a heavy load on the server. In this case, you need to use other methods that will not be described here.

2.2 Puppet
Using Puppet to get user-data is similar to using Chef. The address of the Puppet server and other necessary data for configuring the agent are obtained using the start script. To transfer your facts to the server it is convenient to use the Facter add-on.

Here is an example of a ruby ​​script that receives the necessary data from user-data and sends it to the server as additional facts for this machine:

 require 'facter' user_data = `curl http://169.254.169.254/latest/user-data` user_data = user_data.split(";") user_data.each do |line| user_data_key_value = line.split('=', 2) user_data_key = user_data_key_value[0] user_data_value = user_data_key_value[1] Facter.add(user_data_key) do setcode { user_data_value } end end instance_id = `curl http://169.254.169.254/latest/meta-data/instance-id` Facter.add('instance-id') do setcode { instance_id } end


Of course, it may seem that this is all simple, and there is no need to fiddle around with any complex schemes, you can create the necessary set of images with the pre-installed software, and carry out all the changes with your own scripts that run on SSH and stink changes to the configuration files. This article describes the elementary steps. If we need to start a Hadoop cluster, MySQL or a cluster from Front-End, Back-End, App, DB servers, so that all machines are automatically configured and that you can dynamically delete or add an arbitrary number of machines to the cluster during automatic scaling, without the techniques described above. not enough.

If you know how to transfer metadata for other cloud platforms and know other ways to manage the configuration of a virtual machine at startup, let's discuss in the comments.

Source: https://habr.com/ru/post/166639/

More articles:


All Articles