Chef for everyday tasks

Every day, the system administrator is confronted with tasks that somehow have to be solved by a banal set of commands. Sometimes it comes to ridiculous:

• distribute the file on 100 servers
• distribute the package on 100 servers
• change line in file
• upgrade system
• add user
• restart services

etc., the infrastructure administrator makes his hands, alternately going to all the servers and executing a set of 1-10 commands. Continuing to work this way, soon, the system administrator of a large system turns into a “server enikeyschik”.
And there are two ways to solve this problem: to hire a junior employee and “unload” all the dirty work for him, or to automate simple and not so much tasks.

At the moment there are many systems that allow you to do this, but the most popular are Chef , Puppet and Ansible .
This publication focuses on Chef and how to use it to automate everyday tasks on multiple servers.

Generally why chef?

Firstly, our customer uses it. So, in order not to fence the zoo from the Deployers, Chef left. Somewhere on Habré saw a good description of the differences between Chef and Puppet:

• Chef - what do you want to get?
• Puppet - what do you want to do?

But after watching the video seminar “The Road to the Clouds”, where Mikhail Shcherbakov from Mirantis talked about “Non-standard use of Puppet in deployment”, I somehow did not want to follow this ideology clearly. Yes, and testing the chef-recipes that we made for deploying projects to the customer, suggested that you can use Chef as you like.

Secondly, the language of the chef is closer to me, since not long ago, I became acquainted with ruby, and the syntax of Chef’s manifests is what it uses.
')
Thirdly, chef-recipes can be distributed from the server centrally by simply calling " knife bootstrap --run-list" recipe [somerecipe] "somehost, domain.lan ". This is really convenient.

Well and fourthly, the chef has a normal client under Windows. And sometimes without it anywhere.

Installation

I will continue to use Ubuntu as an example. Only the server will be on Ubuntu, the clients will be on Ubuntu, CentOS and Windows.
Take the installation package here .

# cd ~; wget https://web-dl.packagecloud.io/chef/stable/packages/ubuntu/trusty/chef-server-core_12.1.2-1_amd64.deb # dpkg -i chef-server-core_12.1.2-1_amd64.deb 

After installing the package, you should run the “reconfiguration”, create a user and organization.

 # chef-server-ctl reconfigure # chef-server-ctl user-create grey Sergey K grey@mydomain.lan superpassword123 --filename /etc/chef/grey.pem # chef-server-ctl org-create vst "VST Consulting, Inc." --association_user grey --filename /etc/chef/vst-validator.pem 

The mechanism of organizations allows you to organize a very convenient test site or to unite all customers on one server (oh, I do not advise).
There is information that I have described, plus information on how to install additional components, for example, a web interface.

It is also worth tweaking some configs and creating the necessary directories:

 # cat /etc/chef/knife.rb current_dir = File.dirname(__FILE__) log_level :info log_location STDOUT node_name "grey" client_key "#{current_dir}/grey.pem" validation_client_name "vst-validator" validation_key "#{current_dir}/vst-validator.pem" chef_server_url "https://chef/organizations/vst" cookbook_path ['/root/chef-repo/cookbooks'] # ln -s /opt/opscode/embedded/bin/knife /usr/bin/knife # ln -s /etc/chef /root/.chef # git clone https://github.com/opscode/chef-repo.git # knife ssl fetch && knife ssl check 

Simple Kukbook

The most popular tool in Chef is cookbook `s. It contains certain recipes.
A recipe is a template for performing certain actions on a server with file templates, variables, etc. for the realization of a certain working result. As in cooking, our goal is to get a “soup” that is ready to eat.

As an example, we will write a recipe for preparing some self-written daemon application in java, which, together with the configs, will need to be distributed to a number of servers.

Create a blank:

 # cd /root/chef-repo/cookbooks # knife cookbook create prog # cd prog; ls attributes CHANGELOG.md definitions files libraries metadata.rb providers README.md recipes resources templates 

In our project, we will use only the directories attributes, recipes, templates , as well as the file metadata.rb .
Let's start from the end. The metadata.rb file contains basic information about the recipe, its developer, its license and version, as well as dependencies and a list of supported OS.


In our example, the dependence on the java recipe is indicated in order to avoid problems with its absence. There are ready-made recipes, including java.

Further, in the recipes directory we see the file default.rb . We result in the following form:


In order:

• node.override ['java'] ['jdk_version'] = '7' means that for the java cookbook, we change the default value of the jdk_version attribute to '7'. About the attributes will be discussed later.
• include_recipe "java" says that we must start the recipe default.rb from the java kookbook first. Further preparation according to the recipe will be only after successful preparation according to the recipe java.
• Using the case construction and the standard attribute platform_family, we define the platform to be installed and call the corresponding recipe from the current cookbook. By and large, in those recipes you can make all the manipulations with the installation and launch, but since Most of the actions will be identical, we will there bring some parameters to the required form for a specific platform.
• package_name = node ['prog'] ['package'] says that we declare a variable, which we will use in the future. This variable contains the value of the attribute 'package'
• The remote_file construction allows us to load a file from an open source and save it in a directory with a cache.
• service design says that our service will be in autoload and started.
• The template construction saves the configuration template to the required directory and restarts the service.

In fact, I describe a universal recipe, with which you can install almost any package in almost any system.

Next, we need to fix the attributes. Attributes are a convenient mechanism in the cookbooks, which allows you to set as many parameters as you like from the outside. In our example, 4 attributes will be used: package, conf_dir, prog_name, cluster .
We give attributes / default.rb to the following form:

 default['prog']['prog_name'] = 'somejavademon' default['prog']['package'] = "#{node['prog']['prog_name']}.deb" case node['platform_family'] when 'debian' default['prog']['conf_dir'] = "/etc/#{node['prog']['prog_name']}/conf" when 'rhel' default['prog']['conf_dir'] = "/etc/#{node['prog']['prog_name']}/conf" when 'windows' default['prog']['conf_dir'] = "C:/#{node['prog']['prog_name']}/conf" else Chef::Application.fatal!('Attempted to install on an unsupported platform') end 

Thus, we set the basic attribute values, while taking into account that for different systems, the location of the configuration files will be in different places.

Now, we need to create a configuration file template:


I think after the foregoing it is clear how you can substitute the necessary attributes into the template. You can even use cycles, but I will not focus on this.

I did not mention, but additional recipes for templates should also be created:

 # cat recipes/deb node.override['prog']['package'] = "#{node['prog']['prog_name']}.deb" # cat recipes/rhel node.override['prog']['package'] = "#{node['prog']['prog_name']}.rpm" # cat recipes/windows node.override['prog']['package'] = "#{node['prog']['prog_name']}.msi" 

Yes, this is so simple.

Upload our recipes to the server:

 # knife cookbook upload --all 

Environment

Now we will create an example of a simple environment from which our parameters will be taken.
This is not necessary, but it will be very useful to understand how it works.

 # export EDITOR=nano # knife environment create someserver -d "Environment for group of servers where somejavademon will work." 

A window of the specified EDITOR will open in which we will enter the following information:

 { "name": "someserver", "description": "Environment for group of servers where somejavademon will work.", "cookbook_versions": { }, "json_class": "Chef::Environment", "chef_type": "environment", "default_attributes": { }, "override_attributes": { "prog": { "prog_name": "somejavaserver", } } } 

Now this environment can be used on a number of hosts, where it will be installed exactly somejavaserver .

DEPLOY

Now we come to the most important thing: the distribution of our product across servers.

• We have a recipe, which indicates what, how and where to install.
• We have an Env which indicates what specifically needs to be installed.
• We have a working server Chef, on which it all lies and it has access to all the necessary servers.
• We have 100 servers with a working domainname on Ubuntu, CentOS, on which all this needs to be installed.
• We have root / admin access via ssh to all necessary servers.

Now, to install the recipe we need on the server, we need to run:

 # knife bootstrap --run-list "recipe[prog]" server1.domain.lan -E someserver -P rootpasswordfromnode 

Nothing prevents us from writing a script:

 #!/bin/bash for i in {1..100}; do knife bootstrap --run-list "recipe[prog]" server$i.domain.lan -E someserver -P rootpasswordfromnode done 

After some time (5 minutes / half hour / hour / day) all 100 servers will be in service in full readiness.
In any case, installing a Chef server, writing a recipe and env, as well as deploy, will take less or as much time as manually installing the package on each server. BUT! If everything is ready with us, then re-installation will take much less time and even more effort. The most important thing is that you do not need to do anything manually. Just specify the list of servers, run the script and go read the new publications on Habré.

What else can you do?

This is quite a philosophical question.
You can (and should) install knife-windows . It's not difficult there, but if you have questions, I'm ready to help.
You can download a bunch of ready-made recipes from the supermarket and deploy everything you want for the desired number of servers.
You can write your own recipes for all occasions and reduce server maintenance to installing the OS (which is also quite well automated using PXE), equipment replacement and monitoring.
You can connect Chef to OpenStack (if there is one) and deploy new servers and services in batches. Here you can find out how to do it. There is nothing difficult, especially if you have already dealt with Cloud Init and Chef.

If I forgot to add something, then correct me, please, in the comments.

I hope this publication will help someone to deal with Chef and routine work in the server or office.

UPD: I decided to add an example cloud-config for OpenStack: