How to manage virtual machines if there are many

After we had a few more projects released, and the number of tickets in the tracker on the topic “create user, deploy virtual machine, give access” exceeded all conceivable limits, there was a need to change something.
Task: to organize linux work environment for several development and testing teams. The total number of virtual machines - three or four dozen.




At once I will make a reservation that the article is not a detailed step-by-step tutorial “how and how it should be”, but only a description of practical experience and a brief overview of the tools we use. It is understood that the reader will be able to solve the problems that arise, such as installing the necessary dependencies, packages, network configuration, and so on. We deliberately do not attach screenshots and detailed configuration files and leave the reader the opportunity to independently investigate the listed tools.
')
It is necessary:

• automate the deployment of virtual machines on specific patterns, reduce deployment time,
• simplify user authentication
• provide fast configuration of services on each virtual machine,
• provide the administrator with a single center for managing the entire farm and save it from the routine editing of multiple configuration files of the same type,
• provide real-time monitoring of the state of the system, both in general and for each virtual machine.

Requirements for tools: open-source, easy setup and use, web interface, or a powerful package of utilities for customization.

Solutions used: ubuntu 10.04 and other versions, proxmox openvz, chef, openldap, git, zabbix

1. Authentication.

Expand openldap. Recently, the openldap package has switched to a dynamic configuration scheme, so all configuration is done using special ldif files. I do not know whether this is good or bad, it is unusual and not always trivial at least. For example, you can use this article . For the same article, if necessary, you can configure and LDAP replication.
We did not configure the LDAP frontend; then LAM web control panel (v3.4.0) did it for us. We have not found a more convenient panel for LDAP. GOSA has a little more functionality, but much more difficult to set up, the other products frankly do not fall short in appearance and functionality.

Since we wanted to simplify the authentication procedure for developers and testers, it was decided to collect public keys from them as well as passwords and put them into LDAP.
This scheme will allow us to perform authentication without passwords using SSH keys. An article about keys and general SSH configuration is here , only instead of the authorized_keys file we will use information from LDAP.

To do this, you need to collect openssh with the LPK patch (see below) and connect the openssh-lpk_openldap.schema scheme in LDAP
root:/#cat lpk.ldif
dn: cn=openssh-lpk,cn=schema,cn=config
objectClass: olcSchemaConfig
cn: openssh-lpk
olcAttributeTypes: {0}( 1.3.6.1.4.1.24552.500.1.1.1.13 NAME 'sshPublicKey' DES
C 'MANDATORY: OpenSSH Public key' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.
1.1466.115.121.1.40 )
olcObjectClasses: {0}( 1.3.6.1.4.1.24552.500.1.1.2.0 NAME 'ldapPublicKey' DESC
'MANDATORY: OpenSSH LPK objectclass' SUP top AUXILIARY MAY ( sshPublicKey $
uid ) )
root:/#ldapadd -Y EXTERNAL -H ldapi:/// -f lpk.ldif

Configuring the LAM web interface is trivial and does not require special attention. The panel after installation is able to configure LDAP frontend itself. In the panel in the settings you must also connect the module SSH Public Key (ldapPublicKey).

2. SSH

From the repository, he does not know how to take public keys from LDAP, we will teach him.
Read about the technology LPK and take the patch here
root:/#apt-get source ssh
root:/#cd openssh-5.3p1
root:/#patch -p1 < contrib-openssh-lpk-5.4p1-0.3.13.patch
Everything that has been rejected, and we got two files and two lines there, patch it manually.
In debian / rules you need to add in two places to ../configure
--with-libs="-lldap" --with-ldflags="-L/usr/lib" --with-cppflags="-I/usr/include -DWITH_LDAP_PUBKEY"

In order for this package to successfully get onto the working system and not conflict with the working version from the repository, there are several ways, for example:

• upgrade version of the package
• fix debian / control and debian / rules, name the package with some other name and try to put it all together.

The easiest way is to upgrade the version. The package will rise without problems, but in the future there will be problems with the update. Since our infrastructure is closed outside, the lifetime of virtual machines is relatively small, security problems in this environment are not particularly important for us, we chose this path. For the production environment this method, of course, is not suitable.

Make changes to the changelog and build the package.
root:/#dch -i
root:/#dpkg-buildpackage -b

At the output, we get a certain 'openssh.deb', which is able to query the LDAP for 'sshPublicKey' if certain conditions are met.
User record in LDAP

• with objectclass 'ldapPublicKey'
• c objectclass 'posixAccount'
• with the sshPublicKey attribute filled

Group record in LDAP

• c objectclass 'posixGroup'
• with the attribute 'cn' and the name of the group in it
• with the attribute 'memberUid', where uids belonging to this group will be listed.

Required settings in / etc / ssh / sshd_config
UseLPK yes
LpkServers ldap://10.10.10.10/ # LDAP
LpkUserDN ou=People,dc=office #
LpkGroupDN ou=group,dc=office #
LpkBindDN cn=admin,dc=office # ,
LpkBindPw secret #
LpkServerGroup developers #
LpkForceTLS no # TLS, .
LpkSearchTimelimit 3 #
LpkBindTimelimit 3 #
LpkPubKeyAttr sshPublicKey # LDAP
Depending on your LDAP settings, you can simply comment out unnecessary parameters.

3. Deploy configurations.

Since the entire virtual machine fleet is virtually the same, we used the chef configuration system from the opscode repository .
The system is very powerful and flexible, it allows you to automate the installation of almost everything that you can think of.

An example of a simple recipe for configuring Postgres + Postgis
# Cookbook Name:: postgres
# Recipe:: default
package "postgresql" do
action:install
options "--force-yes"
end

package "postgresql-contrib" do
action:install
options "--force-yes"
end

package "postgis" do
action:install
options "--force-yes"
end

package "postgresql-9.0-postgis" do
action:install
options "--force-yes"
end

script "install_postgis" do
interpreter "bash"
user "postgres"
cwd "/tmp"
code <<-EOH
createdb template_postgis
createlang plpgsql template_postgis
psql -d template_postgis -f /usr/share/postgresql/9.0/contrib/_int.sql
psql -d template_postgis -f /usr/share/postgresql/9.0/contrib/postgis-1.5/postgis.sql
psql -d template_postgis -f /usr/share/postgresql/9.0/contrib/postgis-1.5/spatial_ref_sys.sql
createuser -s pgsql
EOH
end

Configure LDAP on the client. Here we simply decompose pre-configured configuration files to the client machine so that the client can log in to our LDAP.
# Cookbook Name:: openldap
# Recipe:: auth

package "nscd" do
action :upgrade
end

package "libnss-ldap" do
action :upgrade
end

package "libpam-ldap" do
action :upgrade
end

service "nscd" do
supports :status => true, :restart => true, :reload => true
action [ :restart ]
end

service "ssh" do
supports :status => true, :restart => true, :reload => true
action [ :restart ]
end

script "prepare dirs" do
interpreter "bash"
user "root"
cwd "/tmp"
code <<-EOH
mkdir -p /etc/ldap
EOH
end

cookbook_file "/etc/libnss-ldap.conf" do
source "libnss-ldap.conf"
mode 0644
owner "root"
group "root"
end

cookbook_file "/etc/pam_ldap.conf" do
source "pam_ldap.conf"
mode 0644
owner "root"
group "root"
end

cookbook_file "/etc/nsswitch.conf" do
source "nsswitch.conf"
mode 0644
owner "root"
group "root"
notifies :restart, resources(:service => "nscd"), :immediately
end

%w{ account auth password session }.each do |pam|
cookbook_file "/etc/pam.d/common-#{pam}" do
source "common-#{pam}"
mode 0644
owner "root"
group "root"
notifies :restart, resources(:service => "ssh"), :delayed
end
end

Using chef, we also install on each virtual machine the already configured zabbix-agentd and a set of ZTC scripts to it. After starting the virtual zabbix by autodiscovery, add it to the pool and immediately append the necessary templates. The zabbix control panel is also attached to LDAP. The only thing that had to be done here was an automatic synchronization script for LDAP and zabbix users, monitoring does not know how to create users by itself.

Since public ssh keys are now stored in a centralized repository, this scheme also allowed users to integrate GIT process seamlessly and transparently.

Proxmox control panel, pre-configured templates for openvz, LVM storage, allowed us to simplify and speed up the deployment of new virtual machines as much as possible. We control all allocated resources, under load everything behaves predictably and stably. The cost of implementing and maintaining such a virtual machine is extremely low compared to other virtualization systems.

Let me remind you that this is a test and development environment, completely closed to the outside world, without critical data and without the need for backups. Therefore, this scheme misses many very important points in terms of security and fault tolerance. For production in this scheme, you will need to think about certificates, TLS, SSL, password policy, various PAM security mechanisms, sudo, as well as firewall rules and group policies.

Some more links:
LDAP for internet project
Chef or how to manage thousands of servers
Linux Virtualization with OpenVZ
Zabbix Universal Monitoring System - Introduction