Cluster that is always with you

Wanted strange.
First, pile up a bunch of virtual machines directly to your laptop.
And secondly, to smoke one virtualization inside another.

We will talk about the use of containers LXC , and inside another virtual machine.

WTF! Why the hell is it necessary?

First of all, to experiment with various tools:

• Distributed databases, file systems, parallel computing, etc.
• Infrastructure management systems (such as Chef, Puppet, Fabric, etc.)
• Testing and assembly in different environments
• Your option (write in the comments)

In this case, I want to have access to this laboratory always, regardless of the availability of the Internet. Well, why not smoke some Hadoop somewhere on the road? :-)
')
I think this post will be useful both for those who have Windows or Mac OS X on the desktop, and those who have Linux (the LXC part).

Task

Run on an average laptop 10-20 virtual machines that
- can go to the Internet (via NAT)
- see each other and the host computer (that is, our laptop)
- accessible from the host computer (that is, you can access any of these virtual machines via ssh)

Decision

1. On a computer (with Windows or Mac OS X) run VirtualBox
2. Install Linux on it
3. Inside Linux, make a bunch of LXC containers with independent Linux

Actually, this could put an end.
Below is just a step-by-step cheat sheet (Ubuntu was used) how to set it all up quickly.

---

1. Install VirtualBox and system on a virtual machine

There is nothing special to describe here.
After installing the system - do not forget to put Guest Additions.

2. Configure network interfaces in VirtualBox

A virtual machine should have two network interfaces:

• NAT connection type (it is created by default). Through it, our virtual machines will be able to go to the Internet (for example, to download packages).
• The type of connection is Host-only networking (in Russian it is called “Virtual Host Adapter”).

In order to add a Host-only interface, first go to the general Virtualbox settings and add the Host-only adapter on the host machine. Then we add the second interface to the virtual machine configuration.

Important! You must enable "Promiscuous mode" on the Host-only interface. This will allow the containers to see the host machine and each other.

3. Configure network interfaces

In the guest system, you need to configure bridge interfaces. We will need them to operate the network in containers. To do this, you must install the bridge-utils package ( # apt-get install bridge-utils ) and make changes to the / etc / network / interfaces file (see man bridge-utils-interfaces).
You should have something similar:

It was	It became
auto eth0 iface eth0 inet dhcp	auto br0 iface br0 inet dhcp bridge_ports eth0 bridge_fd 0 auto br1 iface br1 inet static address 192.168.56.2 netmask 255.255.255.0 bridge_ports eth1 bridge_fd 0

Let the IP address on br0 (eth0) be assigned by VirtualBox itself via DHCP, we don’t need to know this address. And on br1 (eth1) we will assign the IP address with our hands - this is more convenient, then we will go to it via ssh from the host machine.

We restart the virtual machine to make sure that both interfaces inside the virtual machine go up, the virtual machine itself is available at 192.168.56.2 (via the Host-only interface) and the Internet is accessible inside it (via the NAT interface).

4. Mount the cgroup file system

LXC containers require the cgroup utility file system. The mount point is not important - you can mount anywhere.

Add a line to / etc / fstab :

 cgroup /var/local/cgroup cgroup defaults 0 0 

and mount

 mkdir /var/local/cgroup mount cgroup 

In Ubuntu 11.10 (oneiric), cgroup need not be mounted. The lxc package depends on the cgroup-lite package, which mounts the cgroup in / sys / fs / cgroup /

5. Install packages for working with LXC

 apt-get install lxc apt-get install debootstrap 

The lxc package contains management utilities and scripts for creating containers. The debootstrap package is a utility that downloads the right packages and deploys the minimal basic system (ubuntu or debian). In addition, there is the febootstrap package - it downloads and deploys Fedora.

The fastest way to learn lxc : type lxc- and press " Tab " twice

6. Create the first LXC container

In the / usr / lib / lxc / templates / directory there are files like lxc-debian , lxc-natty , lxc-oneiric , lxc-fedora , etc. These are the so-called "templates". In fact, these are scripts that create an appropriate working environment.

Create a container (with Ubuntu 11.04)

 lxc-create -n node01 -t natty 

Our container will appear in the / var / lib / lxc / node01 / directory .
In the created environment, the root user has the root password. Do not forget to change!

In Ubuntu 11.10 (oneiric) the lxc package is fresher: you can pass parameters to the " ubuntu " pattern, including the desired version of the distribution. To find out which parameters a template accepts, run lxc-create --template ubuntu --help

7. Configure the network in the container

7.1. Network interfaces

There are not enough network parameters in the container configuration file - they need to be added manually. Then you can use the template config or simply clone the container.

Open the file

 vi /var/lib/lxc/node01/config 

and add such lines (we put the necessary MAC-and IP-addresses):

 lxc.network.type = veth lxc.network.flags = up lxc.network.link = br0 <-    lxc.network.name = eth0 <-     lxc.network.hwaddr = ac:de:48:00:00:01 lxc.network.ipv4 = 10.0.2.101/24 lxc.network.type = veth lxc.network.flags = up lxc.network.link = br1 lxc.network.name = eth1 lxc.network.hwaddr = ac:de:48:00:ff:01 lxc.network.ipv4 = 192.168.56.101/24 

Here 10.0.2.101/24 is a network that is commonly used for NAT interfaces in VirtualBox. Through this interface the container will go online.
192.168.56.101/24 - this is our Host-Only network. Through this interface, the container will communicate with the local network and other containers.

I recommend setting the MAC address and IP address manually. This is not necessary, but convenient. For example, when you need to listen to network traffic. I put the same last digit in all the numbers (node0 1 - 10.0.2.10 1 - ac: de: 48: 00: 00: 0 1 ).

Note that the configuration file / etc / network / interfaces inside the container does not need to be touched. Interfaces are enough to configure outside.

7.2. DNS in the container

Add something meaningful to /etc/resolv.conf to make the DNS work:

 echo "nameserver 8.8.8.8" > /var/lib/lxc/node01/rootfs/etc/resolv.conf 

For local address resolution you can do / etc / hosts . The / etc / hosts file can be made common to all containers if mounted with the bind option.

8. Run and check the container.

We start the container

 lxc-start --logfile /tmp/lxc-node01.log --logpriority DEBUG --name node01 

We go (root / root) and check that the network is accessible in all directions.

To run the container in the background, add the key " --daemon "

9. Clone containers

To clone containers, just copy it all.

 cp -a node01 node02 

and fix the configuration files (paths, MAC- and IP-addresses):

 vi node02/config vi node02/fstab vi node02/rootfs/etc/hostname 

Starting with version lxc 0.7.5 , the utility lxc-clone has appeared . It corrects the paths and hostname correctly, but you still need to edit the IP addresses with your hands.

10. Autorun containers

The lxc package includes the /etc/init.d/lxc script, which launches certain containers at system startup. In the / etc / default / lxc file, you need to list which containers to run. This script expects container configuration files to be in / etc / lxc and have the extension * .conf .
I just did some symlinks:

 /var/lib/lxc/nodeXX/config -> /etc/lxc/nodeXX.conf 

11. How else can you refine this kitchen

• Write scripts that automate the routine for creating containers (template configuration, setting IP addresses, creating users, configuring ssh using keys, etc.)
• Install apt-cacher-ng proxy to not download the same packages several times.
• Mount directories and files (using the bind option) so that some files are shared
• Put the containers under the control of libvirt (I have not tried. Perhaps this is a topic for a separate note)
• Try aufs to run “overlay” containers, when unchangeable files can be common to all containers (as well as the lxc-start-ephemeral utility).