Geekly Articles each Day
📜 ⬆️ ⬇️

Virtual Private Cloud: Image Preparation

VPC Image Building

The Virtual Private Cloud service has a large set of ready-made operating system images for creating virtual machines.


However, many users need images that are missing from our service, for example, it may be a less common type or version of the required operating system.


Sometimes it becomes necessary to change the set of pre-installed packages or the system configuration files in the finished image — for example, for users who deploy a cluster of similar servers.


In order not to make the same setup every time after installing the server, you can prepare an image with the required changes and thus speed up the installation of a large number of similar virtual machines. In this article we will show how this is done.


As an example, the image of Ubuntu 16.04 will be used: preparing the working environment, setting the necessary parameters, building and uploading the image to the cloud service.


Also, the necessary steps to prepare the image with full compatibility with all the additional features of the Virtual Private Cloud service will be considered.


We will use diskimage-builder as a tool for building an image. This is a set of components for preparing images of operating systems, file systems, RAM disks with open source code, supported by the OpenStack community.


The tool supports image creation of most common GNU / Linux distributions:



By default, diskimage-builder prepares the image of the cloud version of the operating system, so the image will contain the cloud-init and cloud-utils packages necessary for automatic configuration of the system in the cloud.


Creating a Ubuntu 16.04 Image


We will prepare the image on a machine running Ubuntu 14.04.


First, install the necessary dependencies:


sudo apt update sudo apt -y install python-pip curl

Then install the diskimage-builder:


 sudo pip install diskimage-builder

Create the base directories for work:


 mkdir -p ~/diskimage-builder/{images,elements}

To set up diskimage-builder, we need to specify several additional parameters, the value of which is stored directly in the environment variables of the command shell (so you can specify them on the command line):



The image is assembled using the command:


 disk-image-create vm $BASE_ELEMENTS -t raw -o $IMAGE_PATH

After the -t switch, we specify the desired image format. Images are supported : qcow2, tar, vhd, docker, raw.


Before directly assembling an image, you will most likely need to make additional settings in the system, for example:



To do this, we need to create an additional element in the ~ / diskimage-builder / elements directory.


Create directories:


 mkdir -p ~/diskimage-builder/elements/ubuntu-16-custom/{install.d,post-install.d,finalise.d}

Inside the ~ / diskimage-builder / elements / ubuntu-16-custom directory, create a README.rst file with the description of the new element:


 ================ ubuntu-16-custom ================ Customize Ubuntu 16.04 (Xenial)

In the ~ / diskimage-builder / elements / ubuntu-16-custom / install.d directory create the script 50-install-additional-packages:


 #!/bin/bash packages="python2.7 python python-minimal" apt -y install $packages

The scripts in the install.d directory are executed when the image is assembled during the installation of the main packages. The 50-install-additional-packages script we created will install version 2.7 of the python system, which is still required for many applications to work. By default, only python version 3.5 is present in the Ubuntu 16.04 image.


You can also add to this list the necessary packages that you need to install in the image.


Next, in the ~ / diskimage-builder / elements / ubuntu-16-custom / post-install.d directory, you need to create a 50-configure-system script to modify the system configuration:


 #!/bin/bash # Permit password auth via SSH echo 'ssh_pwauth: true' >> /etc/cloud/cloud.cfg.d/50_remote_access.cfg # Generate new keys in first boot echo 'ssh_deletekeys: true' >> /etc/cloud/cloud.cfg.d/50_remote_access.cfg # Don't disable root access via SSH echo 'disable_root: false' >> /etc/cloud/cloud.cfg.d/50_remote_access.cfg # Don't create "ubuntu" user echo 'users: []' >> /etc/cloud/cloud.cfg.d/50_remote_access.cfg # Change default timezone to MSK echo 'timezone: Europe/Moscow' >> /etc/cloud/cloud.cfg.d/50_timezone.cfg # Change PermitRootLogin value to "yes" sed -i 's/PermitRootLogin .*/PermitRootLogin yes/g' /etc/ssh/sshd_config # Set root password to empty passwd -d root

The scripts in the post-install.d directory are executed immediately after the scripts in the install.d directory.


We also need to change the GRUB settings to add the “net.ifnames = 0” startup parameter, which saves the interface names in the usual ethN format.


Add the 50-configure-grub script to the ~ / diskimage-builder / elements / ubuntu-16-custom / finalise.d directory:


 #!/bin/bash sed -i 's/\(^GRUB_CMDLINE_LINUX.*\)"$/\1 net.ifnames=0"/' /etc/default/grub update-grub

The scripts in the finalise.d directory are executed after all the main stages of preparing the image.


After creating the described scripts, we need to change their attributes, for this we execute the chmod command :


 chmod 0755 ~/diskimage-builder/elements/ubuntu-16-custom/*d/*

Now, before creating the image, we need to specify the path for our new element. This can be done using the ELEMENTS_PATH parameter.


In order not to specify any parameters directly on the command line each time, in the ~ / diskimage-builder / directory, create a script build-ubuntu-16.04, which will set diskimage-builder parameters and build the image:


 #!/bin/bash export ARCH="amd64" export BASE_ELEMENTS="bootloader cloud-init-datasources ubuntu ubuntu-16-custom" export DIB_CLOUD_INIT_DATASOURCES="ConfigDrive, Ec2" export DIB_RELEASE="xenial" export ELEMENTS_PATH="./elements/:/usr/share/diskimage-builder/elements/" export IMAGE_PATH="./images/ubuntu-16.04" disk-image-create vm $BASE_ELEMENTS -t raw -o $IMAGE_PATH

Change the attributes of this file:


 chmod 0755 ~/diskimage-builder/build-ubuntu-16.04

After this you should get this file hierarchy:


 $tree -p ~/diskimage-builder /home/user/diskimage-builder ├── [-rwxr-xr-x] build-ubuntu-16.04 ├── [drwxr-xr-x] elements │ └── [drwxr-xr-x] ubuntu-16-custom │ ├── [drwxr-xr-x] finalise.d │ │ └── [-rwxr-xr-x] 50-configure-grub │ ├── [drwxr-xr-x] install.d │ │ └── [-rwxr-xr-x] 50-install-additional-packages │ ├── [drwxr-xr-x] post-install.d │ │ └── [-rwxr-xr-x] 50-configure-cloud-init │ └── [-rw-r--r--] README.rst └── [drwxr-xr-x] images

Run the build-ubuntu-16.04 script:


 cd ~/diskimage-builder sudo ./build-ubuntu-16.04

The sudo utility when calling is needed so that the suggestion to enter a password does not appear in the middle of the process of preparing the image. This behavior is explained by the fact that some pre-installed diskimage-builder elements contain a call to sudo (for example, a 01-ccache script from a base element, which is executed when building most distributions).


At the end of our script, the new image will be located in the ~ / diskimage-builder / images / ubuntu-16.04.raw directory.


You can use the created image in the Selectel “Virtual Private Cloud” service by downloading it via the web interface of the VPC panel or via the glance API .


Please note that the password for root was deleted at the image build stage, so access to the virtual machine via SSH will initially only be possible by key. You can add an SSH key when creating a machine in the VPC panel. Access to the server is also possible via the no-VNC virtual console in the control panel, in which case you will not need to enter the root password if it has not been set.
The root password is set in the standard way using the passwd utility.


If you try to use an image created using the above procedures in our Virtual Private Cloud service (VPC), you will encounter a number of restrictions. The following VPC features will not be available to you:



In the next section, we will show how to make a fully compatible VPC image.


Creating an image for the service "Virtual Private Cloud"


In order for the created image to be fully compatible with the VPC service, at the assembly stage you will need to add additional property properties for the image and install additional packages into the system.


Add another element ubuntu-16-selectel, for this we create a directory:


 mkdir -p ~/diskimage-builder/elements/ubuntu-16-selectel/

Inside this directory, add the item description in the README.rst file:


 ==================== ubuntu-16-selectel ==================== Build ubuntu image for Virtual Private Cloud

Create the required directories for the new item:


 mkdir -p ~/diskimage-builder/elements/ubuntu-16-selectel/{pre-install.d,install.d,post-install.d}

Change the repository list with the script ~ / diskimage-builder / elements / ubuntu-16-selectel / pre-install.d / 50-add-mirrors:


 #!/bin/bash # Add key for Selectel OpenStack repository apt-key adv --fetch-keys http://repo.os.selectel.org/selectel-openstack.key cat <<EOF > /etc/apt/sources.list ## Selectel OpenStack repository deb http://repo.os.selectel.org xenial main ## Selectel mirrors deb http://mirror.selectel.ru/ubuntu xenial main restricted universe multiverse deb http://mirror.selectel.ru/ubuntu xenial-updates main restricted universe multiverse deb http://mirror.selectel.ru/ubuntu xenial-backports main restricted universe multiverse ## Security updates deb http://security.ubuntu.com/ubuntu xenial-security main restricted universe multiverse EOF

Add additional packages with the ~ / diskimage-builder / elements / ubuntu-16-selectel / install.d / 50-add-selectel-packages script:


 #!/bin/bash packages="crontab-randomizer fstrim-blocks qemu-guest-agent set-root-pw" apt -y install $packages

The following utilities will be installed:



Also during the image build, the cloud-init package will be automatically updated to the version that is in the Selectel OpenStack mirror. This version contains patches that ensure the consistency of network settings between the virtual machine and the VPC service.


Automatic updating of packages during image building takes place using the pre-installed 00-up-to-date script, during the install.d stage.


Next, we need to add additional configuration files for cloud-init, create a script 50-configure-cloud-init in the ~ / diskimage-builder / elements / ubuntu-16-selectel / post-install.d directory:


 #!/bin/bash # Prevent cloud-init to change custom apt mirrors echo 'apt_preserve_sources_list: true' > /etc/cloud/cloud.cfg.d/50_selectel_mirror.cfg echo 'system_info:' >> /etc/cloud/cloud.cfg.d/50_selectel_mirror.cfg echo '- arches: [i386, amd64]' >> /etc/cloud/cloud.cfg.d/50_selectel_mirror.cfg echo 'search:' >> /etc/cloud/cloud.cfg.d/50_selectel_mirror.cfg echo 'primary:' >> /etc/cloud/cloud.cfg.d/50_selectel_mirror.cfg echo '- http://mirror.selectel.ru/ubuntu' >> /etc/cloud/cloud.cfg.d/50_selectel_mirror.cfg echo 'failsafe:' >> /etc/cloud/cloud.cfg.d/50_selectel_mirror.cfg echo '- http://archive.ubuntu.com/ubuntu/' >> /etc/cloud/cloud.cfg.d/50_selectel_mirror.cfg # Prevent cloud-init to disable EC2-style metadata echo 'disable_ec2_metadata: false' > /etc/cloud/cloud.cfg.d/50_enable_ec2.cfg # Perform some commands in boot time # 'runcmd' only runs during the first boot echo 'runcmd:' > /etc/cloud/cloud.cfg.d/50_first_boot_routines.cfg echo '- set-root-pw 2> /dev/null' >> /etc/cloud/cloud.cfg.d/50_first_boot_routines.cfg echo '- crontab-randomizer' >> /etc/cloud/cloud.cfg.d/50_first_boot_routines.cfg

It is also recommended to remove the pre-installed /etc/cron.weekly/fstrim script in case of using fstrim-blocks, since the pre-installed version does not include block-by-block start parameters, which will cause additional load on the system when fstrim starts.


Remove this file with a simple script ~ / diskimage-builder / elements / ubuntu-16-selectel / post-install.d / 51-remove-fstrim-weekly:


 #!/bin/bash rm -f /etc/cron.weekly/fstrim

Change the attributes of the new scripts:


 chmod 755 ~/diskimage-builder/elements/ubuntu-16-selectel/*d/*

Add a new element ubuntu-16-selectel to the script ~ / diskimage-builder / build-ubuntu-16.04:


 #!/bin/bash export ARCH="amd64" export BASE_ELEMENTS="bootloader cloud-init-datasources ubuntu ubuntu-16-custom ubuntu-16-selectel" export DIB_CLOUD_INIT_DATASOURCES="ConfigDrive, Ec2" export DIB_RELEASE="xenial" export ELEMENTS_PATH="./elements/:/usr/share/diskimage-builder/elements/" export IMAGE_PATH="./images/ubuntu-16.04" disk-image-create vm $BASE_ELEMENTS -t raw -o $IMAGE_PATH

The file hierarchy should look like this:


 $ tree -p diskimage-builder diskimage-builder ├── [-rw-r--r--] build-ubuntu-16.04 └── [drwxr-xr-x] elements ├── [drwxr-xr-x] ubuntu-16-custom │ ├── [drwxr-xr-x] finalise.d │ │ └── [-rwxr-xr-x] 50-configure-grub │ ├── [drwxr-xr-x] install.d │ │ └── [-rwxr-xr-x] 50-install-additional-packages │ ├── [drwxr-xr-x] post-install.d │ │ └── [-rwxr-xr-x] 50-configure-system │ └── [-rw-r--r--] README.rst └── [drwxr-xr-x] ubuntu-16-selectel ├── [drwxr-xr-x] install.d │ └── [-rwxr-xr-x] 50-add-selectel-packages ├── [drwxr-xr-x] post-install.d │ └── [-rwxr-xr-x] 50-configure-cloud-init ├── [drwxr-xr-x] pre-install.d │ └── [-rwxr-xr-x] 50-add-selectel-openstack-mirror └── [-rw-r--r--] README.rst

Run the image build:


 cd ~/diskimage-builder sudo ./build-ubuntu-16.04

To download an image with all the necessary properties, you need the glance utility and the RC file of access to the project (you can download it in our control panel; all the necessary instructions are published there) https://support.selectel.ru/vpc/access/ .


The command to load will look like this:


 glance image-create --name Ubuntu-16.04-VPC \ --disk-format raw \ --container-format bare \ --property hw_disk_bus=scsi \ --property hw_scsi_model=virtio-scsi \ --property x_sel_image_owner=Selectel \ --property hw_qemu_guest_agent=yes \ --file ~/diskimage-builder/images/ubuntu-16.04.raw \ --progress

At loading properties will be added:



After downloading, the image will be displayed on the Images tab in your VPC project named Ubuntu-16.04-VPC.


When the server is first started, the password for the root user will be generated and set automatically. It will be displayed on the “Console” tab; in the same place, if necessary, you can generate a new password.


In addition, you do not have to manually configure the network interfaces on the server if you change them in the control panel on the Ports tab. In order for cloud-init to reconfigure the network interfaces, you will need to restart the server on power. After that, new OpenStack metadata with new network settings will be generated.


You can prevent the cloud-init utility from changing the network settings each time the server is restarted. To do this, in the installed system, add the /etc/cloud/cloud.cfg.d/99_disable_network_config.cfg file:


 network: config: disabled

In the near future, we will add the ability to prohibit automatic reconfiguration of the network through the web interface of the VPC panel without the need to edit the system configuration.


Conclusion


In this article, we looked at the basic features of diskimage-builder, explored ways to add your own scripts to change the image configuration, the main assembly steps, and an example of loading the finished image into the cloud environment.


If you have any questions about the main ways of using this tool, or any moments in the management of images and virtual machines in the VPC service are not clear to you, please indicate this in the comments.


We will also be happy if you share your own ways of using diskimage-builder.

Source: https://habr.com/ru/post/311120/

More articles:


All Articles