To help DevOps: a firmware builder for network devices on Debian for 1008 hours

Today we will tell about a successfully implemented project for a large German manufacturer of telecommunications equipment (hubs, routers, network servers, etc.)

Task

Embedded Debian does not provide a ready-to-use environment for building arbitrary system images. To prepare it, you need to use the standard Debian distribution, have system administration skills, select and configure a set of utilities, and write a number of shell scripts to automate the process of cross-compiling packages. Having done all the manual work, you will get a console tool, which can be used exclusively by technical personnel with a certain level of skills in working with Linux OS, and this is not convenient. This raises the question of organizing multi-user access to this environment and restricting rights, as well as ongoing support of the environment condition.
')
The choice of the manufacturer of telecommunications equipment fell on the distribution of Embedded Debian due to the large number of packages included in the OS, allowing to solve a variety of tasks, and the abundance of supported platforms.

EDISON developers were tasked to develop a web service with an intuitive web-based interface in the form of a wizard that allows you to prepare an arbitrary image of the Embedded Debian-based system on the target platform in several steps, which also solves a number of related tasks.

Multi-user access to the Debian image build system.
Storage of prepared images and providing access to them.
Modification and re-assembly of previously prepared images.
Cloning and modification of previously prepared images.
The ability to share the created image with other users.
Mirroring the Embedded Debian distribution, keeping it up to date.
Pre-build distribution packages on various platforms on a schedule.
Providing statistics on the results of assembling packages on various platforms.
Support for classic img images, as well as iso images, including those with support for the Live CD .

An additional wish was the possibility of including packages developed by the manufacturer of equipment directly for manufactured devices into the system.

Implementation and technology.

Python 2 / Django.
Bash shell scripts.
MySQL
Debian 5.0 (Lenny).
Embedded Debian 1.0 (Grip / Crush).

Introduction

Linux is firmly located in the server and client telecommunications equipment segment, because this OS in the box provides all the most essential software for the operation and interaction of network nodes, provides various services on various network protocols, and in addition, the system is free.

The software part of different devices is very similar, the difference is the layout of the necessary OS libraries, as well as unique software from the manufacturer, whose main purpose is to provide a user interface for managing the device, as well as expressing corporate style.

Despite the abundance of ready-to-use libraries, there is no unique solution, all manufacturers offer the same product, but in a different wrapper, the release of which spends an impressive number of man-hours of architects, programmers, system administrators and designers.

The IT market offers a product to solve this problem in the form of a cloud service SUSE Studio , which provides an opportunity to assemble an arbitrary image of the system based on the Open Suse distribution kit in minutes, including for Embedded devices.

But the article is not about this product, but about the process of creating a similar service that automates the process of assembling Debian images for embedded and stationary devices based on the Embedded Debian distribution version 1.0 for the internal needs of one of the EDISON clients engaged in the production of telecommunications equipment based on the ARM architecture, i386 , amd6, PowerPC.

About the Embedded Debian distribution

Embedded Debian (Emdebian) version 1.0 is based on Debian 5.0 "Lenny" OS and includes the Grip and Crush distributions.
Emdebian Grip is a lightweight version of Debian for embedded and stationary devices. The distribution does not contain functional changes and provides the highest level of interoperability with Debian. Includes support for creating custom packages on Emdebian, as well as the ability to mix and match Emdebian and Debian packages with minimal effort. Distribution supports architectures i386 , amd64 , PowerPC , ARM and ARMEL , MIPS and MIPSEL . A typical Emdebian Grip-based system image is a Debian OS for embedded and stationary devices, a coreutils- based root file system, and a set of packages to support the Gnome graphical interface or KDE .

Emdebian Crush is focused on machines that cannot build packages for themselves. The distribution does not support compilers and build tools. A number of distribution packages are highly modified. The distribution supports only the ARM architecture . A typical Emdebian Crush based system image is a Debian OS for embedded devices with a root file system based on Busybox and a set of packages to support the G Palmtop graphical interface for PDAs. The minimum image size of a system without a graphical interface is about 24 MB. Unlike the Grip distribution, building, installing and maintaining a system based on the Crush distribution requires a lot of manual work and detailed knowledge of Debian.

The Grip distribution includes three branches: stable, testing and unstable, which respectively include stable, tested and unstable versions of packages. The Crush distribution includes only the stable branch.

Decision

The manufacturer was only going to switch to Debian OS, with the release of a new line of equipment, so we didn’t have any work from the manufacturer to build Debian images based on the Embedded Debian distribution, everything had to be done from scratch. As part of the pilot version of the service, it was decided to limit support for the ARM architecture and i386.

First of all, it was necessary to prepare an environment for cross-compiling packages of the Embedded Debian distribution kit and building Debian ready-to-use images on embedded and stationary devices, as well as to design and coordinate at an early stage the process of integrating key image assembly processes into the web service being developed. As a result, a set of shell scripts that are included in the kernel of the system was designed.

Creating a mirror of the official distribution - mirror.sh

The script, depending on the name of the distribution, creates a mirror of the official distribution of Embedded Debian on the local machine.

Signature:
mirror [distr_name]
Where:

distr_name - grip or crush

Using:
mirror crush

Conclusion:
crush mirroring process start
Date: 20091201-15: 00: 00
Process pid: 9850
Process finished

Auto-build distribution package - auto_build.sh

The script, depending on the given architecture and the name of the distribution kit, builds packages of the mirror of the distribution kit Embedded Debian. Upon completion of the package building process, a repository creation script based on successfully assembled packages is launched.

Signature:
auto_build [distr_name] [arch_name]
Where:

distr_name - grip or crush
arch_name - i386 or arm (in the case of c crush, only the arm parameter is accepted)

Using:
auto_build crush arm

Conclusion:
Building process of packages started
Architecture: arm
Date: 20091201-15: 00: 00
Process pid: 9850
Start to create a local repository
The making repository process started
Distr version: crush
Date: 20091201-15: 00: 00
Process pid: 9850
The process finished
The auto_build process finished

Creating a repository based on the collected packages - make_repository.sh

Signature:
make_repository [distr_name] [arch_name]
Where:

distr_name - grip or crush
arch_name - i386 or arm (in the case of c crush, only the arm parameter is accepted)

Using:
make_repository crush arm

Conclusion:
The making repository process started
Distr version: crush
Date: 20091201-15: 00: 00
Process pid: 9850
The process finished

List of distribution package packages - get_packages.sh

The script, depending on the given architecture, name and version of the distribution, returns a list of available packages.

Signature:
get_packages [distr_name] [distr_version] [arch_name]
Where:

distr_name - grip or crush
distr_version - stable, testing or unstable
arch_name - i386 or arm (in the case of c crush, only the arm parameter is accepted)

Using:
get_packages crush stable arm

Conclusion:
Package: acl
Version: 2.2.47-2em1
Description: Access control list utilities
This package contains the getfacl and setfacl utilities needed for
manipulating access control lists.
Package: apt
Version: 0.7.20.2em1
Description: Advanced front-end for dpkg
This is the next generation package manager for the dpkg package manager.
It provides the apt-get utility and the APT dselect method that provides a
simpler, safer way to install and upgrade packages.
...

List of dependencies - get_depends.sh

The script returns a list of dependencies for a given list of packages, distribution and architecture.

Signature:
get_depends [distr_name] [distr_version] [arch_name]
Where

distr_name - grip or crush
distr_version - stable, testing or unstable
arch_name - i386 or arm (in the case of c crush, only the arm parameter is accepted)

Due to the fact that the image can include more than a thousand packages by which all dependencies need to be determined, it was decided to transfer the list of packages to the script and read the results through standard I / O. The end of the input is determined by the script on a double carriage return.

Using:
get_depends crush stable arm

Entering the package list:
Package: acl
Version: 2.2.47-2em1
Package: apt
Version: 0.7.20.2em1
Conclusion:
Package: libacl1
Version: 2.2.47-2em1
Description: Access control list shared library
This package contains the libacl.so dynamic library containing
the POSIX 1003.1e draft standard 17 functions for manipulating
access control lists.
Package: libc6
Version: 2.7-18em1
Description: GNU C Library: Shared libraries
Contains the standard libraries
the system. This package includes the standard C library
as well as many others.
...

List of packages and the status of their assembly - get_sources.sh

The script, depending on the given architecture, the name and version of the distribution, returns a list of packages and the status of their assembly.

Signature:
get_sources [distr_name] [distr_version] [arch_name]
Where

distr_name - grip or crush
distr_version - stable, testing or unstable
arch_name - i386 or arm (in the case of c crush, only the arm parameter is accepted)

Using:
get_sources crush stable arm

Conclusion:
Package: acl
Version: 2.2.47-2em1
Status: success
Package: apt
Version: 0.7.20.2em1
Status: error
Package: libc6
Version: 2.7-18em1
Status: unknown

Creating a system image - image_create.sh

The script, depending on the specified set of packages, the name and version of the distribution, the architecture, creates a Debian image file ready for use on the target architecture. As a result of the script, the Debian image is available in the / tmp temporary directory.

Signature:
image_create [distr_name] [distr_version] [arch_name] [name_of_image]
Where:

distr_name - grip or crush
distr_version - stable, testing or unstable
arch_name - i386 or arm (in the case of c crush, only the arm parameter is accepted)
name_of_image - unique file name of the image being created

Due to the fact that the image may include more than a thousand packages, it was decided to transfer the list of packages to the script and read the results through standard I / O. The end of the input is determined by the script on a double carriage return.

Using:
image_create crush stable arm debian_on_arm.iso

Entering the package list:
Package: acl
Version: 2.2.47-2em1
Package: apt
Version: 0.7.20.2em1
...
Conclusion:
Creating new image: debian_on_arm
Architecture: arm
Date: 20091201-15: 00: 00
Process pid: 9850
Image type: iso

During the preparation of the environment and the implementation of the described core scripts, we immediately encountered a problem related to insufficiently detailed documentation on the official distribution site website, which also contains a number of omissions in the available guides, which, in turn, in some cases contributed to the difficulty of moving towards the final result . Another problem was associated with the lack of an img image of the Emdebian Crush distribution ready for download; this image had to be prepared manually.

Ultimately, a working sample environment, allowing you to build a Debian image on i386 and ARM architectures, was obtained. To create a mirror (local copy) of the Embedded Debian distribution, the debmirror utility, which is part of Debian Lenny, is used. To create a temporary environment for cross-compiling packages on target platforms, the utilities debootstrap, pbuilder, chroot were used. The reprepro utility was used to create a repository of compiled packages.

Cross-compilation of packages of a local copy of the distribution kit is carried out according to a schedule at night. Based on its results, statistics are generated for successfully assembled and unassembled packages, the results of which can be obtained upon request by calling the get_sources.sh script, and a local repository of the assembled packages is formed, which helps to accelerate the assembly of an arbitrary image, since for all the necessary architectures, and the build procedure does not need to be run in runtime during the image build process.

Debian img and iso images for the ARM and i386 architecture with a minimum number of pre-installed packages were prepared to build Debian images. During the Debain image creation process, the image template is copied to a temporary folder, then the image is mounted, and using the chroot and apt-get utilities, the collected packages from the local repository are included in the image, after which the resulting image is ready for use on the target architecture. The whole process of building img or iso image takes from several seconds to several minutes depending on the number of included packages.

The development of the web service was carried out in parallel with the process of preparing the environment for assembling Debian images due to the pre-designed kernel script structure. In the process of developing a web service, the scripts for getting the list of packages, dependencies, statistics of package assemblies and image assembling were executed as stubs that return fake data according to the designed format. As soon as a working sample of the environment was obtained, which allows you to build a Debian image, all calls to fake scripts were replaced by workers, as a result, the process of integrating the image assembly procedure into the web service was instantaneous.

Conclusion

The developed web service provides an intuitive web-based interface in the form of a wizard that allows you to create an arbitrary Debian image in just four steps, without having technical knowledge in this area.

Select the Grip or Crush distribution, i386 or ARM architecture, the Linux kernel version, and specify the name of the image being created.
Selecting a list of packages using the get_packages.sh script.
Defining a list of dependencies and the inclusion of additional packages in the created image using the get_depends.sh script.
Create and load a complete Debian image using the image_create.sh script.

The web service does not permanently store the created images on the disk; instead, all the settings of the created images are stored in the database, using them, you can collect a similar image at any time. This approach makes it possible to share the image with an arbitrary number of users, clone existing images and make other configurations based on them, without spending additional disk space.

The web service provides visual statistics on how to build distribution packages, so you can analyze the problems encountered while assembling certain packages and eliminate them.

In addition to the typical img images, Debian also supports Live CD images, thanks to which you can test a particular Debian build on a target device without having to install the OS into the device’s internal memory by booting from a flash card.

Debugging and hardware

The customer provided samples of devices based on the ARM and i386 architecture for testing collected Debian images. Images collected for the i386 architecture were tested on stationary developers' machines. As a result, the service-prepared Debian images were successfully launched on the equipment provided by the customer.