Implementation of corporate Linux in PrivatBank

Story

The introduction of Linux in PrivatBank began in 2007. During this time, a long way has been traveled and I would like to share my experience with the community. At the moment, we have achieved the following indicators: more than 36,500 Linux workstations in 4,000 branches located in 5 countries.
In 2007, ASPLinux 11.2 was taken as the basis. Over time, other distributions were chosen for the alternative - Fedora, openSUSE, Ubuntu. Later it became apparent the need to create your own distribution and workstations management system. Development began in January 2012. Ubuntu 12.04 LTS with a Gnome Classic working environment (no effect) was chosen for the framework. The main arguments: Ubuntu - the most common desktop distribution in recent years; extensive community, where it is easier to find a solution to emerging problems; it was him who chose Google as the basis for implementation, many examples of implementation in state and municipal institutions of Germany and France. The choice of control system stopped at Puppet.
In June 2012, the transition started and by January 2013 about 95% of PCs were already transferred to the corporate OS. This speed of transition is due to the fact that employees already had experience in Linux.

The main tasks that were solved thanks to the current implementation:

• substantial savings of resources with the support of the OS in the workplace of employees;
• keeping the software up to date;
• the ability to quickly apply critical updates (up to 1 hour on all PCs);
• collection and analysis of statistical information about the PC park and peripherals;
• creation of a proactive reaction system for failures (Event Manager).

Further more detailed description of the components of our implementation.

I. Installation

Tasks:

• saving time;
• possibility of remote installation of the OS;
• alternative installation methods.

To install and configure the OS you need to go through 4 steps:

1. Download Live (4 ways);
2. Expand the image;
3. Run privat-setup (setup wizard, connection to Puppet);
4. Configure the peripherals.

Boot menu

Installer (LIVE)

Ideology is not just an installer, but a tool with wide functionality: diagnostics, data recovery, winchester cleaning, etc., also includes OpenSSH and x11vnc for remote administration. In addition, Live creates a reference image of the system. The basis for the creation served as a manual from the office. resource A script was written that creates Live from scratch every time, adds new packages, copies settings and some hacks.

LIVE - graphic mode
')
Ways to boot LIVE:

• over the network - the main way; when it is used, it downloads via dhcp + tftp + nfs from localserver (see below);
• usb-flash - running the script, create a bootable USB flash drive, use isolinux + grub4dos;
• cd - a tribute to history, by analogy usb-flash;
• hdd - download from grub2 to upgrade the version of the OS - the planned use to move to 14.04.

System image and its deployment

Our “bicycle” is created using squashfstools with xz compression, thanks to which its volume is about 1GB. The main advantage of this method is that when synchronizing over the network, only the difference is downloaded. When using an image created by clonezilla or tar + xz - during the update process, the image was practically pumped over again (more than 90%). It should be noted that the PC should have at least 800MB of RAM (older ones are now rare), because xz consumes a lot of memory when unpacking and, if there is less memory, the system will swap, and the deployment process will significantly stretch in time (by 512MB - about 40 minutes).
During the image deployment, the following actions are performed:

• MBR cleaning;
• partition table creation;
• create swap partition;
• creating and formatting the root partition;
• unpacking squashfs image;
• creating and formatting a home partition;
• install grub + edit fstab.

On average, on a typical PC (Intel Atom CPU D525 and above, 2GB of RAM), the process takes about 5 minutes.

Privat-setup

It actually replaces the installer of the distribution kit and is made using debconf. It starts after image deployment and allows you to configure unique parameters: type of PC, localization, host name, connects the host to Puppet.

Connect to puppet

Ii. Localserver

Localserver is a branch PC that serves as a caching and network boot server. This is a very important element of the overall system. Without it, we would overload all channels with update traffic and could not download Live over the network. In most cases, these are panels that show ads, but absolutely any PC running around the clock can fulfill this role.
Performs the following functions:

1. apt-cacher (in some cases, up to 98% of traffic is saved);
2. synchronization of the installation image of the system with the standard (every night);
3. network boot server (tftp + nfs);
4. server synchronization updates intrabank software.

You can make the PC localserver at any time with Puppet Manager by simply adding this property to it.
How do customers find localserver? For this, the avahi protocol is used. A daemon is running on the server that announces the supported services on the network, and a daemon on the client that periodically “polls” the network for the presence of services. The result is written to / etc / hosts.
Additionally, the defaultlocalserver - localserver in the middle-office mechanism was created, which replaces the localserver that is missing in the subnet.

Job localserver

Iii. Control

Puppet
As already mentioned, Puppet is selected as the management platform. The main criteria are simplicity of writing manifestos, a large community, project support by companies such as Cisco, Google, Red Hat.
Clients connect to the management system every 30 minutes. Send the facts, take the job, after completing the job send a report.
Puppet functions:

• control of passwords and configuration files;
• monitoring the relevance of critical packages;
• inventory of PCs and peripherals;
• PC load monitoring;
• monitoring software bugs and hardware.

Puppet manager
At the time of implementation, there were 2 frontends for Puppet - foreman and puppet dashboard, but both could not fully fulfill the functions we needed: the first - due to server orientation, the second - did not have LDAP authorization. It was decided to write our frontend, and it turned out to be very correct, because it gave us flexibility that was unattainable when using other options.
Main features of Puppet Manager:

• generation of host configurations;
• analysis of facts and reports;
• PC grouping by territorial and functional basis;
• separation of administrator rights;
• display of hardware and software configuration, connected peripheral devices;
• graphical display of PC utilization (CPU, RAM);
• filters
• Event Manager (analysis of reports and facts with their subsequent grouping by levels and types).

Filter in Puppet Manager

I would also like to specifically mention our latest development - Event Manager. The idea of ​​creation is “borrowed” from monitoring systems. The main task of the Event Manager is to provide the system administrator with a tool to monitor the status of a remote PC, to find out about the problem before the user reports it. Based on the reports sent by the devices, we can analyze the state of the latter and distribute them according to the degree of criticality, as well as generate any events based on the facts.

Event manager

Iv. Repository + Time-Shift

A mirror of the Ubuntu repository is created on the corporate network. It also created its own repository (100 packages) for the internal software and the one missing in the official repositories.
Categories:

• System and user profile settings;
• Internal specialized software;
• Application software - software for POS terminals;
• Drivers for Wi-Fi, webcams, MFP;
• Software not included in the official repository - Chrome.

The need to create the following appeared after 3 months of using the corporate OS. Another kernel update came out (3.2.0-28) and on a PC with ATI video cards, the monitor did not turn on after “falling asleep”, etc. The solution for eliminating such situations was the creation of Time-Shift.
Time-Shift - additional “freezing” of the repository for 2 weeks (similar to the freezing system in linux mint debian edition).
In fact, now we have 3 repositories:

• current - current ubuntu;
• testing - testing updates for 2 weeks on 1% of the PC;
• stable - stable version for the remaining 99%.

During the use of Time-Shift it was possible to fix versions of packages several times. For example, there was a kernel update (3.2.0-38), in which the video fell off on the Intel Sandy Bridge.

Scheme of work Time-Shift

Total

If there is interest in the above, we will write a series of articles that will describe in more detail the implementation of system components, organizational aspects, use of Windows programs, replacement of TeamViewer, authorization in ldap when using eap-ttls, Puppet optimization for heavy loads, correct boot flash drive, operation of drivers for peripherals, Wi-Fi, etc.
Head of the corporate OS development and support group - Koval Bogdan