Budget solution for backup of the whole office

Most of the articles in our blog are written by developers. We decided to correct this injustice and add some DevOps. Today we will talk about the important thing - about backups.
Since Badoo is actively developing and the number of employees is constantly increasing, we came to the conclusion that centralized backup is much more convenient than partial copying and storage of information in various places.
In this article, we will look at how to “save up” quite a large number of workstations using one storage facility in various ways, without resorting to serious investments and avoiding cumbersome implementation.
We make a reservation in advance that the backup does not cover 100% of employees, since not all of them store their data on local machines, so we didn’t have a goal to make a backup necessarily-forced.
One of the main difficulties of centralized backup was the fact that employees use different operating systems.

How could we gather all on one server?

')
Sample statistics based on the use of our system:
Mac OS ~ 66%;
Linux ~ 27%;
Windows ~ 7%.

And now, in more detail, what is hidden behind these three operating systems and how we will customize them with the user.
1) Mac OS - via Time Machine (in recent versions, Mac OS is supported out of the box, below 10.6 we simply do not have), on the server side - Netatalk ;
2) Windows - standard tools through “Backup and restore” ( Backup and restore ), on the server side - Samba ;
3) Linux - several options: samba, rsync (password access); full ssh disabled as unnecessary and for security purposes.

Let's go ahead and show how the user management interface that needs to be backed up will look like:



The administrator must select an existing account or create a new one, generate a password and specify the type of operating system.

Hardware

A server with 24 3.5-inch and 3 TB disks was assembled as storage, a large amount for small money.
Each disk is mounted separately, RAID or LVM is not used - if one of the disks fails, it can be quickly replaced; It also solves the problem of lack of free space: we squeeze out the maximum volume from all the disks.
“What will happen if one of the disks on which users were backed up dies?” - you ask. Ask for backup again after replacing the disk. The probability that one of the 24 drives will die with the employee’s laptop is really small. In extreme cases, we will try to recover the lost information.
The main function of the server is to store information, so it makes no sense to describe the CPU and memory, any modern processors will cope with this task.

Cooking discs

We format and mount each disk:

parted /dev/sd${i} -s mklabel gpt parted /dev/sd${i} mkpart primary 0GB 2996GB mkfs.ext3 /dev/sd${i}1 

We delete the reserved space of the superuser:

 tune2fs -m 0 /dev/sd${i}1 

In / etc / fstab for each section we register

 UUID=${UUID} /storage/sd${i}1 ext3 noatime,acl,user_xattr,usrquota 0 0 

As a result, we obtain disks mounted to the / storage / sd $ {i} directory, where $ {i} is one of the letters of our disk.

It is better to mount disks with a UUID reference, since “mixing” the disks is not excluded, and with so many devices we will have to restore the correct paths for a long time after a sudden reboot of the server.
We will limit all users with quota in the file system using standard Linux tools, therefore we will prepare disks for this:

 quotacheck -cu /storage/sd${i} 

Software part

Netatalk

In this solution, the most difficult is Mac OS X, so let's start with the afp setting.
Install the necessary packages:

 rpm -Uvh libdb-4_8-4.8.30-18.6.x86_64.rpm rpm -Uvh db-utils-4.8.30-18.6.x86_64.rpm rpm -Uvh netatalk-2.2.4-3.7.x86_64.rpm rpm -Uvh netatalk-devel-2.2.4-3.7.x86_64.rpm 

Netatalk will use version 2. *, because version 3. * does not support the use of variables in path indications ( http://netatalk.sourceforge.net/3.0/htmldocs/afp.conf.5.html , section VARIABLE SUBSTITUTIONS : ).

It is worth noting that we collected the Netatalk package with the following flags:

 --with-cracklib --with-bdb --enable-tcp-wrappers --enable-zeroconf 

Add the line to /etc/netatalk/AppleVolumes.default

 ~/TimeMachine "BackupMachine" allow:@backupuser cnidscheme:dbd options:usedots,upriv,tm volsizelimit:250000 dbpath:/local/netatalk/db/$u 

wherein:

allow:@backupuser - allows users who are in the backupuser group to connect;
BackupMachine - the name that will be displayed as a mounted drive physically mounted in ~ / TimeMachine relative to the user;
volsizelimit:250000 - will limit the user to a quota of 250 GB, but only if the user is not assigned a system quota. Since in our case, we use system quotas, this option is useless;
tm - a mandatory option, allows TimeMachine on the client side to recognize the allocated resource as valid for the backup;
dbpath:/local/netatalk/db/$u - path to information on user metadata. It is necessary in the case when a list of directories and files that we backed up is built. It will be more practical if it is put on a separate disk, for example, on an SSD;
the $u variable substitutes the user name, which we need and does not work in the third version.

Enable afpd logging.
Add a line to /etc/nettalk/afpd.conf

  -setuplog "default log_info /var/log/afpd.log" -keepsessions -transall -savepassword 

wherein:
keepsessions - saves sessions when you turn off afpd;
transall - includes both protocols - AFP-over-Appletalk and AFP-over-TCP;
savepassword - allows you to remember the user's password in the local keychain (without this option, TimeMachine on Mac OS 10.8 does not work).

In /etc/netatalk/netatalk.conf
increase the maximum number of users to 100

 AFPD_MAX_CLIENTS=100 

With netatalk finished. You can connect with the standard Time Machine application, using the address of the form afp://%SERVERNAME%/

Samba

Samba will be used to connect in both Windows and Linux.
The setup is quite simple: add to /etc/samba/smb.conf

 [global] security = user workgroup = Badoo netbios name = BadooBackup local master = no domain master = no preferred master = no socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=8192 SO_SNDBUF=8192 [homes] comment = Home Directories valid users = %S writable = yes create mask = 0700 directory mask = 0700 browseable = No read only = No 

At this setting is over. In Windows, you can connect through the "Explorer", in Linux - automatically mount when you log into the system.

Access control

So, we are done with the hardware and applications. It remains to figure out how to provide users with several ways to access.
We already have a user management system , which means that we will use the familiar Puppet manifests.

User data will be stored in an intuitive MySQL table:

 | backupusers | CREATE TABLE `backupusers` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `uid` int(11) NOT NULL, `username` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL, `backup_username` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `backup_server` varchar(32) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'backupmsk', `password` varchar(100) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', `password_smb` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL, `shell` varchar(32) COLLATE utf8_unicode_ci NOT NULL DEFAULT '/bin/false', `map_drive` varchar(6) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'sdc', `quota` int(5) NOT NULL DEFAULT '250', `sftp` tinyint(1) NOT NULL DEFAULT '0', `isactive` tinyint(1) NOT NULL DEFAULT '0', `os` int(1) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci | 

The framework writes data to this table.
Several features:
The `map_drive` field is the name of the server and drive that will be used for the current user. It is not set by the administrator; it is automatically distributed depending on the space occupied in this or that section and on the server. If there is not enough space for someone, the data will be transferred to another section;
The `isactive` field indicates whether the employee is allowed to use the server for backup. If all OSes are inactive, it will go to the value 0. The user data will not be deleted (useful, for example, if the laptop is lost);
The `sftp` field allows the user to use rsync (more on this below). This method will allow “advanced” colleagues to back up their data using self-written scripts.

After updating the table, a script is launched that generates a manifest with data for each user.

In order to conveniently manage user directories, add a function to our manifest:

  define backupuser_dirs($name,$map_drive,$home="/home/${name}") { file { "$home": owner => $name, ensure => symlink, target => "/storage/${map_drive}/${name}", require => File["/storage/${map_drive}/${name}"]; "/storage/${map_drive}/${name}": owner => $name, ensure => directory, backup => false, mode => 0711; "$home/TimeMachine": owner => $name, ensure => directory, backup => false, mode => 0711, require => File["$home"]; } } 

And here is a snippet of the manifest for Puppet of one of the users:

  @user { "i.ableev": ensure => $hostname ? { /^%servername%$/ => present, #       ,     default => absent, #     ,     ,  }, home => "/home/i.ableev", #   ; (!)    /storage/$map_drive/$name uid => "1217", groups => ['backupuser'], # ,      password => 'V2UgYXJlIGhpcmluZyEgaHR0cDovL2NvcnAuYmFkb28uY29tL2pvYnMvCg==', #     Netatalk  rsync shell => "/bin/false", #   shell,     ssh } @backupuser_dirs { "i.ableev": name => "i.ableev", map_drive =>"sdh", # ,         /storage/$map_drive/$name require => User["i.ableev"]; } @exec { "i.ableev_quota": command => "/usr/sbin/setquota -u i.ableev 262144000 262144000 0 0 -a", #   250  … path => "/usr/sbin", onlyif => "/usr/bin/test `/usr/sbin/repquota -ua | /usr/bin/egrep '^i.ableev\s*' | /usr/bin/awk {'print \$4'}` -ne \"262144000\"", # …      ,     250 . } @line { "i.ableev_smb": #    /etc/samba/smbpasswd  : # @user:$uid:$hash file => '/etc/samba/smbpasswd', line => 'i.ableev:1217:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:5CDA711BBD899465D8F57D12BDF2BF68:[U ]:LCT-5058462B:', #         } 

Users now have the ability to connect to the server using Mac OS, Windows or Linux.

rsync

To provide access through rsync (but not ssh), we will use chroot plus a limited shell. The manifesto and function in this case will look somewhat different:

  define backupuser_dirs_sftp($name,$map_drive,$home) { file { "/home/${name}": owner => $name, ensure => symlink, target => "${home}", require => File["$home"]; "$home": ensure => directory, owner => root, mode => 0755; "$home/sftp": owner => $name, ensure => directory, backup => false, mode => 0711, require => File["$home"]; } } @user { "i.ableev": ensure => $hostname ? { /^backupmsk$/ => present, default => absent, }, home => "/storage/sdh/i.ableev", uid => "1217", groups => ['backupuser'], password => 'V2UgYXJlIGhpcmluZyEgaHR0cDovL2NvcnAuYmFkb28uY29tL2pvYnMvCg==', #     shell => "/bin/badooshell", #  shell    rsync } @backupuser_dirs_sftp { "i.ableev": name => "i.ableev", map_drive =>"sdh", home => "/storage/sdh/i.ableev", #    ~/sftp ―     require => User["i.ableev"]; } @file { "/storage/sdh/i.ableev/bin": #   chroot' ―     badooshell ensure => directory, recurse => true, purge => true, force => true, backup => false, owner => root, group => root, source => "puppet:///modules/officebackup/bin/"; #    } @file { "/storage/sdh/i.ableev/lib64": #   ,    rsync: ld-linux-x86-64.so.2 libc.so.6 libpopt.so.0 ensure => directory, recurse => true, purge => true, force => true, backup => false, owner => root, group => root, source => "puppet:///modules/officebackup/lib64/"; } 

All manifestos are ready.

results

What we got in the end:

• inexpensive server with 72 terabytes of useful data;
• on one server are backups of all types of user operating systems;
• access is granted to anyone who wants to make backups.

Enjoy!

Ilya Ableyev, ableev , maintenance officer, Badoo Development

Original comic: dilbert.com/strips/comic/2007-11-21