📜 ⬆️ ⬇️

Centralized backup of Windows and * nix servers using Bacula

Greetings to all the farmers!

As you might guess, it will be about backups.
Timely backup is an extremely important part of the work of the system administrator. A timely backup makes your sleep calm, and your nerves become steel, gives strength and protects your health.

I think it would be quite reasonable to assume that this topic has already turned sore, but still I would venture to share my experience. The client-server implementation of the backup scheme will be presented to the reader. As a tool, I chose the open source Bacula project. For more than half a year of experience in using it, I remain satisfied with my choice.
')
Bacula consists of several demons, each of which carries its own functional load. The figure below schematically shows the relationship of these demons.

Under habrakat I will describe all demons in detail


In my case, the backups are:
  1. Configuration files of various daemons from all servers.
  2. MySQL database.
  3. Document management from a Windows file server.
  4. Various important data from nix servers (site / forum engines, etc ..)




1.Description of Bacula Demons



The system is built on client-server technology, and uses TCP for data transfer. Backups are created in their own, fully open format.

The Bacula data backup system consists of four main elements: Director Daemon, Storage Daemon, File Daemon and Bacula Console. All these elements are implemented as standalone applications.

Director Daemon (DD) is the central element of the system that manages its remaining components. Its tasks include managing the backup / restore process, providing a management interface for administrators, and more. Simply put, it is a dispatcher who initiates all processes and monitors their progress.

Storage Daemon (SD) is an application that is responsible for reading / writing data directly to storage devices. Accepts control commands from DD, as well as backup data from / to File Daemon.

File Daemon (FD) - this element can still be called the Agent. After all, it works within the operating system, the data of which must be backed up. File Daemon performs the entire routine by accessing the backed up files and their further transfer to SD. Also on the FD side, backup encryption is performed, if defined by the configuration.

Bacula Console (BC) - system administrator interface. At its core, it is a command interpreter for managing Bacula. Strictly speaking, the Bacula Console can be expanded with the help of graphical control systems, which, as a rule, are just a superstructure above BC. Such systems include Tray Monitor and Bat. The first one is installed on the system administrator’s computer and monitors the backup system operation, while the second provides the ability to control via a graphical interface.

Bacula Catalog is a database that stores information about all the reserved files and their location in backup copies. The directory is necessary to ensure effective addressing to the required files. MySql, PostgreSql and SqLite are supported.
This structural division allows you to organize a very flexible backup system when the Storage Daemon is deployed on a dedicated server with multiple storage devices. Also, Bacula Director can manage multiple instances of SD, providing backup of part of data to one storage device, and parts - on the other.

2. OS and iron


Now, when the reader has formed an idea of ​​the work of the demons of Bacula, I will go on to describe how I realized all this beauty in myself.
As a hardware for DD, SD and Bacula Catalog I have a PC with the following characteristics
DeviceModelquantityCapacity / Frequency
HDDHitachi HDS723020BLA64232Tb
CPUAMD Phenom (tm) II X4 970 Processorone3500 Mhz
MotherboardGigabyte GA-880GA-UD3Hone-
Ram3541 Mb

About OS and Software Versions Used on the Server
# lsb_release -a LSB Version: :core-4.0-ia32:core-4.0-noarch:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-ia32:printing-4.0-noarch Distributor ID: CentOS Description: CentOS release 5.7 (Final) Release: 5.7 Codename: Final # uname -a Linux backupsrv.domain.ru 2.6.18-274.7.1.el5PAE #1 SMP Thu Oct 20 17:03:59 EDT 2011 i686 athlon i386 GNU/Linux # rpm -qa |grep -E "syslog-ng|bacula|mysql-ser" bacula-libs-5. 0.3-1 syslog-ng-2.1.4-9.el5 bacula-mysql-5. 0.3-1 mysql-server-5. 0.77-4.el5_6.6 


Data storage is handled by several software (mdadm) RAID arrays.
Under the system, three partitions on three disks, you can boot from any of them, under the backups of one array of two partitions.
Array namefrom which partitionsmount pointFile systemArray level
md0/ dev / sda1, / dev / sdb1, / dev / sdc1bootext2one
md1/ dev / sda2, / dev / sdb2, / dev / sdc2/ext3one
md2/ dev / sda3, / dev / sdb3/ backupext4one

3. Description of the backup scheme and settings of the Bacula daemons


I have a total of 19 Bacula clients configured, but I’ll discuss in detail the description of the backup of the billing server and documents from the Windows file server. The focus on these two servers is due to the fact that other clients are configured in a similar way, and you can build your configurations using the example of these server clients.

Backup server billing is, in fact, backup mysql database and configuration files of demons.
BD allows you to run a local script on the client both before and after the job.
Every night, at the start of the task on the backup server, a local script is launched (on the billing server itself), which creates an archive of the billing database, then this archive picks up the BD and places volumes on the corresponding pool (in fact, SD controls read / write operations, but it is not important now). Immediately after the task is completed, another script is launched, which in turn moves the created archive to a separate folder on the billing server, for greater reliability. Thus, the database archive will be both in Bacula and locally on the billing server (yes, I am paranoid). These mechanisms and scripts will be described in more detail below.

From the Windows file server, we save all the necessary workflow. On Sunday, a full backup is created, every next day, from Monday to Saturday, Diff backups.

Now about the configuration files of the Bacula daemons. Let's start with the most voluminous - bacula-dir.conf.
The configuration files of all Bacula daemons consist of descriptions of the so-called resources. Each of the resources describes a specific functional demon.
I will comment on every line in the config, so the Bacula file resource blocks (bacula-dir.conf, bacula-sd.conf, bacula-fd.conf) will follow, if something needs to be explained in more detail, indicate this in the comments.
Dirtector Resource
 Director { #  bacula director' Name = backupsrv.domain.ru-dir #   ,   default DIRport = 9101 #   ,   sql     Bacula Catalog(mysql database) QueryFile = "/usr/lib/bacula/query.sql" #      WorkingDirectory = "/backup/bacula-work/" # pid   PidDirectory = "/var/run" #      Maximum Concurrent Jobs = 1 #     BC    Password = "bacula_paS$w0rD10*" #   mail',    Messages Messages = Daemon #      DirAddress = 10.1.19.2 } 


Resource catalog, describe the connection to the database
 Catalog { Name = MyCatalog dbname = "bacula"; dbuser = "bacula"; dbpassword = "edsfweo8vhwpe" } 


Resource Messages
 Messages { #      Director, ? Name = Daemon #    email mailcommand = "/usr/sbin/bsmtp -f \"\(Bacula\) \<%r\>\" -s \"Bacula daemon message\" %r" #     (root   admins@domain.ru) #   ,     mail = root@backupsrv.domain.ru = alert,error,fatal,terminate, !skipped #     console = all, !skipped, !saved #    append = "/var/lib/bacula/log" = alert,error,fatal,terminate, !skipped } 


For each client in the tasks indicated Pool and Storage.
Pool, sorry for the tautology, this is a pool of volumes (volume) on which backup copies of customer data are placed. My volumes are bacula files located on the software raid array. Different pools of volumes can be defined for different clients. For example, I have created 6 pools for different types of clients. In the example below, only one of them is described, for billing data.
Storage describes which physical devices will be used as volumes.
(Storage BGB-ST is described in the SD config)
Pool resource
 Pool { #  ,      Name = bgb #  ,       Pool Type = Backup # #   (   1-,   2-, #  3-, 3-  -   1-) Recycle = yes #    bacula catalog( mysql  ) #    AutoPrune = yes #       (volumes) #    ,      #  Volume Retention = 90 days #    Maximum Volume Bytes = 100G #      Maximum Volumes = 3 #       LabelFormat = "Vol" } 


Resource Storage
 Storage { #     (        #  Bacula,   ) Name = BGB-F Password = "StoRage_PaSSw0rD" # fqdn   Address = backupsrv.domain.ru #    SDPort = 9103 #      SD Device = BGB-ST #            # bacula( /backup/bgbilling/Vol0001) Media Type = File } 


Task for example backup database billing.
Resource Client
 Client { #  Name = bgbilling-fd # ip   Address = 10.103.2.5 # ,    FDPort = 9102 #  mysql   Bacula Catalog = MyCatalog #   FD Password = "Fd_paSSw0rd" #         #    ,      # (   !!) File Retention = 45 days #  ,     Job Retention = 90 days #    ( mysql)    AutoPrune = yes } 


The task itself.
Resource Job
 Job { #   Name = "BGBilling" # (backup or restore) Type = Backup # (,   ) Level = Full #   Client=bgbilling-fd #  -(   ,    ) FileSet="bgbilling-set" #  SD  Storage = BGB-F #  (     (volume)    # ) Pool = bgb #      (   - #    !) ClientRunBeforeJob = "/root/sh/before_bg_db_backup.sh" #      ClientRunAfterJob = "/root/sh/after_bg_db_backup.sh" #   messages,       Messages = Standard #   Schedule = "DaylyFullBGBilling" #       ,     # ,     , #     -    ,    Write Bootstrap = "/backup/bsr-files/bgbilling.bsr" } 


I promised to elaborate on the scripts running before and after the task.
Script to set
 $ sudo cat /root/sh/before_bg_db_backup.sh #!/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin mysql -e "flush tables with read lock" --user=root --password="ololo" bgbilling lvcreate -L20G -s -n backup_db /dev/BGB-LVM1/billing_db mysql -e "unlock tables" --user=root --password="ololo" bgbilling mount /dev/BGB-LVM1/backup_db /backup tar -czf /usr/backups/`date +%Y-%m-%d_%H-%M`.bgb.tgz /backup/bgbilling/ umount /backup lvremove -f /dev/BGB-LVM1/backup_db 

Script after the job
 $ sudo cat /root/sh/after_bg_db_backup.sh #!/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin rm /usr/backups/after_run_bacula_backup/* mv /usr/backups/*.tgz /usr/backups/after_run_bacula_backup/ 


Resource FileSet (what we backup and what not)
 FileSet { Name = "bgbilling-set" Include { Options { #      , #      md5 signature = MD5 } #  ,    File = /usr/backups File = /etc File = /root/sh } Exclude { #      File = /usr/backups/after_run_bacula_backup/* File = /usr/backups/after_run_bacula_backup } } 


Schedule launch tasks.
Resource Schedule
 Schedule { #   Name = "DaylyFullBGBilling" #    Run = Full sun-sat at 1:10 } 


I will not comment on resources for backing up documents from a Windows server in detail, I’ll give the full part of the bacula-dir.conf config
 Storage { Name = WINDOWS-F Address = backupsrv.domain.ru # NB Use a fully qualified name here SDPort = 9103 Password = "StoRage_PaSSw0rD" Device = WINDOWS-ST Media Type = File } Pool { Name = windows Pool Type = Backup Recycle = yes # Bacula can automatically recycle Volumes AutoPrune = yes # Prune expired volumes Volume Retention = 60 days Maximum Volume Bytes = 30G # Limit Volume size to something reasonable Maximum Volumes = 5 # Limit number of Volumes in Pool LabelFormat = "Vol-Windows" } Job { Name = "centra-bdk" Type = Backup Level = Full Client= centra-bdk-fd FileSet="centra-bdk-fd-fs" Storage = WINDOWS-F Pool = windows Messages = Standard Schedule = "Windows_Centra-bdk" Write Bootstrap = "/backup/bsr-files/centra-bdk.bsr" } FileSet { Name = "centra-bdk-fd-fs" Include { Options { signature = MD5 Compression=GZIP } #       ! File = "D:\\Public\\!!!\ " File = "D:\\Public\\\ " File = "D:\\Public\\tex\\Maps" File = "D:\\Public\\\  " File = "D:\\Public\\\ 1" } Exclude { File = "*.mp3" File = "*.avi" File = "*.wmv" } } Client { Name = centra-bdk-fd Address = 10.1.19.50 FDPort = 9102 Catalog = MyCatalog Password = "Fd_paSSw0rd" # password for FileDaemon File Retention = 30 days # 30 days Job Retention = 2 months # two months AutoPrune = yes # Prune expired Jobs/Files } Schedule { Name = "Windows_Centra-bdk" Run = Level=Full on sun at 07:10 Run = Level=Differential on mon-sat at 22:15 } 

The BD configuration file is now complete. Go to the SD configuration - the description of the file bacula-sd.conf

Resource Storage
 Storage { #   SD Name = backupsrv.domain.ru-sd #   SDPort = 9103 #   (  ) WorkingDirectory = "/var/lib/bacula" # pid   Pid Directory = "/var/run/bacula" #    ip SDAddress = 10.1.19.2 } 


Resource Director (connection to BD described in the bacula-dir.conf config)
 Director { #  DD,  ,     Name = backupsrv.domain.ru-dir #  Password = "StoRage_PaSSw0rD" } 


The description of various devices begins, in total 4 different devices are used by me. I will give as an example two, for billing and for Windows.
Resource Device for billing.
 Device { # ,         Name = BGB-ST #  Media Type = File #      (, volumes) Archive Device = /backup/bgbilling #       Pool'( Vol*) . #  DD LabelMedia = yes; #    File    Random Access = Yes; #   ,   AutomaticMount = yes; #   =) RemovableMedia = no; #   ,     AlwaysOpen = no; } 


Resource Device for Windows File Server
 Device { Name = WINDOWS-ST Media Type = File Archive Device = /backup/windows LabelMedia = yes; Random Access = Yes; AutomaticMount = yes; RemovableMedia = no; AlwaysOpen = no; } 


Resource Messeges.
 Messages { #  Name = Standard #  DD =   =   director = backupsrv.domain.ru-dir = all } 


Configuration file bconsole.conf, access to the console Bacula.
 Director { Name = backupsrv.ray-com.ru-dir DIRport = 9101 address = 10.1.19.2 Password = "bacula_paS$w0rD10*" } 


Do not forget to create the appropriate Storage folders and assign bacula the owner of these folders.
Advice from the comments
@ / usr / local / etc / bacula / client.conf

@ / usr / local / etc / bacula / job.conf

@ / usr / local / etc / bacula / pool.conf

@ / usr / local / etc / bacula / fileset.conf
Configs can be divided into different files.
Options {signature = MD5 compression = GZIP}
and enable compression.

Configuring the server side is complete.

Config client

It is important to note that each of the clients must rezolvit fqdn server name in its ip address! Provide it with dns or write to the hosts!


Resource Director.
 Director { #  BD Name = backupsrv.domain.ru-dir #     BD  (   Client  BD) Password = "Fd_paSSw0rd" }  FileDaemon FileDaemon { #   Name = bgbilling-fd #   9102 FDport = 9102 WorkingDirectory = /usr/lib/bacula Pid Directory = /var/run/bacula FDAddress = 10.103.2.5 }  Messeges Messages { Name = Standard director = backupsrv.domain.ru = all, !skipped, !restored append = "/var/bacula/log" = all, !skipped } 


I mentioned in the comments of the configuration files about the correspondence scheme of passwords and daemon names in various configuration files, so if you get confused somewhere, use the image below.


4. Sample recovery procedure


To monitor and restore your backups, it is convenient to use the bat utility.
In ubuntu it is put so
 # sudo aptitude install bacula-console-qt 

I didn't find it in Gentoo Portage, so I collected it from sources.
The configuration file bat.conf is completely analogous to bconsole.conf given earlier.
So, for example, I want to restore the archive of the billing database for a certain number. The algorithm that I use is as follows:
1. Open the bat and find the desired task.

2. enter the command list files jobid = 3059 to make sure that the task contains the necessary files

3. Now go to the console (it's easier for me just to =)). In the console, I will restore the billing archive to another client

 $ sudo bconsole [sudo] password for onotole: Connecting to Director 10.1.19.2:9101 1000 OK: backupsrv.domain.ru-dir Version: 5.0.3 (30 August 2010) Enter a period to cancel a command. *restore Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" First you select one or more JobIds that contain files to be restored. You will be presented several methods of specifying the JobIds. Then you will be allowed to select which files from those JobIds are to be restored. To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Select full restore to a specified Job date 13: Cancel Select item: (1-13): 9 Defined Clients: 1: 192.168.15.12-fd 2: 1.1.1.1-fd 3: 1.1.1.75-fd 4: ASTERISK-configs-fd 5: DHCPD-configs-fd 6: GW1-configs-fd 7: GW2-configs-fd 8: NAS-20-configs-fd 9: NAS-21-configs-fd 10: NAS-6-configs-fd 11: NAS-ololo-configs-fd 12: NS_AND_MAIL-configs-fd 13: RADIUS-ololo-configs-fd 14: VIRTSRV1-configs-fd 15: bgbilling-fd 16: configs-fd 17: domain.ru-fd 18: mydomain.ru-fd 19: tv.domain.ru-fd 20: zabbix.domain.ru-fd Select the Client (1-20): 15 Automatically selected FileSet: bgbilling-set +-------+-------+----------+----------------+---------------------+------------+ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName | +-------+-------+----------+----------------+---------------------+------------+ | 3,292 | F | 1,666 | 10,874,552,420 | 2011-12-19 02:31:08 | Vol0014 | +-------+-------+----------+----------------+---------------------+------------+ To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Select full restore to a specified Job date 13: Cancel Select item: (1-13): 12 Enter JobId to get the state to restore: 3059 Selecting jobs to build the Full state at 2011-12-06 02:28:47 You have selected the following JobId: 3059 Building directory tree for JobId(s) 3059 ... +++++++++++++++++++++++++++++++++++++++++++++ 1,535 files inserted into the tree. You are now entering file selection mode where you add (mark) and remove (unmark) files to be restored. No files are initially added, unless you used the "all" keyword on the command line. Enter "done" to leave this mode. cwd is: / $ ls etc/ root/ usr/ $ ls usr usr/ $ mark usr 1,667 files marked. $ done Bootstrap records written to /backup/bacula-work//backupsrv.domain.ru-dir.restore.8.bsr The job will require the following Volume(s) Storage(s) SD Device(s) =========================================================================== Vol0010 BGB-F BGB-ST Volumes marked with "*" are online. 1,667 files selected to be restored. Run Restore job JobName: restore Bootstrap: /backup/bacula-work//backupsrv.domain.ru-dir.restore.8.bsr Where: /usr/restore Replace: always FileSet: bgbilling-set Backup Client: bgbilling-fd Restore Client: bgbilling-fd Storage: BGB-F When: 2011-12-26 15:01:38 Catalog: MyCatalog Priority: 10 Plugin Options: *None* OK to run? (yes/mod/no): mod Parameters to modify: 1: Level 2: Storage 3: Job 4: FileSet 5: Restore Client 6: When 7: Priority 8: Bootstrap 9: Where 10: File Relocation 11: Replace 12: JobId 13: Plugin Options Select parameter to modify (1-13): 5 The defined Client resources are: 1: bgbilling-fd 2: GW1-configs-fd 3: GW2-configs-fd 4: NAS-6-configs-fd 5: NAS-20-configs-fd 6: NAS-21-configs-fd 7: NAS-ololo-configs-fd 8: DHCPD-configs-fd 9: ASTERISK-configs-fd 10: NS_AND_MAIL-configs-fd 11: VIRTSRV1-configs-fd 12: mydomain.ru-fd 13: tv.domain.ru-fd 14: domain.ru-fd 15: 1.1.1.1-fd 16: 1.1.1.75-fd 17: zabbix.domain.ru-fd 18: 192.168.15.12-fd Select Client (File daemon) resource (1-18): 2 Run Restore job JobName: restore Bootstrap: /backup/bacula-work//backupsrv.ray-com.ru-dir.restore.8.bsr Where: /usr/restore Replace: always FileSet: bgbilling-set Backup Client: bgbilling-fd Restore Client: GW1-configs-fd Storage: BGB-F When: 2011-12-26 15:01:38 Catalog: MyCatalog Priority: 10 Plugin Options: *None* OK to run? (yes/mod/no): yes Job queued. JobId=3453 You have messages. * 


4. We are waiting for the successful completion of the task, the status can be monitored in the same bat.


Some more screenshots

image




I thank everyone who read my opus to the end.
By way of completion I will allow myself some more tips.
It is important not only to make backups and keep track of what they have completed without errors, but also deploy and check them regularly. This practice gives another +100 to those indicated at the beginning of calmness, nerve endurance and health. It is also a very good practice to regularly back up the bacula database and bsr files.

Happy New Year to you !!!

Sources used:


1. www.ibm.com/developerworks/ru/library/l-Backup_4
2. www.bacula.org/en/?page=documentation

Source: https://habr.com/ru/post/135291/


All Articles