⬆️ ⬇️

Holland - MySQL / PostgreSQL backups without a headache

Holland Logo One day I got tired of using self-written scripts to create backup copies of databases. It does not matter, they were developed by me or found somewhere on the Internet. Based on the principle that time is the most expensive resource of the system administrator (engineer, architect), a solution was found that meets the following requirements: simple installation, quick setup and, as a sum of previous requirements, quick commissioning.



According to the official website , Holland is an open source backup framework developed by Rackspace and written in Python. The project aims to create backups with great configuration flexibility, logical structure and ease of use. Currently, Holland is working with MySQL and PostgreSQL, but in the future it will include a greater variety of databases, and even applications that are not related to databases. Due to the modular structure, Holland can be used to create backups of anything, anything.



Imagine that our script provides for a daily backup of one MySQL database (using the mysqldump utility) with a rotation of seven copies.

And first you need to download the subject and install it on the server. In package distributions, this should not cause difficulties. Also imagine that we have CentOS.



Download



Holland exists in repositories for:



Also present on github.com

')

I suppose with this choice there should be no problems with the installation even on the “old” servers. The package manager of the distribution kit with whom it is necessary to work, will help with this business. Following the task, we enter into the console, if necessary, adding elevated privileges to ourselves:

yum install -y holland holland-mysqldump 




The next step, which is logical, will be the setting.



Tune



The structure of / etc / holland is simple and straightforward. In the directory with the configuration of the program in addition to the configuration file with the main parameters of the work there are two directories:

providers contains templates with settings for working with utilities like mysqldump or xtrabackup

backupsets contain specific backup plans with type parameters: number of copies, method and compression ratio, etc.



I found it necessary to provide examples of configurations in an intact form for quick reference by the reader.



/etc/holland.conf
 ## Root holland config file [holland] ## Paths where holland plugins may be found. ## Can be comma separated plugin_dirs = /usr/share/holland/plugins ## Top level directory where backups are held backup_directory = /var/spool/holland ## List of enabled backup sets. Can be comma separated. ## Read from <config_dir>/backupsets/<name>.conf # backupsets = example, traditional, parallel_backups, non_transactional backupsets = default # Define a umask for file generated by holland umask = 0007 # Define a path for holland and its spawned processes path = /usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin [logging] ## where to write the log filename = /var/log/holland/holland.log ## debug, info, warning, error, critical (case insensitive) level = info 


providers / mysqldump.conf
 ## Global settings for the mysqldump provider - Requires holland-mysqldump ## ## Unless overwritten, all backup-sets implementing this provider will use ## the following settings. [mysqldump] ## Override the path where we can find mysql command line utilities #mysql-binpath = /usr/bin/mysqldump ## One of: flush-lock, lock-tables, single-transaction, auto-detect, none ## ## flush-lock will place a global lock on all tables involved in the backup ## regardless of whether or not they are in the backup-set. If ## file-per-database is enabled, then flush-lock will lock all tables ## for every database being backed up. In other words, this option may not ## make much sense when using file-per-database. ## ## lock-tables will lock all tables involved in the backup. If ## file-per-database is enabled, then lock-tables will only lock all the ## tables associated with that database. ## ## single-transaction will force running a backup within a transaction. ## This allows backing up of transactional tables without imposing a lock ## howerver will NOT properly backup non-transactional tables. ## ## Auto-detect will choose single-transaction unless Holland finds ## non-transactional tables in the backup-set. ## ## None will completely disable locking. This is generally only viable ## on a MySQL slave and only after traffic has been diverted, or slave ## services suspended. lock-method = auto-detect ## comma-delimited glob patterns for matching databases ## only databases matching these patterns will be backed up ## default: include everything #databases = "*" ## comma-delimited glob patterns to exclude particular ## databases #exclude-databases = ## only include the specified tables #tables = "*" ## exclude specific tables #exclude-tables = "" ## Whether to dump routines explicitly ## (routines are implicitly included in the mysql database) dump-routines = no ## Whether to dump events explicitly. ## Note that this feature requires MySQL 5.1 or later. dump-events = no ## Whether to stop the slave before commencing with the backup stop-slave = no ## Whether to record the binary log name and position at the time of the ## backup. bin-log-position = no ## Whether or not to run FLUSH LOGS in MySQL with the backup. When FLUSH ## LOGS is actually executed depends on which if database filtering is being ## used and whether or not file-per-database is enabled. Generally speaking, ## it does not make sense to use flush-logs with file-per-database since the ## binary logs will not be consistent with the backup. flush-logs = no ## Whether to run a separate mysqldump for each database. Note that while ## this may initially sound like a good idea, it is far simpler to backup ## all databases in one file, although that makes the restore process ## more difficult when only certain data needs to be restored. file-per-database = no ## any additional options to the 'mysqldump' command-line utility ## these should show up exactly as they are on the command line ## eg: --flush-privileges --reset-master additional-options = "" ## Compression Settings [compression] ## compress method: gzip, gzip-rsyncable, bzip2, pbzip2, or lzop ## Which compression method to use, which can be either gzip, bzip2, or lzop. ## Note that lzop is not often installed by default on many Linux ## distributions and may need to be installed separately. method = gzip ## Whether to compress data as it is provided from 'mysqldump', or to ## compress after a dump has finished. In general, it is often better to use ## inline compression. The overhead, particularly when using a lower ## compression level, is often minial since the entire process is often I/O ## bound (as opposed to being CPU bound). inline = yes ## What compression level to use. Lower numbers mean faster compression, ## though also generally a worse compression ratio. Generally, levels 1-3 ## are considered fairly fast and still offer good compression for textual ## data. Levels above 7 can often cause a larger impact on the system due to ## needing much more CPU resources. Setting the level to 0 effectively ## disables compresion. level = 1 ## If the path to the compression program is in a non-standard location, ## or not in the system-path, you can provide it here. ## ## FIXME: Currently not implemented, compression binary is looked up by ## which. ## #bin-path = /usr/bin/gzip ## MySQL connection settings. Note that Holland will try ot read from ## the provided files defined in the 'defaults-extra-file', although ## explicitly defining the connection inforamtion here will take precedence. [mysql:client] defaults-extra-file = /root/.my.cnf,~/.my.cnf, #user = hollandbackup #password = "hollandpw" #socket = /tmp/mysqld.sock #host = localhost #port = 3306 


backupsets / mysqldump.conf
 ## Holland mysqldump Example Backup-Set ## ## This implements a vanilla backup-set using the mysqldump provider which, ## in turn, uses the 'mysqldump' utility. ## ## Many of these options have global defaults which can be found in the ## configuration file for the provider (which can be found, by default ## in /etc/holland/providers). [holland:backup] plugin = mysqldump backups-to-keep = 1 auto-purge-failures = yes purge-policy = after-backup estimated-size-factor = 1.0 # This section defines the configuration options specific to the backup # plugin. In other words, the name of this section should match the name # of the plugin defined above. [mysqldump] ## Override the path where we can find mysql command line utilities #mysql-binpath = /usr/bin/mysqldump ## One of: flush-lock, lock-tables, single-transaction, auto-detect, none ## ## flush-lock will run a FLUSH TABLES WITH READ LOCK prior to the backup ## ## lock-tables will instruct 'mysqldump' to lock all tables involved ## in the backup. ## ## single-transaction will force running a backup within a transaction. ## This allows backing up of transactional tables without imposing a lock ## howerver will NOT properly backup non-transacitonal tables. ## ## Auto-detect will choose single-transaction unless Holland finds ## non-transactional tables in the backup-set. ## ## None will completely disable locking. This is generally only viable ## on a MySQL slave and only after traffic has been diverted, or slave ## services suspended. lock-method = auto-detect ## comma-delimited glob patterns for matching databases ## only databases matching these patterns will be backed up ## default: include everything databases = "*" ## comma-delimited glob patterns to exclude particular ## databases #exclude-databases = ## only include the specified tables tables = "*" ## exclude specific tables #exclude-tables = "" ## Whether to dump routines explicitly ## (routines are implicitly included in the mysql database) dump-routines = no ## Whether to dump events explicitly. ## Note that this feature requires MySQL 5.1 or later. dump-events = no ## Whether to stop the slave before commencing with the backup stop-slave = no ## Whether to record the binary log name and position at the time of the ## backup. bin-log-position = no ## Whether or not to run FLUSH LOGS in MySQL with the backup. When FLUSH ## LOGS is actually executed depends on which if database filtering is being ## used and whether or not file-per-database is enabled. Generally speaking, ## it does not make sense to use flush-logs with file-per-database since the ## binary logs will not be consistent with the backup. flush-logs = no ## Whether to run a separate mysqldump for each database. Note that while ## this may initially sound like a good idea, it is far simpler to backup ## all databases in one file, although that makes the restore process ## more difficult when only certain data needs to be restored. file-per-database = no ## any additional options to the 'mysqldump' command-line utility ## these should show up exactly as they are on the command line ## eg: --flush-privileges --reset-master additional-options = "" ## Compression Settings [compression] ## compress method: gzip, gzip-rsyncable, bzip2, pbzip2, lzop, or xz ## Which compression method to use, which can be either gzip, bzip2, or lzop. ## Note that pbzip2 and lzop are not often installed by default on many Linux ## distributions and may need to be installed separately. method = gzip ## Whether to compress data as it is provided from 'mysqldump', or to ## compress after a dump has finished. In general, it is often better to use ## inline compression. The overhead, particularly when using a lower ## compression level, is often minial since the entire process is often I/O ## bound (as opposed to being CPU bound). inline = yes ## What compression level to use. Lower numbers mean faster compression, ## though also generally a worse compression ratio. Generally, levels 1-3 ## are considered fairly fast and still offer good compression for textual ## data. Levels above 7 can often cause a larger impact on the system due to ## needing much more CPU resources. Setting the level to 0 effectively ## disables compresion. level = 1 ## If the path to the compression program is in a non-standard location, ## or not in the system-path, you can provide it here. #bin-path = /usr/bin/gzip ## MySQL connection settings. Note that these can be inherited from the ## provider itself allowing for global defaults. Providing connection ## information for a backup-set can often be helpful when, for instance ## a backup-set is backing up a remote MySQL server. #[mysql:client] #user = hollandbackup #password = "hollandpw" #socket = /tmp/mysqld.sock #host = localhost #port = 3306 




And, despite the variety of variables that configs abound, for the implementation of ideas we need only:

  1. Specify script name in holland.conf

     backupsets = mysqldump 


  2. Copy the script from /usr/share/doc/holland-*/examples/mysqldump.conf to / etc / holland / backupsets

     cp /usr/share/doc/holland-*/examples/mysqldump.conf /etc/holland/backupsets/ 


  3. Specify in the mysqldump.conf script the number of copies, the required databases, and access with sufficient rights

     backups-to-keep = 7 databases = «somedb» user = hollandbackup password = «hollandpw» socket = /tmp/mysqld.sock 


  4. Add a daily command entry to the scheduler (for example, cron)
     holland backup 





Other


The configuration for PostgreSQL will differ only by another installed plugin (holland-postgresql) and another copied example. However, the sample files interested me even by their name, take a look:



Do not forget to log in to the server after a few days and check whether the backup plan is running, its success.



I hope these few steps will help you save time and energy in such an unpopular lesson as backup.

Source: https://habr.com/ru/post/213929/



All Articles