Stream Firebird 2.5 bases conversion to ODS12 format (Firebird 3.0)
Each version of Firebird has its own version of the format of disk database structures - O (n) D (isk) S (tructure). Up to version 2.5 inclusive, the Firebird engine could work with ODS of previous versions, that is, the bases from the old versions opened with the new version and worked in compatibility mode, but the Firebird 3.0 engine works only with the database in its own ODS version 12.0.
To switch to 3.0, the database from 2.5 must be converted to a new format via backup / restore. Of course, we assume that the database was previously prepared for conversion - i.e. metadata and queries were tested for compatibility with Firebird 3.0.
If you follow the standard approach, this means that you need to make a backup on version 2.5, then install 3.0 and restore it. Such a procedure is acceptable if there is enough time, but when migrating large databases, or when several dozens of databases are simultaneously migrating when time is tight, you can use stream conversion, which is 30-40% faster. How exactly to make it (under Windows and under Linux), read under a cat.
The general idea is that to speed up, we will use a pipeline:
')
Gbak from 2.5 generates backup in linear format and sends it to stdout, which immediately picks up gbak from 3.0 through stdin and creates a new database.
It is necessary to organize such a pipeline by the local (file) access method, since network access (even through localhost) will significantly slow down the process.
Below we review the details for Windows and Linux.
Windows
In the case of Windows, the easiest way to do this is to do a fully autonomous Firebird build To do this, we take the Firebird 2.5 embed-archive , rename fbemded.dll to fbclient.dll, add gbak.exe utilities from the “usual” 2.5 archive and (optionally) - isql.exe.
Firebird 3.0 uses a single assembly and does not require any refinement.
The most minimal option (not requiring installation of VS2008 / VS2010 runtime libraries on the target system) contains the following files:
An experienced administrator may notice that in 2.5 the files intl / fbintl.dll and intl / fbintl.conf are not included. This is true because gbak does not use the connection charset and does not convert data between charts, but on the “receiving” side of Firebird 3.0 these files are necessary when creating indexes.
In Firebird.conf Firebird 3.0, it is recommended to add:
Also, it is advisable to set different IpcName values ​​for 2.5 and 3.0.
When choosing the values ​​of other firebird.conf parameters, we proceed from a simple consideration: at the stage of transferring data in one process, gbak runs 2.5, and in the other, 3.0, then 2.5 completes, and 3.0 starts building indexes.
To speed up the step of building indexes in 3.0, it is recommended to increase the size of the TempCacheLimit parameter to ~ 40% RAM (if this is a dedicated server, of course).
For example, if the server has 16 GB of RAM, then you can put
Of course, this value can only be set for 64bit Firebird 3, since any 32-bit process will not be able to allocate more than 2 GB of memory.
In 2.5, this parameter does not need to be changed - it cannot be more than 2 gigabytes anyway, and it does not affect the speed when backing up.
Before performing the operation, it is necessary to check that the page cache in the database header is set to 0 (command
If the cache is explicitly specified in the database header, then it overrides the values ​​from firebird.conf (and databases.conf in 3.0), and in case of inadequately large values, it can lead to excessive memory consumption and a swap.
Next, copy the files to the target system.
Conversion is performed after stopping the “system” service Firebird 2.5, on the command line with elevation of rights to the local administrator (example):
In this example, the “direct slash” is used in quotation marks (valid “unix-style”), and the “hat” (“^” symbol) escapes the newline character, which is convenient when typing long commands. The option -st (atus) appeared in Firebird 2.5.8 and allows you to record in the log data on the running time of the gbak process (see documentation for details).
Linux
On Linux, Firebird 3 depends on the tommath library. On CentOS (RHEL), this library is in the epel repository, on Ubuntu (Debian) in the system repository.
For CentOS, you first need to connect the epel repository and only then do
Ubuntu does not need to connect additional repositories, but different versions of the packages are installed on Ubuntu 16 and Ubuntu 18 - libtommath0 and libtommath1, respectively.
Firebird 3.0 is looking for tommath.so.0 and for Ubuntu 18 you also need to create a link (symlink) from tommath.so.0 to tommath.so.1. To do this, you first need to find tommath.so.1.
The search path in Ubuntu is
The second problem is that before Firebird 3.0.1, inclusive, there was no easy way to install two different server versions. We do not consider the option “compile from source with the necessary prefix” because of its relative complexity.
For Firebird 3.0.2 and above, an build with –enable-binreloc and a separate installer option (-path path) is implemented.
Assuming that the tommath library and, if necessary, the symlink for tommath.so.0 are added to the system, you can install the actual (at the time of this writing) distribution of Firebird 3.0.4 in, for example, / opt / fb3:
After that, you can stop the Firebird system service and start streaming conversion.
When stopping Firebird, you should take into account that Firebid 2.5 processes in Classic mode usually starts xinetd - therefore, you must either disable the firebird service for xinetd or stop xinetd completely.
In firebird.conf for 3.0 on Linux, you do not need to set MaxUnflushed parameters (they work only on Windows) and change the settings for Firebird 2.5.
In Linux, local (file) access to Firebird 2.5 is not equivalent to the embeded option under Windows - server 2.5 will work in the gbak process (without the network part), but access rights will be checked against the user base, which means you will need not only a login, but also a password :
After successful conversion, you must first remove the “additional” Firebird 3.0, then the “main” Firebird 2.5, and after that perform a clean installation of Firebird 3.0 - and this is best done from the tar.gz installer, and not through the repositories, since version in the repositories may lag.
Also, after the restaurator of the database on Linux and reinstallation, you should check that the new database has the owner of the firebird user.
If it is not, then you will need to fix it.
Total
In addition to saving time and disk space for stream conversion, there is another important advantage - database conversion is done without deleting the existing Firebird 2.5, which makes it much easier to roll back if conversion is unsuccessful (most often due to lack of space or an unexpected reboot during the migration process).
The saving of time is due to the fact that the “classical” conversion is “backup time” plus “recovery time”. Recovery consists of two parts: reading data from a backup file and building an index.
With on-line conversion, the total time is obtained as “backup time plus five to ten percent” and “index building time”.
Specific results depend on the base structure, but on average the recovery time is approximately equal to double the backup time. Therefore, if we take backup time per unit, then the “classical conversion” is three units of time, and streaming is two units of time. Increasing TempCacheLimit also helps reduce time.
In general, in-line conversion in practice allows you to save 30-40% percent of the time of alternate backup and restores.
Questions?
Please write all questions in the comments, or send to the author of the methodology and co-author of this article - Vasily Sidorov, Lead Systems Engineer, iBase, at bs at ibase ru.
To switch to 3.0, the database from 2.5 must be converted to a new format via backup / restore. Of course, we assume that the database was previously prepared for conversion - i.e. metadata and queries were tested for compatibility with Firebird 3.0.
If you follow the standard approach, this means that you need to make a backup on version 2.5, then install 3.0 and restore it. Such a procedure is acceptable if there is enough time, but when migrating large databases, or when several dozens of databases are simultaneously migrating when time is tight, you can use stream conversion, which is 30-40% faster. How exactly to make it (under Windows and under Linux), read under a cat.
The general idea is that to speed up, we will use a pipeline:
')
gbak -b … 25 stdout | gbak -c … stdin 30 Gbak from 2.5 generates backup in linear format and sends it to stdout, which immediately picks up gbak from 3.0 through stdin and creates a new database.
It is necessary to organize such a pipeline by the local (file) access method, since network access (even through localhost) will significantly slow down the process.
Below we review the details for Windows and Linux.
Windows
In the case of Windows, the easiest way to do this is to do a fully autonomous Firebird build To do this, we take the Firebird 2.5 embed-archive , rename fbemded.dll to fbclient.dll, add gbak.exe utilities from the “usual” 2.5 archive and (optionally) - isql.exe.
Firebird 3.0 uses a single assembly and does not require any refinement.
The most minimal option (not requiring installation of VS2008 / VS2010 runtime libraries on the target system) contains the following files:
25/gbak.exe 25/fbclient.dll 25/firebird.conf 25/firebird.log 25/firebird.msg 25/ib_util.dll 25/icudt30.dll 25/icuin30.dll 25/icuuc30.dll 25/Microsoft.VC80.CRT.manifest 25/msvcp80.dll 25/msvcr80.dll 30/fbclient.dll 30/firebird.conf 30/firebird.msg 30/gbak.exe 30/ib_util.dll 30/icudt52.dll 30/icudt52l.dat 30/icuin52.dll 30/icuuc52.dll 30/msvcp100.dll 30/msvcr100.dll 30/intl/fbintl.conf 30/intl/fbintl.dll 30/plugins/engine12.dll An experienced administrator may notice that in 2.5 the files intl / fbintl.dll and intl / fbintl.conf are not included. This is true because gbak does not use the connection charset and does not convert data between charts, but on the “receiving” side of Firebird 3.0 these files are necessary when creating indexes.
In Firebird.conf Firebird 3.0, it is recommended to add:
MaxUnflushedWrites = -1 MaxUnflushedWriteTime = -1 Also, it is advisable to set different IpcName values ​​for 2.5 and 3.0.
When choosing the values ​​of other firebird.conf parameters, we proceed from a simple consideration: at the stage of transferring data in one process, gbak runs 2.5, and in the other, 3.0, then 2.5 completes, and 3.0 starts building indexes.
To speed up the step of building indexes in 3.0, it is recommended to increase the size of the TempCacheLimit parameter to ~ 40% RAM (if this is a dedicated server, of course).
For example, if the server has 16 GB of RAM, then you can put
TempCacheLimit=6G Of course, this value can only be set for 64bit Firebird 3, since any 32-bit process will not be able to allocate more than 2 GB of memory.
In 2.5, this parameter does not need to be changed - it cannot be more than 2 gigabytes anyway, and it does not affect the speed when backing up.
Before performing the operation, it is necessary to check that the page cache in the database header is set to 0 (command
gstat -h databasename , see the line Page buffers).If the cache is explicitly specified in the database header, then it overrides the values ​​from firebird.conf (and databases.conf in 3.0), and in case of inadequately large values, it can lead to excessive memory consumption and a swap.
Next, copy the files to the target system.
Conversion is performed after stopping the “system” service Firebird 2.5, on the command line with elevation of rights to the local administrator (example):
set ISC_USER= "25/gbak" -z -b -g -v -st t -y 25.log 25 stdout|^ "30/gbak" -z -c -v -st t -y 30.log stdin 30 In this example, the “direct slash” is used in quotation marks (valid “unix-style”), and the “hat” (“^” symbol) escapes the newline character, which is convenient when typing long commands. The option -st (atus) appeared in Firebird 2.5.8 and allows you to record in the log data on the running time of the gbak process (see documentation for details).
Linux
On Linux, Firebird 3 depends on the tommath library. On CentOS (RHEL), this library is in the epel repository, on Ubuntu (Debian) in the system repository.
For CentOS, you first need to connect the epel repository and only then do
yum install libtommath Ubuntu does not need to connect additional repositories, but different versions of the packages are installed on Ubuntu 16 and Ubuntu 18 - libtommath0 and libtommath1, respectively.
Firebird 3.0 is looking for tommath.so.0 and for Ubuntu 18 you also need to create a link (symlink) from tommath.so.0 to tommath.so.1. To do this, you first need to find tommath.so.1.
The search path in Ubuntu is
/usr/lib/x86_64-linux-gnu/ , but it may be different with other Debian-based distributions.The second problem is that before Firebird 3.0.1, inclusive, there was no easy way to install two different server versions. We do not consider the option “compile from source with the necessary prefix” because of its relative complexity.
For Firebird 3.0.2 and above, an build with –enable-binreloc and a separate installer option (-path path) is implemented.
Assuming that the tommath library and, if necessary, the symlink for tommath.so.0 are added to the system, you can install the actual (at the time of this writing) distribution of Firebird 3.0.4 in, for example, / opt / fb3:
./install.sh -path /opt/fb3 After that, you can stop the Firebird system service and start streaming conversion.
When stopping Firebird, you should take into account that Firebid 2.5 processes in Classic mode usually starts xinetd - therefore, you must either disable the firebird service for xinetd or stop xinetd completely.
In firebird.conf for 3.0 on Linux, you do not need to set MaxUnflushed parameters (they work only on Windows) and change the settings for Firebird 2.5.
In Linux, local (file) access to Firebird 2.5 is not equivalent to the embeded option under Windows - server 2.5 will work in the gbak process (without the network part), but access rights will be checked against the user base, which means you will need not only a login, but also a password :
export ISC_USER=username ISC_PASSWORD=password /opt/firebird/bin/gbak -b … 25 stdout\ |/opt/fb3/bin/gbak -c … stdin 30 After successful conversion, you must first remove the “additional” Firebird 3.0, then the “main” Firebird 2.5, and after that perform a clean installation of Firebird 3.0 - and this is best done from the tar.gz installer, and not through the repositories, since version in the repositories may lag.
Also, after the restaurator of the database on Linux and reinstallation, you should check that the new database has the owner of the firebird user.
If it is not, then you will need to fix it.
chown firebird.firebird database Total
In addition to saving time and disk space for stream conversion, there is another important advantage - database conversion is done without deleting the existing Firebird 2.5, which makes it much easier to roll back if conversion is unsuccessful (most often due to lack of space or an unexpected reboot during the migration process).
The saving of time is due to the fact that the “classical” conversion is “backup time” plus “recovery time”. Recovery consists of two parts: reading data from a backup file and building an index.
With on-line conversion, the total time is obtained as “backup time plus five to ten percent” and “index building time”.
Specific results depend on the base structure, but on average the recovery time is approximately equal to double the backup time. Therefore, if we take backup time per unit, then the “classical conversion” is three units of time, and streaming is two units of time. Increasing TempCacheLimit also helps reduce time.
In general, in-line conversion in practice allows you to save 30-40% percent of the time of alternate backup and restores.
Questions?
Please write all questions in the comments, or send to the author of the methodology and co-author of this article - Vasily Sidorov, Lead Systems Engineer, iBase, at bs at ibase ru.
Source: https://habr.com/ru/post/445204/
All Articles