Experienced trivia-4, or "Measuring backups?"

The continuation of "experienced trifles." Previous parts: one , two , three .

Today I will talk about the principles of doing backup , which are lined up as a result of trial and error, and more than once saved the situation at the most seemingly unexpected moment.

Backups are an accident insurance company. Making backups is not even a rule, it is an axiom. Whatever the hardened admin is, his hand can sometimes “flinch”, and a script written with an error, or a randomly pressed button can bloat a lot of misfortunes, what can we say about sprinkled HDDs, fire and other cataclysms. Yes, backups are costly (time, resources, equipment), their momentary effect is not obvious, but the main thing that they give is the insurance of company risks. And this is worth a lot. It would seem that the statements are clear to everyone, but often a situation arises when the server is down, there is a backup, but nothing can be done, the system is not restored. And begin dancing for a tambourine with an orchestra, pulling at least some data, etc. joy

First, let's introduce and clearly separate such concepts as archiving and backup. For example, like this:

• Archiving is a backup with LONG-TERM data storage. The procedure, when once copied, and carried to a bank safe. Archiving is an exception rather than a rule, and the procedure can be quite lengthy (depending on where and how the archives are stored).
• Backup is a backup with SHORT-TERM storage. The procedure, when copying occurs regularly, the media is overwritten, there is the concept of "depth of storage." At the same time, access to backup should usually be as fast as possible.

Archiving

Archiving is not necessary for everyone and not always. Most often, this question arises when the company goes beyond the “me, my friend, wife and courier” framework, and usually only concerns files, financial databases, postal databases, i.e. those documents that are not kept alive in archives (letters, orders, accounting, primary, etc.).
Archives are usually made once a year and are stored on external media somewhere in a safe, in a safe deposit box, in the country house of the general director (underline the appropriate). Everything is generally clear here, the main thing is not to forget to check the archive status at least once a year (for example, simultaneously with recording a new part, to restore some data from the old one) and monitor the media so that you can always read the archive of any depth (for example, I had a situation when it took an archive that was made 5 years ago and was stored on a tape, a streamer for which was not only broken and decommissioned, but was no longer issued. The archive was of course read, but how many nerves and connections it took was better not to recall ).
')
The last couple of years, by the way, more and more for archives we use not tapes, but banal HDDs. Their volume is enough, the speed of work is sufficient for the archive, the price is within reasonable limits, and a single connection interface (formerly IDE, now SATA) eliminates the problem of a “broken tape drive”, which is important because data can be read on almost any computer, without the involvement of special equipment (as is the case with tapes). When the IDE completely disappears, we will probably copy it to SATA (or whatever it will be after).

Backup

The task of the backup itself is to keep fresh backups, for quick recovery of “accidentally \ specially deleted”, or “burned out”, or “incorrectly configured”. If archives are usually stored for as long as the organization lives, and sometimes longer, then for backups you can enter the concept of the depth of storage , i.e. the time after which the backup will expire and can be overwritten with more recent data. Backup as a whole can be divided into two main parts: data backup and system backup , and a stand-alone backup of system content . The latter is a very extensive and specific topic, which strongly depends on the contents of which particular systems you want to back up (mail server, database, CRM, software settings, etc.), we will not describe it here.

Backup systems

Backup systems - this is when you need to backup not individual files, but the entire system, which can consist of several components (for example, special software, SQL database, file data), and restore it better in whole than in parts. Everything is quite simple here: choose the backup software that suits you (price, functionality, convenience) and back up the system so that you are guaranteed to be able to restore it, even in case of a complete server crash. All sorts of Acronis, Symantec System Recovery, etc. are suitable. It is important to remember this:

• You should be able to restore from backup. Train anywhere, anytime, but you MUST BE ABLE to do it.
• think over what exactly you back up. This is significant for universal servers, when, for example, on one server the database is spinning, your internal corporate site and an additional volume attached to file resources. In such a situation, it’s not worth it to back up the “complex SQL + your site” bundle to back up a second file volume of 1500 TB with it. In general, aspire to the situation of “one task - one server”, do not hang everything, everything, all on one machine, and if you do hang it, back up it “systemically”.
• Your software should be able to restore to another hardware that is different from the current one. And you should try this at least once!
• “System” backups, especially systems that are constantly used, should not be stored more than a week deep. Think for yourself, why do you need backup of your domain controller a month ago? In the case of a critical situation, you will need the most recent backup. And all the rest is a safety net, in case the “most recent” for some reason has not worked. It is illogical and unnecessarily consumes space to allocate to such a safety net more than 1-2 iterations.
• There are systems that are complexly interconnected with the entire infrastructure, and it can be difficult to restore them, even with a fresh “system backup” of a specific server at hand. Offhand: Active Directory, Exchange, etc. Allocate time, study the documentation, and in a test environment, at least once - but try to restore ABSOLUTELY EVERYTHING! It is better to learn how to do this in a relaxed atmosphere with the Internet at hand, than with an angry boss over your head.

Data backup

Data backup is a backup of individual, independent data (basically it is the contents of your corporate file storage). There is no need for a lot of wisdom - take it and copy it, you can only think about the backup scheme. For example, I adhere to the following scheme:

1. On weekends, a full backup of all data is made. The storage depth is 28 days, i.e. There are four independent full backups. Thus, for the last month we will be able to recover data for any day off.
2. On working days, a differential backup is made, with a storage depth of 14 days. Thus, over the past two weeks we can recover data for any of the days.
3. on the first weekend of each month, separately from the rest of the work, a monthly backup is made, with a storage depth of 12 months. This is a cross between backup and archive. On the one hand, the storage period is quite large, on the other hand, there is often a situation when you need to restore data “a couple of months ago, a maximum of six months”. As an option, you can not make the monthly backup a separate work, but simply copy the appropriate weekly.

In addition, I try to follow these rules:

• I use differential, but not incremental. (If you do not know what it is - be sure to read the documentation, these are very important concepts). More important to me is the gain in speed of recovery, rather than in the volume of backup copies.
• I plan backup time so as to keep up until the morning. If a backup takes longer, I split it into several jobs, several servers, or raise a question about other, faster, equipment.
• in the backup schedule I try to stick to the intervals associated with the weeks (7 days, 28 days, etc.) and not be attached to the “first / last days of the month”. A week is a fairly constant value, 7 days, and in most cases, Saturday, Sunday is a weekend.
• I use disks, not tapes. I do not like the expensive intermediary streamer between the data in the backup and the data in the file storage. If it for some reason does not work, then the whole backup system is broken for a long time. If you use hard drives, then this can be avoided.
• I try that the logically independent part of the data lay in a separate backup. For example, if we talk about Symantec Backup Exec, then one job = one media. I really do not like the situation when one work is “smeared” over several files. This not only adds confusion to the system, but in the case of accidentally overwriting one of the files (“the hand trembled”) harms not only one job but also all the neighboring ones.
• I always use the software with notifications by mail. If this is a self-written script, no one bothers to add a piece that will check at least the availability of a new backup, its size and send the data by mail. This saves time in monitoring the backup.

Among other things, it is also fashionable to use so-called. "Instant" backups, shallow depth (1-2 days) a la VSS service in Windows, when users can choose what to restore from the latest versions of the document. This is very helpful when the user edited the document in the morning, and at lunch it deleted it and asked to restore the morning version.
You can also use the DFS system as a permanent online backup, but it is worth describing it in a separate post, which I will do later.

To be continued.

PS Let me remind you, I do not set the rules, but I share my experience, non-universal experience gained in small organizations (30-500 PCs). If you have decided to do otherwise - with pleasure I will read about it in the comments.