Backup Strategy Planning

Protecting your firm's information assets through a sound backup program is simply one of the most important responsibilities of an Information Technology function. Unfortunately, this task has been growing more complicated as a result of client-server, enterprise integration, web services and '.NET' -- which have all contributed to muddying the transaction and event boundaries essential for recovery data integrity. This white paper was originally drafted during the run-up to Year2000 and in technical specifics has become a bit dated. A short white paper called Practical Backups describes how Technology Strategists currently does this function using basic tools included in Microsoft Windows -- while not totally state of the art, our current solution has proved workable and remarkably inobtrusive -- an ideal state for a small business that must focus elsewhere than on technical operations execution.

Why Backup?
What is the Backup Environment
Costs
Backup Media Consideration
Endnote - Special Problems of SAN, NAS and storage utilities

Why Backup?
Backups are the principle means available to provide protection to a firm's information resources from loss. It should not be considered an exaggeration to think that reliable backups are the most important product of any computer operations group. This discussion focuses on the issues of departmental and large scale systems, but the desktops and personal computing devices such as smart cellphones and PDAs have similar considerations.
1. The Objective of All Backups is to Restore a System to a Known State.
  There are many reasons why a backup could be needed:
  1. Recover from accidental file deletions.
    The most common need for backups is to recover from user errors and accidents.
  2. Recover from Application Errors.
    Bad data or processing faults may make it necessary to restore application data to a known good state.
  3. Restore a Prior State of the System
    Changes to the application environment such as applying patches or operating system upgrades may have a bad impact on a production system. Undoing such a change may be very tricky if it can be done at all. Performing a full backup prior to the change is good insurance that upgrade faults can be recovered.
  4. Recover from Hardware Failure
    Hardware faults can cause data errors or more likely crash the system. It is not unusual for interrupted processing to leave an application in an indeterminate state. Many commercial database management systems include recovery mechanisms to cleanup the mess.
  5. Recover from Loss of System or Site
    Commercial applications and operating environments may be recoverable from vendor media but application data and customization recovery requires a sound backup. Media should be stored off-site. If the building is destroyed it is not very helpful if all the backups were sitting in a rack next to the computer.
  Backups are only part of the information recovery picture. Most backup processes are problematic at best if recovering to technology that is not identical to the source platform. This is particularly true of low level backups such as DD partition images. Backups in general do not preserve hardware configuration information -- SCSI bus locations, etc. A separate configuration management strategy and platform rebuild process is an essential part of a good backup strategy.
2. When Must Backups Be Done?
  Backups should be done whenever there is a logical breakpoint in the business cycle -- at the end of a business day, for example. This is not always easy -- global businesses with 24x7 activity may be very difficult. Business requirements will dictate the time constraints around the backup window. There is almost never enough time. It is not an uncommon practice to oversimplify backups to save time, but this is almost always at the expense of recovery time -- and recovery time is almost always critical.
  1. Business Requirements for Recoverability are paramount.
    Recovery time for a critical backup may determine whether a firm will survive or fail -- too extensive a delay in restoring a key system could cost business. Toleration for processing interruptions should provide direction for planning backups focused on achieving a timely recovery. The prudent planner should reconsider the chosen backup approach if it will not facilitate recovery within the necessary timeframes.
  2. Problem Detection Considerations
    Very often, backup cycles are determined by financial or legal considerations -- one of the drivers for the common weekly, monthly and annual full backups pattern. An alternate perspective on this is to consider how long a problem might 'fester' before being noticed and recovery initiated. Application data problems may be extant for many days before being noticed. If, for example, a firm chose to retain the last 3 backups and a data problem was discovered on the 4th day, after the last good backup was overwritten, recovery could be difficult. So in deciding the optimum backup strategy, problem recognition delays should be factored in when deciding on the most economical strategy.
  3. Backup Windows
    1. User Access Requirements
      Business access to application data is often blocked while backups are being performed. There are special considerations for getting around this problem -- but not without cost. The conflict between business access needs and backups can be problematic if the system is accessed from multiple time zones. File shadowing tools, like the Microsoft Volume Shadow Service, make it possible to capture a consistent file content image with minimum application disruption -- the reason VSS is invoked behind the scenes by NTBackup to capture registry content. The volume shadow copy does require additional disk space (of course) and can fail if there is insufficient room to snapshot all open files.
    2. Batch Cycle Requirements
      Batch processing time needs to be considered when designing a backup strategy. Routine processing extremes such as month end should be included. Once a system is in production, both processing times and backup duration's will increase as application data accumulates. Without planning and on-going management it is inevitable that backups, batch processing and user access requirements will collide.
  4. Special Considerations for performing backups
    1. Files closed, file system quiescent.
      1. Everything shut down, access constrained until backups are complete.
        This is the ideal state for a system to be saved -- having the production schedule time available is often an unaffordable luxury.
      2. Backup a copy of the system from a mirror disk snapshot.
        Some storage vendors provide tools that can create a snapshot of the production disks at a point in time. This snapshot is then logically separated from the production storage for backup to concurrent with processing - locally or mounted on another host on the same storage network. This is an elegant solution but can be expensive. With modern operating systems that rarely close key files, it is can be very tricky.
      3. Some operating systems provide facilities to create a temporary filesystem mirror by copying pointers & duplicating any blocks that are modified. This process makes the filesystem available sooner for processing but does require additional disk (it all has to go somewhere).
    2. Files open, application active.
      There are a number of tools and strategies for running backups while applications are active. These can be helpful but should be carefully tested so that any side-effects are well understood.
      1. Skipping Open Files
        The most common approach for any open files encountered during a backup is to ignore them. Some products pause for a while to permit the file to close. Unfortunately, it is all too common for an open file to be backed up in an inconsistent state -- i.e. some process was writing to it while the backup was running or the application had not yet flushed key internal elements to disk. This is a particular vulnerability of NAS devices and volume replication tools -- backups can be done transparently by mapping and saving the changed pages, but being able to restore a consistent image is much trickier. In general, it is not possible to snapshot open files successfully unless the application being snapshot can be forced to flush transactions to disk. This means that backups and transaction flushes must be coordinated. Discovering that key files were not backed up can be a real shock to someone complacent because backups were being run at regular intervals. Looking at the backup logs for problems is prudent -- most backup products complain when they must skip a file, but not always.
      2. Bypassing File Protections
        For those who are truly bold, ignoring open files and sweeping the contents of storage onto external media is one way of doing backups with an active system. Testing the backup periodically by restoring it onto a test system is prudent to ensure that everything that is important was recoverable and there were no corrupted files (as could occur if backup swept through while a file was being updated and internal control information was recorded in an inconsistent state). Not all backup tools will even permit this.
      3. Disk/Volume Caching software
        Tools are available in the desktop environment to utilize the operating system file cache to permit backups of open and active application files. This requires that the data manager be aware of the backup process and cooperate with it - triggering a cache flush when backup/snapshots are initiated, for example. At issue is not the ability to perform a backup, but of doing a consistent restore. The Windows VSS service fits in this category. Layered products built upon volume caching tools can provide a more granular recovery environment. See the Microsoft Data Protection Manager product as an example that works well in a smaller environment.
      4. Proprietary Database vendor tools
        Some database management products provide tools to permit backups while the application database is active. Some constrain the user to read-only access while backups are running. Others facilitate backup to a point in time by using their internal timestamp and transaction commit information to supply the backup stream with consistent information. There are resource and performance consequences for any of these approaches. But in some cases, they may be the only practical way to perform regular backups. Be aware also that 'hot backups' may have very specific restore considerations.
3. Other Considerations
  1. Business/legal retention requirements
    How long backup information needs to be retained before the media can be recycled requires consideration of both business needs and legal record retention requirements. Legal requirements are often driven by taxation legislation. It may be helpful to consult with an external auditor for verification -- there is a lot of apocryphal information floating around. And it may be necessary to specify whether the backups per se constitute the 'books' of the firm.
  2. Extant media -- need to read old files.
    Businesses can sometimes need to do analysis on old business data, often for marketing or regulatory purposes. This can be problematic if the application or backup media has changed substantially since the backups were taken. Backup media also deteriorates in storage -- but this is usually a smaller problem than technological obsolescence. The issue of access to old backups for any purpose must be thought through whenever a change in backup technology is considered - and it may be a business question. Converting to a new backup technology may have the side effect of rendering all the old backup media unreadable. This is a significant problem with archival storage of digital documents in institutional environments.
  3. Replication -- When Backups are not enough.
    Backups exact two time penalties -- when they are taken and when they are being restored. Sometimes, the recovery time objective (RTO) and recovery point objective (RPO) are too short for backup restores to be an acceptable solution. Replication -- transparently making a second (or third... etc) copy of changes to additional locations, is an alternate solution for the backup window problem. Replication can be applied at two levels -- to files (as atomic entities) or to application databases (as transactions). Replication can be thought of as disk mirroring with separation between the pairs. Distance (and other factors) inevitably causes the remote mirror to lag behind the source -- necessitating asynchronous update processes to avoid impacting overall performance. This delay exaggerates inconsistencies created by cached or staged disk writes. As with regular backups, transaction writes must be applied in the correct order to avoid file corruption. This problem becomes more complex when multiple databases are involved. File updates, on the other hand, can be queued for replication when an updated file is closed. Distributed file systems can be used for simple files to maintain replicas automatically across multiple servers. Many databases offer tools to remotely replicate database changes by (often automatically) shipping transaction logs and reapplying them to remote databases. This is often the most efficient way of maintaining a remote database replica. One common fact of all replication strategies -- they have ongoing communications requirements that can be very expensive over long distances.
What is the Backup Environment?
1. Single System with Local Backup Device
  The traditional backup environment -- most common now would be a PC with attached tape drive (like the one on which this is being written). Backup performance was once an issue of waiting for the tape drive (and still is with low end tape drives) but is now more commonly an issue of waiting for the disk -- high end tape drives are now capable of capturing data faster than it can be transferred from a disk.
2. Multiple Systems with Shared Backup Devices
  Backup devices can be shared among multiple systems by accessing the device(s) across a network or via a specialized switch. A fast tape drive can be shared among multiple systems through use of a SCSI switch, for example. Or local backups could be performed to a network-attached tape drive. Or backups from multiple systems could be interleaved into a single file by special software. This later approach is quite common in distributed office environments where individual backups are small and the time required to unthread individual files is not critical. Where devices are shared the constraint is most often the available network capacity to transfer between the source system and the recording system. One common vulnerability of shared backup systems is the dependency upon a common database for restore volume identification -- this can be a real stumbling block if using the backups for contingency, especially when the host site is not available. It is also very common for this portion of the backup equation to be ignored when planning/deploying a system -- 'decide in haste, repent in leisure...'.
3. Multiple Redundant Systems
  Additional choices are available in a high availability environment where multiple systems, perhaps in multiple locations, are used to provide fault tolerance or business continuity. There are a lot of considerations in determining the optimum strategy -- the application may be unacceptably vulnerable during backups or it may be difficult to restore the system to a known state (no known synchronization checkpoints). This could be particularly true if what is being backed up are database transaction logs rather than the database itself. The key is to test the recovery of a realistic backup to confirm integrity and usability -- this is sometimes not a simple issue.
4. What is Being Backed Up?
  It is not difficult to be too clever in designing a backup strategy that ends up with something unusable. But in the battle to minimize backups it is worth thinking about some of the bulky but slowly changing storage content such as operating system and application binaries. One must be careful of dependencies between the backed up and skipped information to ensure that the scope of selection is adequate.
  1. Base System
    1. Operating System
      If the operating environment supports the capability to boot a system from the backup media it may be necessary to routinely backup operating system binaries. This can be superfluous if an OS must be built before a backup can be restored -- but not if configuration information is intermingled. One common approach is to build a base system from distributed installation media, then restore site specific files.
    2. Configuration Information, User parameters
      System configurations, user account information, system registry files are essential to rebuilding the 'personality' of a system. These are almost always volatile and should be saved regularly. Some operating systems, like Windows XP Pro and Server, have built-in functions to 'hot' copy critical configuration files.
    3. Layered products (DBMS, Applications)
      These can often be quite bulky, particularly in the desktop world. A current set of installation kits is a good substitute. Automatic application of patches is an exposure that should be considered. Patches sometimes effect changes that require data conversions. Review of the vendor release notes is strongly suggested. In most cases, re-installing the kit provides a good starting point, so only the proprietary data needs to be backed up regularly.
  2. Application Configurations & User Data
    Part of the planning process around application backups is to consider the nature of the user data being captured by the backup. If the information can be quickly regenerated from external sources it may be more practical to have a rebuild strategy rather than a restore strategy. In general one should strive to understand the nature of the persistent data being captured when planning a backup/restore strategy.
    1. Whole Disks or Filesystems
      The common approach is to backup entire devices or file systems. This works until storage volumes begin to get 'large'. The definition of 'large' continues to change but generally means any backup that takes longer than one is willing to wait for to complete. Applications that are big enough to spread across multiple disks or file systems can be both a curse and a blessing. They are a curse because that means that backups are likely to be very time-consuming. But a blessing because it may be possible to split the backup into multiple independent streams running to separate backup devices.
    2. Selective File Backups
      If only a few files change it may be possible to backup only the new or changed files and skip the rest. This forms the basis of incremental and differential backups. Incremental backups capture all new or changed files since the last backup. Differential backups capture all new or changed files since the last full backup. Or specific files could be 'hard-coded' in a backup procedure -- this approach is potentially dangerous if it is possible for the 'hot' file population to change (and not get backed up) without updating the backup script.
    3. Specialized backup interfaces
      Application developers may sometimes implement journaling capabilities in their code rather than using a vendor backup -- focused solutions such as this can be very effective if done right. Also, database vendors sometimes license their internal backup tools to proprietary backup software vendors to enable direct product backups. Both the time/cost to backup and complexity of restore should be evaluated before embracing a proprietary backup interface -- these interfaces can sometimes be much slower than alternative approaches.
5. Common Backup Selections
  Commercial backup products normally offer a number of different ways to select which information is to be backed up.
  1. Full
    A full backup selects all the files in the specified path for transfer to media. Usually the directory entry for the file is updated to show that the file has been archived. When this can be overridden so that the archive information is not changed then the backup is referred to as a 'copy'.
  2. Incremental
    An incremental backup selects all the files that have changed since the previous backup by consulting the archive date in the file directory entry. When the backup is complete the archive information is updated to show the files as having been backed up. When backup is run the next time, these files are ignored with the rest of the previously archived data, and only the new and changed files are saved. When a system is rebuilt the process is to first restore the last available full backup. Then each incremental tape must be restored in the chronological order in which they were made. If a single file restore is necessary then the specific tape on which it was captured must be known. This can be a slow process if a tape management system is not being used. System administration guides can be helpful in designing more complicated nested incremental approaches -- 'father/son', 'grandfather/father/son', etc.
  3. Differential
    Differential backups are an alternative approach to capturing small changes. Like incremental backups, differential backups capture all files that have been created or changed since the last archive date (according to the file's directory information). Where these approaches differ is that the archive information is not changed when the backup completes. So the same files will be backed up every time to a growing collection on the media (the prior copy is usually overwritten). The virtue is that when a rebuild is needed, only the full backup and the latest differential need to be restored. This works well with cartridge tape on a standalone desktop where backups run automatically over lunch (for example).
Costs
A number of costs need to be considered when evaluating a backup solution:
1. Software - for proprietary systems like Legato or Tivoli Storage Manager.
  1. Licensing - the cost of the tool and any supporting components.
  2. Maintenance Fees - for software problem fixes and updates to keep backups working with changing technology.
  3. Support - should be available in the timeframes when backups are being run for the most timely problem resolution.
2. Hardware
  1. Capital Investment/Lease Costs - for the backup devices or juke boxes.
  2. Hardware Support
  3. Periodic Maintenance and Emergency Repair
  4. Downtime
    There is at least an availability cost to the business for downtime during processing periods. The potential for downtime should be considered to increase as the complexity of the backup devices increases.
3. Training
  Whatever tool and approach is used, people need to be trained in its use and retrained as both the user and technical populations change.
  1. Operations
    May need troubleshooting as well as operating training.
  2. System Administration
    May need installation, configuration and troubleshooting training.
  3. End Users
    Should understand what is being backed up when and why. And need to understand the process for requesting a restore or a special backup.
  4. Software Development
    More than just users, software development needs to understand the backup environment to ensure that new applications integrate successfully with it.
4. Operational Costs
  1. Backup Time
    There may be an availability cost to the business incurred while the system is not available because of backups.
  2. Media management time
    Larger shops need to consider how backup media is indexed, stored, retrieved and maintained. This process can be time-consuming so it is useful for a business to evaluate whether they can afford to not have a media librarian (and backup).
5. Communications Costs
  Backup to an external service will usually incur line charges -- especially if a dedicated ISDN or T1 line (or more) is needed. These costs should be included in the cost evaluation. They will be a major constituent of any remote replication strategy.
6. Offsite Storage/Distribution Costs
  External storage vendors will charge by the media unit or storage volume retained per unit time -- usually a month. There are also pickup and delivery costs for routine service. There may be some value in factoring in an allowance for emergency media returns -- depending upon the environment.
7. Media
  1. Volumes
    Start with trying to estimate the number of backup media required to perform a single backup. Using an upper bound of the total storage volume, less the planned minimum free space for the volume, divided by the uncompressed media capacity will provide a starting point for the number of media required. (See discussion of media capacity.) The number of tapes (or other media) that will be required in total over the duration of the retention cycle should be evaluated -- how many media for each backup, multiplied by the rotation cycle, multiplied by the off-site storage cycle. This can be a surprisingly large number -- with the price of state of the art media at hundreds of dollars per unit the total investment should be calculated. Media life should not be ignored -- very few vendors are up front about the useful life of their products, which can be surprisingly short. A common brand of QIC (quarter inch wide cartridge) which advertises a 'Lifetime Warranty' has a useful life of 47 full backups -- or about a year of weekly use.
  2. Preparation and Maintenance Costs
    Resources are required to select and label media prior to backups. Periodic maintenance may also be required -- tapes, for example, need to be re-tensioned occasionally to preserve readability.
  3. Storage Conditions
    The conditions under which backup media is to be stored needs to be considered. All backups are ultimately physical changes in specific locations on the storage media. These locations will change relative to the read-write device if the temperature of the media is different between when it was written and when the read attempt is made. Magnetic tape, for example, expands when it gets warm -- this expansion is not uniform in all directions and will distort the 'tracks' of information recorded on it. Helical scan tapes, such as 4mm or 8mm, can become unreadable when the tape expansion distorts the diagonal 'tracks' into curves. It is important that media be stored in a controlled environment at a temperature not too dissimilar from that of the data centre for optimum readability. It should also be noted that tapes are affected by humidity -- although low humidity can be dangerous because of increased exposure to static discharges.
Backup Media Considerations
1. Cost
  Media unit costs are often initially quite high but tend to decline over time as a technology becomes accepted and vendors recoup their development costs. This high cost is often made more acceptable by the large storage volumes possible -- so the cost per megabyte (or per gigabyte) could be quite reasonable.
2. Capacity (Native, Compressed)
  Capacity is much like automotive gas consumption -- actual capacity will vary. Native (raw) capacity should be predictable. But compressed capacity is another story. Effective compressed capacity will be determined by the specifics of the information being backed up and the technique being used for compression. If the vendor is minimizing inter-record gaps when a recording device is allowed to stream (write continuously) then compression could be minimal if being supplied from a slow device or across a busy network. Compression that uses run length encoding to squeeze out lengthy sequences of identical characters will be defeated by highly variable data. And sometimes, the 'compressed' size of an object can be larger than the object itself (try zipping a zip file!) If compressed capacity is critical there is no substitute for testing against a copy of production data.
3. Media Life Vs Technology Life
  All backup media will deteriorate over time, even under ideal storage conditions. Compounds in the material react with moisture in the air, pollutants and themselves. Imposed magnetic fields will weaken over time. Optical media will loose its reflectivity or opacity as it ages. This inevitable deterioration imposes a life span on the recorded information. There will be a point at which the original information cannot be recovered through ordinary means. With highly stressed materials (like some cartridge tape technologies) this can be as little as a year. Where long term recovery is important it may be helpful to contact the media manufacturer directly for longevity information and storage recommendations. Some media requires periodic maintenance to preserve long term readability -- 9 track tapes and many cartridges need to be retentioned periodically. Do not be surprised if this information is not readily available -- over the years, many public studies of media life have been suppressed, possibly by the same manufacturers.
  The rapid changes in backup technology should not be ignored. Upward compatibility is not always a consideration in developing the next generation of storage devices. A change in backup technology may well force a 'flag day' where all old backups become unreadable because the recording devices were replaced. This issue should be considered when there are business or legal requirements to be able to read the old backups.
4. Interchangeability
  One should not assume that all backup media can be used interchangeably among different drives of the same type, even from the same manufacturer. This is more of a problem with magnetic technology where changes in head position or alignment can affect the readability of the written media. An off-spec drive can easily read and write its own tapes that may be completely unrecognizable to another drive of the same kind. If interchangeability is critical then there is no substitute for testing with multiple devices.
Endnote - Special Problems of SAN, NAS and storage utilities
Organizational storage volumes have continued to grow. With this growth has come new means to connect systems to storage and new perspectives on the 'right' approach to storage management. Matching storage and backup needs to systems has always been difficult. The use of optical fibre or high speed network attachments has enabled many systems to take advantage of large hardware-based storage devices or pools of distributed storage. There are operational efficiencies to be gained by pooling storage in these ways and treating storage as a 'utility' service.

The storage 'utility' is still subject to the same technical constraints as its simpler predecessors -- most critically that backups need to be coordinated with the applications using the storage to ensure that recoverable snapshots can be created. Handshaking mechanisms must be established to ensure that the logic writing to the storage is aware of the backup process and coordinates its updates accordingly. This is a feature of many database management systems, but must be triggered explicitly. This will be true even if the adopted backup technique is to break, snapshot and re-synchronize mirror disk sets.

A second consideration about storage utilities is one of sheer size and acceptable restoration windows. As storage volumes for very large databases continues to grow past multiple gigabytes, terabytes and petabytes, the problem of recovery becomes acute. Removable media has simply not kept up with the explosion of storage volumes. So recovery from media of a multi-terabyte database would simply take far too long. This inevitably forces organizations to rely on disk backups and replication rather than traditional removable media like tape. And these very large storage farms are most often found only in fortress-like data centers designed to withstand almost any conceivable attack.

But the unthinkable can happen, so prudent management will seek to replicate these information resources at other sites. This is doable with existing technology -- vendors such as EMC have provided replication utilities that can copy updated blocks to remote storage instances. Database vendors likewise have built mechanisms to update multiple databases to facilitate the same goals. And large bandwidth network connections are available almost anywhere at a cost. The key issue that needs to be considered is how readily the application data architecture replicates -- since it is unlikely that the application can become quiescent long enough to create a static snapshot. The specific issues of database update strategy and performance are outside the scope of this paper. But simplistically, it must be noted that application and database design must include performance and recovery considerations to be effective.

The overall issue remains the same -- protection of corporate information resources. The task has become more difficult as storage volumes have grown and operations maintenance windows have vanished. But the requirement for information protection and availability continues and cannot be ignored -- but must be designed into the application architecture and supporting infrastructure to be effective.