Data flow and SLP design best practices
When constructing Storage Lifecycle Policies (SLP), it is important to construct a data flow that considers several factors resulting in a sustainable solution. These may include, but are not limited to:
Typical backup windows and duration
Backup deduplication rates
Image sizes (many small images less than 5 GB, few large images greater than 10 TB, etc.)
Workload types (database, stream handlers, NDMP, VMware, etc.)
Source and target storage types (Data Domain, MSDP, cloud, tape, etc.)
Retention levels
Immutability duration
Encryption requirements
Total number of desired copies
Network bandwidth
Performance/throughput expectations
Concurrent read/write IO to the source and destination storage
A data flow refers to the paths the data takes between copies. For example, with optimized duplication or replication with OpenStorage (OST), deduplicated data on the source is cached and then unique segments are transferred to the target. With most workload types and deduplication rates in the 90% range this is very efficient with little IO impact.
A second example is OST to tape. This data is rehydrated on the source storage server into its raw native format, transferred, and then written to tape storage. This data flow is much more IO-intensive than optimized duplication or replication and is best done as a final copy and as sparingly as possible. When an SLP is configured, the following items are defined: the number of copies, retention of each copy, the storage target for each copy, and the order in which each copy is made. The creation of these copies can be scheduled but is managed automatically by the nbstserv service in NetBackup.
A storage target can be located on a BYO server, a Veritas appliance, a 3rd party OST target, a private or public cloud storage target, or tape. When choosing a storage target it is important to choose the most performant storage target for the backup in order to complete the primary backup copy as quickly as possible. High-performance storage targets like MSDP residing upon storage HW that provides the highest level of speed and resiliency should be prioritized for the primary backup copy.
It is also important to determine what performance features are supported with the storage target, like deduplication, compression, and I/O bandwidth (that is, the speed of each spindle and number of spindles). When using MSDP as your primary storage target for backup, replications and duplications to MSDP and MSDP-C leverage optimized duplication, which minimizes storage utilization and optimizes duplication speed.
Creating an SLP that copies data from dissimilar OST devices (such as MSDP to Data Domain) is not recommended. Rehydration and an additional deduplication cycle will create performance bottlenecks that significantly reduce performance and also reverses the benefits of other features. For example, a 3rd-party OST storage target uses a different deduplication engine than MSDP/MSDP-C. Therefore, a copy of the backup image from MSDP/MSDP-C to/from a 3rd party OST storage target will require full backup image rehydration and then full image deduplication. These steps take longer to complete and consume more memory, CPU, network, and I/O resources than would be required when leveraging the optimized duplication feature. Basically, once a backup image is deduplicated, any subsequent copies made to different storage targets do not require rehydration and another deduplication activity as long as the OST devices are similar (MSDP to MSDP/C, Data Domain to Data Domain, etc.).
For long-term retention (LTR) copies, the use of tape may be chosen. Although we support duplication from OST to tape, a full rehydration of the data is required. This operation is resource intensive and reduces performance significantly when done at scale.
When using public cloud storage, it is important to consider both the performance costs and the financial impacts. Performance factors to consider include the speed, resiliency, and sizing of the storage tier that was chosen for the copy, as well as the network bandwidth between the source storage and target storage. Financial impacts are also extremely important, such as deduplication rates, total bucket size, retention targets, and the cost of warming and restoring the data. Optimized duplication will reduce the total data transferred compared to traditional cloud (cloud without deduplication).
The use of storage unit groups (STUGs) is not recommended with SLP other than with tape storage units. There are two main reasons for this:
NetBackup evaluates the sum of all the streams of the disk pools, the sum of all active jobs using all the storage units for the disk pools, and then has to add/subtract those from SLP workgroups. This is very resource intensive when done at scale and leads to significant delays in job submissions and excessive queueing that impacts backup operations as well. This is less significant for tape because the concurrent jobs that are written to a tape storage unit is relatively low (typically dozens, potentially 100) compared to the concurrent jobs to a disk storage server (typically hundreds, potentially thousands).
Optimized duplication is not supported for disk storage units in STUG, even with similar storage device types. The data is rehydrated, transferred in full, then deduplicated again at the target where this is used, and incurs all of the negative impacts of doing so. Immutability is not supported with STUG configurations, with or without SLP.
When designing an SLP configuration, it is important to properly size the backup copy storage so that most restore operations will be done from it for the best performance. From a resource perspective it is more expensive to restore from secondary copies such as cloud or tape than from a local copy or an on-premises copy. Longer retention copies require more space and needs to be considered for secondary copy storage.
When determining the number of copies to be retained of each backup image, it is best not to apply a single universal approach to all data. For example, less critical images such as QA/dev should not require as many copies or be retained as long as critical production images. Treating them the same leads to extraneous copies, space, network consumption, and other impacts. Consider prioritizing or tiering the data into separate SLPs to remove inefficiency.
SLP operations should cascade with retentions, where copy 1 is the shortest retention for high performance, copy 2+ is longer than copy 1, and copy 3 is the longest. Creating more than three copies with SLP in a large configuration is not a best practice, as the resource overhead needed to create and track the additional copies with a large volume of them compounds exponentially from a database perspective.
Objectives including immutability should be considered, and implemented where WORM duration increases with each subsequent copy, similar to backup retention. If copy 1 is locked for too long it may cause capacity management problems and impact backup operations. Data cannot be easily deleted to make space for backups in this scenario. If an SLP backlog starts to build, this may impact the expiration and cleanup of WORM images also.
Data characteristics are attributes of specific data such as segment object size, encryption (MSDP, 3rd-party, KMS, etc.), compression, database with stream handlers, VMs with stream handlers, etc. These operations are often most critical for backup copies.
How backup copies are created impacts secondary copies. Take the following scenarios for example:
Images with exponentially more segments (DB stream handler) have much larger caches, require more memory, and/or may take longer to process if the cache limit for them is exceeded.
Data compressed or encrypted before the source backup is taken leads to low deduplication rates, often zero. It is best to use native features such as Data in-Transit Encryption (DTE), MSDP encryption, KMS, Client Direct, etc., to have NetBackup deduplicate the data first and then encrypt it.
Individual large images such as over 10 TB may require much larger cache sizes and take significantly longer to restore, duplicate, or rehydrate. Increasing the cache for these increases memory use for the storage servers involved and must be considered in designing an SLP.
If third-party encryption must be used on the source data, consider not deduplicating it and instead writing to non-OST storage to avoid the performance impacts.
Not properly considering the data characteristics for an SLP configuration often leads to poor performance, large SLP backlogs, and capacity management problems. The simplest configurations are often the best performing and most scalable configurations. Carefully consider the requirements end to end for all copies and the resources for each to avoid large-scale problems in an environment.
When designing an SLP configuration, consider the network path that the data will traverse. Keep in mind that the ideal configuration has copy 1 designed for the fastest backup and restore and shortest retentions, where copies 2+ are longer retention copies.
Traversing a slow or unstable network, excessive rehydration, shared network links between sites or applications, and other external factors such as bandwidth limits can all have significant impacts on the SLP configuration.
As a best practice, the fastest and most resilient backup storage should be targeted as the primary location for the initial backup and most of the anticipated restores. Each additional copy should have the same or increasing retention period with the longest term retention (LTR) copy being the final one. Often these reside in the cloud, tape, or Access appliances.
NetBackup provides a number of tunables for SLP processing such as:
SLP windows to defer or schedule secondary operations.
Window close behaviors that cancel or stop submitting jobs after a certain time.
Batching logic:
FIFO: First In, First Out
LIFO: Last In, First Out
Minimum and maximum batch sizes
Resource multipliers (controls how many jobs can be submitted simultaneously)
There are more parameters outlined in the NetBackup Administrator's Guide, Volume I for the desired NetBackup version.
The entire environment must be considered to design an efficient, sustainable SLP configuration. Think end-to-end what is required to create each copy, the path it must take, the resources required to create each, and the impacts of doing so.
Broadly speaking, the best practices include:
Tier the data to protect the most critical first and the less critical second.
The simplest configurations are usually the best performing ones.
When using OST, do not use dissimilar devices in the same SLP.
Avoid rehydration and duplication to tape as much as possible.
Use native NetBackup features for encryption, compression, stream handlers, etc., over third party features when possible.
Create copy 1 on the fastest storage available, and size it to satisfy most of the anticipated resource requests.
Create subsequent copies with the same or longer retention in sequence where the final copy has the longest retention in the SLP.