27 May 2016

How to deactivate TDPO obsolete backups without TDPOSYNC utility ?

Tivoli Data Protection for Oracle (TDPO) backups will have unique version of backup stored in TSM server storage and that will be an active version. Any future backups will not be dependent on the old backups and all the backup objects will be in active version because each backup piece is stored under different filespace name and also the default retention setting for TDPO backups is 1,0,0,0. To know more about troubleshooting TDPO backups check the TDP tutorials in this website.

In order to expire the old TDPO/RMAN backups, DBA team need to regularly run the TDPOSYNC utility to mark the previous backups as expired as per the their RMAN retention settings and allow TSM server to delete the obsolete backups from TSM server storage. 

However, in some environments people doesn't run the TDPOSYNC utility in regular intervals and since active backup data is not managed by TSM inventory expiration policies, the data is not deleted automatically, and uses server storage space indefinitely. As a TSM admin, we get worried about the storagepool free space when this kind of situation occurs. 

To address this kind of situation and to free the storage space that is used by obsolete active data which is not needed, you can deactivate the data. This feature is enabled from TSM V7.1.3 and we can deactivate the obsolete active TDPO backups by providing the DATE before which all backups to be deactivated.

When you run the deactivation process, all active backup data that was stored before the specified date becomes inactive. The data is then deleted as per the retention settings  and it cannot be restored. In this way, you can free storage space on the Tivoli Storage Manager server without the need of TDPOSYNC utility or help from DBA team.


deactivate tsm data

Example: deactivate data PRD_TDPO todate=01/01/2016

The DEACTIVATE DATA command affects only the files that were copied to the server before the specified date and time. Files that were copied after the specified date are still accessible, and the client can still access the server. This command applies only to application clients that protect Oracle databases.

Automatic and controlled decommissioning of TSM client node

The conventional approach which we are following from many days to delete a node and its backups from TSM server storage is by following the DECOMMISSIONING NODE process. In this process, once we get an approval to delete a node and its backup data we dissociate that node from all the backup schedules, then we lock the node and delete all the filespaces associated to that node and then eventually we end-up removing node from the TSM inventory. This process needs TSM administrator's monitoring and intervention.

Starting from TSM V7.1.3, we have a new command (DECOMMISSION NODE) which will do all the above work in the background for us. This single command will remove a client node from the TSM server in a systematic & controlled way without our intervention.

When you start the decommission process, the server locks the client node to prevent it from accessing the server. Files that belong to the client node are gradually deleted, and then the client node is deleted. You can decommission BAclient node and as aswell as TDP & NAS nodes with this command. Remember that for decommisioning a VM node we have a separate command (DECOMMISSION VM).

Also we cannot run this command on client node that is configured for replication.We need to remove the client node from replication by using the REMOVE REPLNODE command before decommissioning this kind of node..

Issue this command to initiate a gradual, controlled decommission operation. This single command completes the following actions automatically:
  • This process will delete all schedule associations for the client node. Schedules are no longer run on the client node. This action is equivalent to issuing the DELETE ASSOCIATION command for every schedule with which the client node is associated.
  • Next, it prevents the client from accessing the server. This action is equivalent to issuing the LOCK NODE command.
  • Next, the client node data is no longer backed up to the server. Data that was backed up before the client node was decommissioned is not immediately deleted from the server. However, all backup file versions, including the most recent backup will be now marked as inactive copies. The client files will be retained as per the retention settings.
  • Once all client backup and archive file copies are removed from server storage, Tivoli Storage Manager deletes the file spaces that belong to the decommissioned node. This action is equivalent to issuing the DELETE FILESPACE command.
  • After all file spaces for the decommissioned node are deleted, the node definition is then deleted from the server. This action is equivalent to issuing the REMOVE NODE command.

DECOMMISSION NODE 

You should use this command to remove normal FILE-LEVEL node, TDP node or NAS node from TSM server. Any backup data that is stored for the client node expires according to policy settings unless you explicitly delete the data.

Decommission node

Exampledecommission node ecom_prod

Please note that this action cannot be reversed and causes deletion of data. Although this command does not delete the client node definition until after its data expires, you cannot recommission the client node. After you issue this command, the client node cannot access the server and its data is not backed up. The client node is locked, and can be unlocked only to restore files. 

DECOMMISSION VM 

You should use this command to initiate a controlled removal of the virtual machine file spaces from the server. This command will mark all data that was backed up for the virtual machine as inactive, so it can be deleted according to the data retention policies. 

decommission vm


After all data that was backed up for the virtual machine expires, the file space that represents the virtual machine is deleted. The DECOMMISSION VM command affects only the virtual machine that you identify. The data center node, and the other virtual machines that are hosted by the data center node are not affected by this command.

Example: decommission vm salesvm

Now we can start TSM server in maintenance mode

Starting from TSM V7.1.3, we can also start the TSM server in maintenance mode if we need to do any house keeping activities. Starting TSM server in maintenance mode will help us to any kind of avoid disruptions during maintenance and reconfiguration tasks because they are automatically disabled. 

Starting TSM server in maintenance mode will disable the following automatic server operations and also clients are prevented from starting sessions with the server.
  • Administrative command schedules
  • Client schedules
  • Reclamation of storage space on the server
  • Inventory expiration
  • Migration of storage pools


However, While the server is running in maintenance mode, you can manually start the storage-space reclamation, inventory expiration, and storage-pool migration processes.

dsmserv maintenanace


To start the maintenance mode issue the DSMSERV command with the MAINTENANCE parameter. We do not have to edit the server options file, dsmserv.opt, to start the server in maintenance mode.

dsmserv maintenance

To again resume server operations in the normal mode, we need to shut down the server by issuing the HALT command and then restart the server by using the same methods which we generally use (dsmserv script). Operations that were disabled during maintenance mode are reenabled and the clients can start taking backups.

IBM Spectrum Protect (TSM) V7.1.5 Directory Container Storage pools Overview

Starting from Tivoli Storage Manager Version 7.1.3, you can use the new storage pool type called directory-container storagepool to protect backup and archive data. Data that is stored in a directory-container storage pool can use either inline data deduplication, client-side data deduplication, inline compression, or client-side compression. Tivoli Storage Manager offers the following features when you use directory-container storage pools for data storage
  • You can apply data deduplication and disk caching techniques to maximize data storage usage.
  • You can retrieve data from disk much faster than you can retrieve data from tape storage.

Inline data deduplication or inline compression reduces data at the time it is stored. By using directory-container storage pools, you remove the need for volume reclamation, which improves server performance and reduces the cost of hardware. You can protect and repair data in directory-container storage pools at the level of the storage pool by using AUDIT CONTAINER command.

The directory-container storage pool provides the following benefits:
  • Simplified storage management.
  • Optimized data-deduplication processing.
  • Reduced database growth and size.
  • Increased server performance and scalability.
  • Improved recoverability of damaged extents

Some TSM internal process does not support for directory-container storage pools. You cannot use any of the following functions with directory-container storage pools:
  • Migration
  • Reclamation
  • Aggregation
  • Collocation
  • Simultaneous-write
  • Storage pool backup
  • Virtual volumes

How to define directory container storagepool

Check the below syntax to define a direcory container storagepool. Please read the description of the parameters which are provided in the syntax to get better customization of this feature.

TSM Directory container storagepool

Example: define stgpool prod_stg stgtype=directory compression=yes

Auditing Directory-Container Storagepools

Use the AUDIT CONTAINER command to scan for inconsistencies between database
information and a container in directory container storage pool. 

Audit directory container storagepool


This command is used to complete the following actions for a container in a directory-container storage pool
  • Scan the contents of a container to validate the integrity of the data extents.
  • Remove damaged data from a container.
  • Mark an entire container as damaged

Example: audit container stgpool=newdedup action=removedamaged

26 May 2016

IBM Spectrum Protect (TSM) V7.1.5 Cloud Container Storage pools Overview

Starting from Tivoli Storage Manager Version 7.1.3, we can define a new storagepool type called cloud-container storage pool. We can back up the data to either an IBM SoftLayer or OpenStack Swift cloud computing system as of now. Hoping IBM will let us take backups to AWS and Azure in future as well but now we can only configure cloud backups to either IBM SoftLayer or OpenStack Swift cloud storage.

The cloud storage can be on premises or off premises. The cloud-container storage pools that are provided by Tivoli Storage Manager can store data to cloud storage that is object-based. By storing data in cloud-container storage pools, we can exploit the cost per unit advantages that clouds offer along with the scaling capabilities that cloud storage provides. 

Tivoli Storage Manager manages the credentials, security, read and write I/Os, and the lifecycle for data that is stored to the cloud. When cloud-container storage pools are implemented on the server, we can write directly to the cloud by configuring a cloud-container storage pool with the cloud credentials. This type of storage pool is used for data deduplication. Cloud storage pools are not supported on Linux on Power Systems and Linux on System z.

The Tivoli Storage Manager server can write deduplicated and encrypted data directly to the cloud. We can back up and restore data or archive and retrieve data directly from the cloud-container storage pool. We can define the following types of Tivoli Storage Manager cloud-container storage pools

On premises
We can use the on premises type of cloud-container storage pool to store data in a private cloud, for more security and maximum control over our data. The disadvantages of a private cloud are higher costs due to hardware requirements and on-site maintenance.

Off premises
We can use the off premises type of cloud-container storage pool to store data in a public cloud. The advantage of using a public cloud is that we can achieve lower costs than for a private cloud by eliminating administrative & maintenance costs. However, we must balance this benefit against possible performance issues due to connection speeds and reduced control over your data.

The cloud-container storage pool provides the following benefits:
  • Reduced on-premises storage requirements and cost.
  • Simplified storage management
  • Improved server performance
  • Use of public, off-premises, or private, on-premises cloud storage
  • Secure data through server-side encryption

How to define Cloud Container Storagepool

Check the below syntax to define a direcory container storagepool. Please read the description of the parameters which are provided in the syntax to get better customization of this feature.

Cloud container stgpool

Example: 

define stgpool cloud_stgpool stgtype=cloud
cloudtype=swift cloudurl=http://192.168.133:5000/v2.0
identity=admin:admin password=password description="OpenStack Swift cloud"


Auditing Cloud container storagepools

Use the AUDIT CONTAINER command to scan for inconsistencies between database information and a container in a cloud-container storage pool. 

Audit container


This command is used to complete the following actions for a container in a cloud-container storage pool:
  • Scan the contents of a container to validate the integrity of the data extents.
  • Remove damaged data from a container
  • Mark an entire container as damaged

IBM Spectrum Protect (TSM) V7.1.X Storagepools Concepts (Revised)

IBM is adding new features into TSM architecture in a regular basis, and as a TSM backup specialist we need to understand and apply these new features as applicable in our backup environment and check if we can save any hardware or storage resources.

In this post we will review on the new features added to the important TSM component STORAGE POOLS.

Storagepools are the important components in the IBM Tivoli Storage Manager infrastructure. With the new features introduced in TSM 7.1.X, we can optimize the usage of storage devices by manipulating the properties of storage pools and volumes.

TSM V7.1.x storagepools


Storagepools or Server Storage:

As we all know that Storagepool is the logical group of similar media where we save our client backup/archive. We can have number of storagepools configured in our TSM environment and this group of storagepools is also known as TSM Server Storage. Any TSM infrastructure can have 3 types of storage-pools.
  •  Primary storage pools
  • Copy-storage pools
  • Active-data storage pools

Primary storage pools

Primary storage pools are the first destination of all the backups whether FILE-LEVEL or TDP backups. So when a user tries to restore, retrieve, recall, or export file data, the requested file is obtained from a primary storage pool. So it is TSM administrator responsibility to make sure that the backup data on primary storagepools are safe and restorable when needed.

You can arrange primary storage pools in a storage hierarchy so that data can be transferred from disk storage to lower-cost storage such as tape devices. Depending on the type of primary storage pool, the storage pools can be located on site or off site. Starting from TSM V7.1.3 there are 2 new types of primary storagepools and depending on the type of primary storage pool, the storage pools can be located on site or off site. There are 4 types of primary storagepools
  1. Random Access Storagepool
  2. Sequential Access Storagepool
  3. Directory-container storage pools
  4. Cloud-container storage pools

Random Access & Sequential Access Storagepools

Random Access Storagepool uses DISK devices to store the clients backup/archive and Sequential Access Storagepool uses TAPE or FILE devices to store the client backup/archive. Since these are old types, we are not going deep about these storagepool types. Please refer my previous post for their explanation....

Directory-container storage pools

This type of storage pools contain containers that are stored in storage pool directories. In directory-container storage pools, data can be deduplicated at the same time that the data is stored. By using directory-container storage pools, there is no need for volume reclamation, which improves server performance and reduces the cost of storage hardware. We can protect and repair data in directory-container storage pools at the level of the storage pool itself.

Cloud-container storage pools

This type of storagepool can store data to an object-store based cloud storage provider. By storing data storage in cloud-container storage pools, you can exploit the cost per unit advantages that clouds offer along with the scale-up and scale-out capabilities that cloud storage provides.

Tivoli Storage Manager manages the credentials, security, read and write I/Os, and the data lifecycle for data that is stored to the cloud. When cloud-container storage pools are implemented in the server, you can write directly to the cloud by configuring a cloud-container storage pool with the cloud credentials. 

The Tivoli Storage Manager server writes deduplicated and encrypted data directly to the cloud. You can back up and restore data or archive and retrieve data directly from the cloud-container storage pool.

cloud-container storage pools also supports ON-premises cloud and OFF-premises cloud to store the backup data. ON-premises feature can be used to store data in a private cloud, for additional security and maximum control over your data. The disadvantages of a private cloud are higher costs due to hardware requirements and on-site maintenance. 

OFF-premises feature can be used to store data in a public cloud and achieve lower costs than for a private cloud, for example by eliminating maintenance. However, these benefits must be balanced against possible performance issues due to connection speeds and reduced control over your data.

Copy storage pools

Copy storage pools contain active and inactive versions of data that is backed up from primary storage pools. 

A directory-container storage pool cannot be used as a copy storage pool. In addition, data from a directory-container storage pool cannot be copied into a copy storage pool. Copy storage pools provide a means of recovering from disasters or media failures. 

For example, when a client attempts to retrieve a damaged file from the primary storage pool, TSM server will choose copy storage pool if the file is damaged or non-restorable from primary storagepool. A copy storage pool can use sequential-access storage only, such as a tape device class or FILE device class.

Active-data storage pools

An active-data pool contains only active versions of client backup data. In this case, the server does not have to position past inactive files that do not have to be restored.

A directory-container storage pool cannot be used as an active-data storage pool. You use active-data pools to improve the efficiency of data storage and restore operations, for example this type of storage pool can help you to achieve the following objectives:
  • Increase the speed of client data restore operations
  • Reduce the number of onsite or offsite storage volumes
  • Reduce the amount of data that is transferred when you copy or restore files that are vaulted electronically in a remote location

Data that is migrated by hierarchical storage management (HSM) clients and archive data are not permitted in active-data pools. As updated versions of backup data are stored in active-data pools, older versions are removed during reclamation.

Active-data pools can use any type of sequential-access storage. However, the benefits of an active-data pool depend on the device type that is associated with the pool. For example, active-data pools that are associated with a FILE device class are ideal for fast client restore operations because the FILE volumes do not have to be physically mounted and the client sessions that are restoring from FILE volumes in an active-data pool can access the volumes concurrently, which improves restore performance.