Archive for April 2013

25 April 2013

Different DSMADMC options in IBM Tivoli Storage Manager TSM

03:49 No Comments

UNIX Server Command Line

To start a TSM server command line in AIX, you use the dsmadmc command with an optional servername. The command line is useful if you want to do bulk upgrades, as you can then do them with a script. For example, if you want to register 10 new nodes, rather than go through the tedium of adding 10 nodes through a panel, it is much easier to set this up as 10 commands within a UNIX script, then just execute the script. A typical script could look like

  dsmadmc -se=servername -id=youruserid -pa=yourpassword
 register node nodename nodepassword passe=9999 con=\"This server contact details\"
 do=policy-set clo=cloptset userid=none comp=no archdel=no backdel=no
you obviously need to substitute your own values for the parameters, and note how special characters like quotes are escaped out with a forward slash. The command is also split over 3 lines to make it readable, but it is actually all one line.

Other dsmadmc options

The command can also be very useful if you want to start a TSM server up in different modes. for example, if you want to run some queries that you want to feed into an excel spreadsheet or maybe programs, and you do not want the screen titles displayed you would use the following. -commad means comma delimited output, the dataonly switch suppresses the headers.

 dsmadmc -se=servername -dataonly=yes -comma

 an alternate option is -tab for tab delimited. These commands can be usefully argumented with the -out parameter, which writes the console output into an external file. For example if you start a server session as below, then the output of any commands you issue will be placed in sqlout.txt

 dsmadmc -se=servername -dataonly=yes -comma
 -out="c:\program files\tivoli\tsm\sqlout.txt"

 Two other useful options are
 dsmadmc -con
 dsmadmc -mount

 The first one starts a server session in console mode, so all the console messages automatically scroll past your terminal. This is useful if you are waiting for a process to complete, as it saves you from keep typing q actlog or q pr commands.

The second option is similar, but instead of displaying all the console messages, it just displays tape mount messages. This one was intended for a media librarian that had to mount tapes manually dsmadmc with Storage Agents

 If you are logged into a client that is running a Storage Agent called ClientA_Sta you would log into it the command

dsmadmc -se=ClientA_Sta

 and then you can run a limited commandset from the Agent's command line

To get out of the Storage Agent type 'quit'. If you type 'halt', you will stop the storage agent then exit.

Logging into a remote TSM server

With Windows servers, you can log into any remote TSM server, provided you have the dsmadmc command code loaded with your TSM client, and provided your firewall rules permit remote access. If you have a remote server on a Windows box with IP address 99.245.24.123 and with a DNS name of WSTSM01, you can start a session by navigating to \program files\tivoli\tsm\baclient and running either

  dsmadmc -tcps=99.245.24.123
   or
 dsmadmc -tcps=WSTSM01

 If you get a message like "dsmadmc not found" then you've got a basic client install and you need to install the server management component. This works on the IP address of the Windows box that is hosting TSM. What if you are running more than one TSM server on that box? By default you will get TSM Server1. To reach the other servers you need to add a -tcpp switch that specifies the port of the other server

  dsmadmc -tcps=WSTSM01 -tcpp=1502

How to synchronize or match TSM tape library inventory with tivoli storage manager TSM database entries

03:32 No Comments

Auditing Tape Libraries

You need to run an audit occasionally to make sure that what TSM thinks is in your library, matches reality.  The audit command is

  AUDIT LIBRARY library-name CHECKLABEL=BARCODE

 The CHECKLABEL=BARCODE switch is optional, but it will make the audit go pretty fast, say 5 minutes. With that switch, all the audit involves is your robot scanning the barcode labels of all the tapes. Without that switch, the default action is to mount each tape, which will take a long time. The audit may wait until all tapes are dismounted from drives, so it could take a while for an audit to start. Consider canceling tape processes if your audit is waiting, and you need it in a hurry.
---------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------

Auditing Tape Volumes

You can audit a tape volume with the command

 audit volume vol001 fix=yes

 and this will check all the backups on the tape and fix the database entries for any that are damaged. It is also possible to audit all the volumes in a storage pool with a single command

  audit volume stgpool=pool-name fix=yes

 This could audit a lot of tapes and take a long time so you can restrict it by date. For example say you had a tape drive that went faulty on March 25th 2005 and was fixed on April 1st 2005. You want to check all the tapes that were written in that period for errors. Use the command

 audit volume stgpool=pool-name fromdate=032505 todate=040105

If you chose a volume that is part of a volume set because it contains files that span volumes TSM will select the first volume in the set and scan them all, even if you pick a volume in the middle. If you just want to audit one specific volume in a set then you need to use the skippartial parameter

  audit volume volser fix=yes skippartial=yes

-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------

Controlling wait times for tape mounts

By default, a process will wait 60 minutes for a tape mount, before it gets cancelled. To change this, use the server option

 MOUNTWAIT n

 where n is a number in minutes from 0 to 9999. The parameter will only start counting once the process gets the mount message, so if a process is waiting because all the drives are in use, it will not time out

How to manage and monitor TSM tape library and tape volumes utilization and health status

03:22 No Comments

TSM Tape Library media management Tasks

UNIX and Windows Tape utilities

Windows has a utility called ntutil which can be useful for faulty tape drives, to decide if the problem is with TSM or external. The only difficulty with it is that there is no correlation between TSM devices and Windows devices, you have to work out how to match them. For example, in TSM you call your tape drives DRIVE00 to DRIVE32, and DRIVE28 is faulty. First go to TSM and run the following (the '*' just means you don't have to input the library name). Make a note of the drive serial number.

 Q DRIVE * DRIVE28 F=D

Now open a DOS command prompt and type ntutil. Take option '1 manual test' from the first menu, then take option '20 open' to open the default drive, usually \\.\tape0\. Then take option '63 get tape bus info', and you will see a list of tape drives called TAPE00 to TAPE32. However TAPE28 does not correspond to TSM Drive28, that would be much to easy. Scan down the list looking at the tape serial numbers until you spot your tape, then make a note of it's tape number, in this case it was Tape04.
Now take option '1 set device special file' and reply to the prompt with Tape04.

Take option '2 close' to close the previous tape session, then option '2 open' again, and you will open Tape04, otherwise known by TSM as Drive28. If the drive opens up at this point, it's probably OK.
However you can run various commands against your faulty drive once you open it, you can see the list from the menu. '49 enquiry' and '58 device info' can be useful.

Persistent Binding

When a server is rebooted, the tape drive definitions can change, and this can make the tape paths in both servers and storage agents incorrect. You can prevent this from happening by using Persistent Binding.

In AIX, install the Atape driver. This allows you to rename the tapes in AIX to a standard that suits you, and these names will survive a server reboot.

On Windows, you can get persistent binding if you use Qlogic device adaptors. In Qlogic, bring up the Fibre Channel Port Configuration dialog box, right click on Host Adapter, device or LUN in the HBA tree, then click on Configure in the drop down menu. Select the BIND box, and that will bind each port to its target ID.
---------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------

Library Sharing

How you define library sharing with TSM depends on whether its a SCSI library or a 3494 library, and whether or not you have a SAN. Library Sharing for SCSI libraries requires that you define the library as follow :

 for the library manager
   DEFINE LIBRARY lib-name LIBTYPE=SCSI SHARED=YES DEVICE=drive-name
 for the library client
   DEFINE LIBRARY lib-name LIBTYPE=SHARED PRIMARYLIBMANAGER=server-name

Library Sharing for 3494 libraries does not use the library manager/client configuration as described above. It needs the '3494SHARED YES' server option instead. You still need to use separate categories for the different servers otherwise you may end up with two servers having the same private/scratch volume in the library inventory. What 3494 library sharing brings is the ability to define all drives to all the servers sharing the 3494. The TSM server will detect if a drive is available and will retry based on the new retry options that were added in 4.1 (DRIVEACQUIRERETRY and MPTIMEOUT options).

-----------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------

Moving Tapes offsite

There are two parts to volume movement, updating TSM so its knows what is happening with the volumes; and physically managing the automated library inventory by using checkin/checkout commands.

If you are planning to take tapes offsite to a vault, then the important step is to update the access mode of those volumes to 'OFFSITE'. This tells TSM that it can still so some data processing like Reclamation and Move Data commands, but it will NEVER request a mount of the actual volume (it uses the primary copy instead). Note that only copy pool volumes can be set to 'OFFSITE' - this is because TSM always expects to have its primary pool volumes available (i.e.:mountable). The checkout operation for the offsite copypool volumes is a necessary extra step to get the tapes out of the library inventory.
----------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------

Freeing up library slots

If your library becomes full, you may need to free up some slots. The trick is to eject older tapes from the library that you are not likely to use for a while, and keep the active tapes in the library. The 'MOVE MEDIA' command gives us the ability to have a combination automatic/manual library. We can 'move' tapes outside of the library to a nearby 'location', but the tapes are still considered as mountable. The distinction is whether they have the media state of 'MOUNABLEINLIB' or 'MOUNTABLENOTINLIB', and this tells TSM whether to ask the robot for to mount the volume or to issue a manual mount request. When processing an manual mount request, you must use the 'Checkin Libvol' command to update the library inventory and tell TSM that the tape is back in the robot (since that is ultimately how the tape gets mounted).

TSM will automatically toggle the volume's access mode from ReadOnly to ReadWrite and back again as it is moved in and out of the library. This is to allow any read operations to proceed (e.g.: restore) and cause a manual mount request, but write operations will not attempt to access the volume.
----------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------

Reclaiming offsite tapes

You don't have to bring your offsite tapes in to do reclamation. Set your copypool reclamation to a reasonable level, say 60%. TSM knows what files are still valid on offsite volumes that are to be reclaimed. It finds the copies of those files in the primary storage pool (which is still in the library); it moves a scratch tape to the copy pool and copies the files from the primary tape pool to the new copypool tape. The new copy tape is then marked to go offsite, and the old one marked for return.
-------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------

Re-generating a library definition

Sometimes, especially with TSM servers hosted on Windows, it is necessary to delete and redefine a tape libary. After you remove the tape paths, you have to delete the library itself to remove the TSM server library inventory. You do this with the 'delete library library_name' command. Once you re-define the library and paths, you need to re-generate the library inventory by running the following commands, which must be run in the sequence shown.

    Checkin the scratch volumes
    checkin libvol library_name search=yes checklabel=barcode status=scratch
    Checkin the private volumes
    checkin libvol library_name search=yes checklabel=barcode status=private
------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------

Reclaiming tapes that are assigned to another TSM server

Imagine the scenario, you are using one TSM as a Library Manager, called TSM1, with maybe 3 other instances sharing the library. You decommission one of those instances, say its called TSM3, reclaim the physical server, then discover that TSM3 still has a lot of tapes allocated in the Library manager. The data is defunct, it's all been moved to other TSM servers and you want to reclaim those tapes as scratch. You can't change the tape to scratch status from TSM1 as it does not own the tapes. If you check them out and back in again, they are still owned by TSM3. You can't change the owner, so what do you do?

The problem is that TSM1 contains records about these tapes in it's volhist file. You need to delete the volhist record on TSM1 for each tape with this command. Put your own volume names in and be absolutely sure to get the names right or you will delete the wrong tapes!

 DEL VOLHIST TODATE=TODAY TYPE=REMOTE VOLUME=volume-name FORCE=YES
 UPD LIBR library-name volume-name STAT=SCR
-------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------

How to check Tape usage ?

The following query will produce a report of the space usage of all storage pool volumes, summarised by storage pool and status.

 SELECT STGPOOL_NAME AS STGPOOL, 
  COUNT(VOLUME_name) AS COUNT, STATUS, 
  CAST(MEAN(EST_CAPACITY_MB/1024) 
   AS DECIMAL(5,2)) AS GB_PER_VOL 
 FROM VOLUMES 
 GROUP BY STGPOOL_NAME,STATUS
Example output is shown below
 STGPOOL              COUNT     STATUS        GB_PER_VOL
 -------------  -----------     ---------     ----------
 ARCHCOPYPOOL           417     FULL                1.10
 ARCHIVEPOOL              2     ONLINE              2.78
 ARCHTAPEPOOL             2     FILLING            27.58
 ARCHTAPEPOOL             2     FULL               49.60
 BACKUPPOOL               7     ONLINE             38.26
 CARTPOOL                 1     EMPTY               0.00
 CARTPOOL               221     FILLING            22.01
 CARTPOOL              1021     FULL               33.99

   -----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------

Checkin and Checkout

TSM needs to know where it's tapes are if they are stored in tape libraries and to keep it informed you use CHECKIN and CHECKOUT commands. A tape library has a small compartment usually called an IO station with a door. You place your new cartridges into the IO station then run a checkin command. The syntax of the command varies slightly depending on what type of library you have.

 The command for a SCSI library is 
 Checkin libv Library_name search=bulk checklabel=yes status=scr
 
This will read in all the tapes in the IO station, read the labels and define them to TSM as scratch. To check in a single named tape that contains required data, maybe something you are importing from a different system, try 

 Checkin libv Library_name Volume_name  checklabel=yes  
 
Both CHECKIN and CHECKOUT commands will issue a console message asking you to confirm you are ready. Reply to the message with 'REPLY nn' where nn is the message number. However be aware that the robot will select the first tape from the IO station, it will not scan the IO station for your tape. I've had the Library Manager tell me the IO station was empty apart from a specific tape that I wanted, yet TSM kept selecting an incorrect tape on checkin. Eventually I tried a bulk checkin and then discovered there was four foreign tapes in the IO station. 

If you have lots of tapes to input to TSM you can power your library down, open the main door, then just put the tapes directly into empty libary slots. You then power the library up and run an audit, then a bulk checkin to TSM like this

 CHECKIN LIBV library_name SEARCH=YES CHECKL=B
 
If you have some unallocated tapes in your library, maybe because they were checked out with the REMOVE=NO parameter, and you want them checked in as scratch, use the following command - it compares the TSM database with the library database and will just check in tapes that TSM did not know about and set them to scratch status.

 CHECKIN LIBVOL library_name STATUS=scratch SEARCH=YES CHECKL=B
 
Most of us have several TSM servers that share a physical library. This is done by partitioning the library into several logical libraries. Say you have two servers, TSM1 and TSM2 each with virtual libraries VLIB1 and VLIB2 In this case it is quite easy to transfer data between TSM servers. You run an Export, note the tape used (xyz123) then check it out VLIB1 like this 

 CHECKOUT LIBV VLIB1 xyx123 REMOVE=YES
 
This will place the tape into the IO station. You then log into the Library Manager (the control software for the tape library) and re-assign the tape from VLIB1 to VLIB2. Finally you check it back in with 

 CHECKIN LIBV VLIB2 tapeno  xyz123 STAT=PRI  CHECKL=YES
 
If you find the tape is rejected with an invalid label, try a bulk checkin.
If a volume has been removed from the library, but TSM has not been informed you can clear it from TSM with the command below. Checklabel=no and remove=no means that TSM will do no validation, it just removes it from the database. 

 Checkout libvolume Library_name Volume_name  checklabel=no remove=no

----------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------

Reuse delay

If a tape has no active files left on it, but it still does not become scratch then the ussue might be the reusedelay parm on the copystoragepool.

A tape is not necessarily released for scratch straight away. Imagine the worst has happened, you've had a disaster and have had to restore your database back 48 hours. If tapes have been reused in the past 48 hours, then the database will not be accurate, data will be missing. To prevent this, you have a parameter called REUsedelay. This specifies the number of days that must elapse after all files are deleted from a volume before the volume can be rewritten or returned to the scratch pool. The default value for this is 0, but it may have been set to 5, say, to avoid problems with database rollback. That's one reason why tapes do not get recycled quickly.
------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------

Tape Mount and Dismount issues

TSM will keep adding data to a 'filling' tape until it is full. However, it will sometimes mount a scratch tape even if there is a 'filling' tape available for that node. This is because TSM will not wait for a tape that is currently dismounting. The logic is that it is faster to ask for a new scratch tape than to wait while a filling tape is dismounted, stored, retrieved then remounted. There is no easy answer to this feature, except to juggle your KEEPMP and MOUNTRetention values to minimise the risk.
----------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------

Maxscratch parameter

The name of parameter can be a bit confusing, as it limits the total number of tapes that a storage pool can contain, not the total number of scratch tapes. If your tape pool processing starts failing with insufficient space errors, then one cause can be that the maxscratch limit has been reached. You may have plenty of scratch tapes in your library, but TSM will not use them. The maxscratch sets the maximum number of volumes that can be used by each storage pool.
-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------

How to find out how many scratch tapes are available ?

The q vol command will only give you information about storage pool volumes, and so does not report on scratch volumes as they are not associated with s storage pool. You need to use the following SQL

select count(*) as Scratch_count
  from libvolumes
  where status='Scratch'

TSM thinks a tape contains data, but it is empty

You have a CopyPool volume, which is EMPTY and OFFSITE, but the tape does not change to scratch as normal. You cannot move the data off the tape because it is empty. You cannot delete the tape, because it contains data, not even with the discard data option. The tape needs to be audited, but to do this it must be on-site. recall the tape to your site and run an 'AUDit Volume VolName Fix=Yes'.

Altering the MOUNTABLE state

A volume is empty, but is not in scratch status because the volume STATE is mountablenotinlib. To change the STATE of the volume use the command

MOVE MED vol_name STG=pool_name WHERESTATE=MOUNTABLENOTINLIB

This will move the volume back into the scratch category

How to troubleshoot TSM tape drive issues or errors on Windows

02:53 No Comments

Are the Drives / Paths online?

At the TSM Server, check that the drives and paths are online with these commands

q drive
q path

 If any drives or paths are not online, use these commands to update them, substituting your own server, library and drive names.

update drive lib-name drive-name online=yes
update path server-name drive-name srct=server destt=drive libr=lib-name online=yes

Removable Storage manager (RSM)

TSM does not co-exist very well with Windows RSM, So check that the service is not started and is disabled. If it is started, stop it and disable it unless you plan to use it.

External Tape Problems

It's possible that there is an external problem with the library or with drives. A good place to start is to check that Windows Device Manager lists the library (Medium Changer) and/or Tape Drives. Right click on 'My computer' and select 'Manage' then 'Device Manager'. If Device Manager indicates a problem then TSM will have problems.

If you cannot see any devices in device manager, then right click on the server name in the right side display and select 'Scan for Hardware Changes'. If this does not discover the devices, you have a problem with your cabling or SAN ports.

Device Drivers

If you can see the devices but they are in an incorrect state, it could be that the incorrect device driver is loaded. Check the device driver version through Windows Device Manager by right-clicking on the device, select 'Properties', then 'Device Driver' tab.

You will need to check up-to-date TSM and product information to see which is the correct device driver, but generally, IBM tape devices use IBMTape except that 3494 libraries use ibmatl. Non-IBM tape drives and libraries typically use the tsmscsi device driver that comes packaged with TSM Server and Storage Agent. A non-IBM library that uses IBM tape drives will typically use tsmscsi for the library and IBMtape for the drives.

Make sure TSM is DOWN before you do anything with the drivers, even on the Lanfree clients (put the Storage Agent on the Lanfree client in Disabled or Manual mode until all drivers are in, return the Storage Agent to Auto when all is well).

To install the IBMtape driver, run install-exclusive.exe.

To install the tsmscsi, from Device Manager, right-click on the device, select Update Driver -> Install from a list or specific location (Advanced) -> Don't search. I will select the driver to install -> Select the IBM Tivoli Storage Manager device driver -> Continue Anyway.

Windows device names incorrect

Windows device names can change after a reboot, use tsmdlst to check them. tsmdlst runs quicker if the TSM service is down, so it's often a good idea to run it as a routine before you start the service.
Open up a DOS command line navigate to C:\Program Files\tivoli\tsm\console, and run tsmdlst
You should see output something like the following. The example below shows a single library and drive. 

C:\Program Files\Tivoli\TSM\console>tsmdlst
Tivoli Storage Manager -- Device List Utility
 
Computer Name:      z999UPS01
OS Version:         5.2
OS Build #:         3790
TSM Device Driver:  TSMScsi - Not Running
 
2 HBAs were detected.
 
Manufacturer        Model          Driver      Version            Firmware Descr
iption
--------------------------------------------------------------------------------
--------------------------------
QLogic Corporation  QLA2340        ql2300.sys  9.1.4.15           3.03.21  QLogi
c QLA2340 Fibre Channel Adapter
QLogic Corporation  QLA2340        ql2300.sys  9.1.4.15           3.03.21  QLogi
c QLA2340 Fibre Channel Adapter
 
TSM Name     ID   LUN  Bus  Port SSN                   WWN               TSM Typ
e  Device Identifier
--------------------------------------------------------------------------------
mt0.0.0.2    0    0    0    2    1110018309            500308C1424E8001  LTO
    IBM     ULT3580-TD2     73V1
lb0.1.0.2    0    1    0    2    0000013201681000      500308C1424E8001  LIBRARY
    IBM     ULT3583-TL      6.10


Use the query path command to check if paths are using the wrong devices, and if so then correct them with the update path command 

To query library paths:
   q path destt=libr f=d 


 To query tape paths for one server:
   q path server-name f=d


To update a library path: 
   update path server-name lib-name srct=server destt=libr 
      device=<lb#.#.#.#> online=yes


To update a tape path: 
   update path server-name drive-name srct=server destt=drive libr=lib-name 
      device=<mt#.#.#.#> online=yes

After a Windows server reboot, many people as a matter of routine will run the following process, just to make sure no problems appear. They have most of these action scripted, so it's a simple case of running the script for each step.

 •    Run tsmdlst
•    Start TSM server with client sessions disabled
•    delete tape and library paths
•    delete tape drives
•    delete tape library
•    generate path definitions from tsmdlst output (various ways to do this with scripts or spreadsheets
•    define tape library
•    define library path
•    define tape drives
•    define tape paths
•    checkin scratch tapes
•    checkin private tapes
•    enable client sessions
Subscribe to: Posts (Atom)