1. Analysis of Virtual Tape Subsystems Ned Diehl The Information Systems Manager, Inc. ned.diehl@perfman.com www.perfman.com 610-865-0300 NCACMG Vienna, Virginia 5 June 2002

2. Trademarks (Omissions are unintentional) ISM PerfMan IBM Parallel Sysplex RMF OS/390 z/OS Magstar VSM StorageTek HSC Nearline

3. Objectives Discuss sources of performance analysis and capacity planning data for virtual tape subsystems (VT) in an OS/390 environment Primary focus on hardware implementations –IBM VTS –StorageTek VSM Present graphical examples System rather than application focus

4. Contents Introduction Virtual Tape Structure Data Sources Key Performance Metrics Recommendations Summary References

5. Introduction Solves many traditional tape problems –Small data sets –Low activity open data sets –Allocation wait for tape drives Weakest with “good” tape activity –Full volumes –High transfer rates Exploits high capacity tapes –VT helps remove implementation impediments

6. Introduction Consider disaster recovery –VTS Peer-to-Peer –Clustered VTSS Configuration Mount management considerations –Deferred mount should generally be avoided –Premount can be desirable Not yet everything for everyone – but getting there –Fewer problem datasets with current implementations

7. VT Structure General Looks (almost) like 3490E to host Combination of hardware and software –RISC processor –Library of high capacity tapes –RAID DASD buffer or CACHE –Tape drives Amount of host software varies with implementation

8. VT Structure Logical Components

9. VT Structure VTS - Physical Components

10. VT Structure VSM - Physical Architecture HSC Control Path CDS VTCS VSM Control & Data Path Migrate/ Recall VTSS MVC MVC MVC VTV VTV VTV VTV VTV Physical tape drives Physical tape volumes Physical Libraries & slots Virtual tape drives Virtual tape volumes

11. Data Sources Application –SMF 30, 72, 14, & 15 Hardware –SMF 94 - tape library statistics –STK user SMF record –SMF 14 & 15 – data set activity –SMF 21 – error statistics by volume –RMF 73 - channel path activity –RMF 74-1 - device activity –RMF 78-3 - I/O queuing –Real time controller interface

12. Data Sources SMF 94 Easy to work with though not flexible Hourly summary from VTS or Library –Difficult to synchronize (do not use SMF94HHI) –Identical data to all attached recording images Base segments reflect all library statistics –ATL, mount, dismount, eject, & insert –Native and VTS activity Identical data in multiple records with a different serial number if VTS and native drives attached VTS segments reflect single VTS –VTS, import / export, enhanced statistics

13. Data Sources SMF 94 One record produced for each –Library with native drives, identified by ATL serial (SMF94SNO). Contains base segments. –VTS, identified by VTS serial (SMF94SNO & SMF94VLS). Contains base and VTS segments. With peer-to-peer, three (local) or four (remote) VTSs involved –Only two have physical tape –No easy way to associate them Generally consistent with RMF –RMF allows finer detail of comparable metrics

14. Data Sources SMF 94 Data Issues Hour index (SMF94HHI) is interval end Drives available (SMF94VTA) not always reset after service –Drives used (SMF94VTV, VTN, & VTX) can be greater than available Alignment problem with early Import/Export statistics (SMF94ACA & SMF94ACB) Duplicate library data with a different serial number if VTS and native drives attached Average age in cache (SMF94VCA) has had changed definitions

15. Data Sources SMF 94 Data Issues Tape data transfer (SMF94VTR & SMF94VTW) reported as zero in some recent samples Max cache volume age (S94MTVCA) recorded as seconds, documented as minutes Backstore compression ratio (S94BSRAT) sometimes reported as less than one Recall throttle percent (S94RCPRT) sometimes greater than 100

16. Data Sources SMF 94 Data Issues Reference Flash 10124 dated 20 November 2001. Recalls might be recorded as hits. Effected metrics: –S94MAXCH Maximum Cache Hit Mount Time –S94AVGCH Average Cache Hit Mount Time –SMF94VMH Number of Cache Hit Mounts –S94MAXRM Maximum Recall Mount Time –S94AVGRM Average Recall Mount Time –SMF94VMS Number of Recall Mounts –SMF94VPS Number of Physical Mounts for Recall

17. Data Sources STK User Record Complex to work with –Many subtypes –Repeating segments Allows detail analysis –Configuration info –CU and device busy –Volume level data –VTSS interval of 15 minutes –Grouping and summarization required –Some logical metrics require multiple subtypes

18. Data Sources STK User Record Interval metrics by VTSS (not image) to each recording OS/390 –Subsystem, channel interface, and RTD performance –Values vary with OS/390 recording time –Logically identically but physically different Event metrics recorded once –VTV mount, dismount, delete, migrate, recall, movement, replicate –RTD mount, dismount, vary –MVC status

19. Data Sources STK Data Issues Must process event records from all recording images Mount events not necessarily recorded to original requesting image Dismount events not necessarily recorded to same image as mount Might be extra mounts or dismounts Calculated times might be very large or negative Some time values in complex formats

20. Data Sources STK Data Issues RMF consistency varies –Most metrics close across multiple samples I/O Rates Connect time Allocated time Total mount time –Significant variation with mount counts and thus average mount time Probably a code level issue

21. Key Performance Metrics Application –Service levels Hardware –Data transfer –Device usage –Virtual mount time –Mount miss (recall) rate –Storage usage –Tape volume age (in DASD buffer or cache)

22. Key Performance Metrics Application Performance Should have reasonable and measurable performance objectives Objectives will vary with –Installation –Application –VT –Time & day

23. Key Performance Metrics Data Transfer Good for reporting work performed –Mounts and I/O rates also options Saturation varies with environment SMF 14, 15, and 21 provide read and write RMF 73 provides read and write for FICON. VTS provides data transfer –Host read (SMF94VBR) and write (SMF94VBW) are good throughput indicators –Read (SMF94VTR) and write (SMF94VTW) between cache and real tapes

24. Key Performance Metrics Data Transfer VTSS provides many data transfer metrics but assumptions are required –RTD bytes read (SMF20BTR) and written (SMF20BTW) –RTD connect (SMF20DCT) and utilization (SMF20DUT) –Host and RTD channel interface busy (SMF11CUB) –VTV dismount has VTV size (SMF14VSZ) Probably best throughput indicator

25. Key Performance Metrics Data Transfer

27. Key Performance Metrics Device Usage RMF 74-1 provides virtual by device or group. –Allows separation of mount and allocation components (e.g. connect, pend, wait). VTS provides minimum, maximum, average, and configured for both virtual and physical devices. All values are integers. VTS average (SMF94VTV) and maximum (SMF94VTX) physical are good for trending While not normally a problem, virtual device use should not be ignored.

28. Key Performance Metrics Device Usage Difficult to calculate VTSS minimum, maximum, average, and available for either virtual or physical devices –Mount, dismount, and vary subtypes must be grouped and summarized –RTDs can be statically shared (operator command) by multiple VTSSs and OS/390 VTSS RTD connect and utilization, which were previously discussed, are useful

29. Key Performance Metrics Device Usage

32. Key Performance Metrics Device Usage (RMF)

33. Key Performance Metrics Virtual Mount Time RMF 74-1 provides average mount time by device or group VTS provides minimum, maximum, and average virtual and physical mount times Average virtual (SMF94VRA) is good for trending High maximum virtual (SMF94VRX) often correlates with problems

34. Key Performance Metrics Virtual Mount Time VTSS mount time calculated from mount event records (SMF13MET - SMF13MST) –Min, max, average and counts require grouping and summarization –Average might be distorted by extra mounts VTSS provides several potentially interesting identifiers –Job, step, DSN, VTV management class –Scratch, existing

35. Key Performance Metrics Virtual Mount Time Mount times tend to increase with: –Reduced tape volume age on DASD (SMF94VCA) –Increased mount miss rate (SMF94VMS) –Increased average real drives mounted (SMF94VTV) Published targets: –Daily average less than 30 seconds –Hourly average less than 300 seconds –Maximum less than 900 seconds

36. Key Performance Metrics Virtual Mount Time (RMF)

37. Key Performance Metrics Virtual Mount Time

38. Key Performance Metrics Max Virtual Mount Time

39. Key Performance Metrics Average Virtual Mount Time

40. Key Performance Metrics Mount Miss Rate VTS provides mounts by type –Fast ready (SMF94VFR) –Specific mount hits (SMF94VMH) –Specific mount misses (SMF94VMS) Related to average time on DASD (SMF94VCA) and application cycles Published target of miss percentage less than 20% –Miss % = 100 * Misses / Total Virtual Mounts –Miss % = 100 * VMS / (VFR + VMH + VMS)

41. Key Performance Metrics Mount Miss Rate VTSS mount hit indicator added in recent update (SMF13RCI) –Scratch mounts indicated (SMF13VMT) –If older data, could assume a hit if calculated mount time less than a threshold

42. Key Performance Metrics Mount Miss Rate

43. Key Performance Metrics Specific Mount Miss Rate

44. Key Performance Metrics Mount Miss Rate

45. Key Performance Metrics Storage Usage VTS provides estimate of data on stacked volumes (SMF94VBA) and available capacity on empty cartridges (SMF94VEC) –Reconciliation will probably free space High recalls (SMF94VMS) and low tape volume age on DASD (SMF94VCA) indicate need for more DASD More DASD can relieve saturated physical tape drives

46. Key Performance Metrics Storage Usage

47. VTSS provides cache and DASD metrics –Configuration info DASD capacity (SMF10TCP), cache & NVS size, channels –Used (SMF10FCP) and contiguous (SMF10CFP) free DASD space No easy way to report VTSS MVC metrics –Must produce & process MVC status records

48. Key Performance Metrics Storage Usage

49. Key Performance Metrics Tape Volume Age VTSS delete reference age can sometimes be related to application cycles –SMF delete time stamp – SMF15LTR –Consider selective summarization –Separate migrate immediate delete Subtype 18 with SMF18MTI set, closely followed by related Subtype 15 (same virtual VOLSER) Occasionally very high values

50. Key Performance Metrics Tape Volume Age

51. Recommendations Report and track application service levels Host data transfer is good workload measure Virtual mount time (average and maximum) is good response measure Keep history Vary period and interval size with need –Longer (e.g monthly) for trends –Shorter (e.g. daily) for glitches –Hour is good interval for most purposes

52. Summary Many existing analysis tools useful Some valuable data only available in unique VT records Packages probably needed for VT metrics VT will become more common and important

53. References WP100146 - IBM Magstar Model B16 Virtual Tape Server Elements of Performance, IBM WP100157 - VTS and Logical Paths, IBM WP100199 - IBM Magstar 3494 Peer-to-peer VTS Performance White Paper, IBM WP100187 - IBM Magstar 3494 Virtual Tape Server Performance White Paper, IBM WP100239 - IBM TotalStorage Virtual Tape Server Performance, IBM WP100243 - IBM TotalStorage Peer-to-peer Virtual Tape Server Performance, IBM SG24-4917 - IBM Virtual Tape Server & Enhancements to Magstar, IBM SG24-2229 - IBM Magstar VTS: Planning, Implementing, and Monitoring, IBM SG24-4409 - Guide to Sharing and Partitioning IBM Tape Library Data Servers, IBM SC28-1950 - OS/390 RMF Report Analysis, IBM GC28-1783 - OS/390 System Management Facilities (SMF), IBM

54. References Cheryl Watson’s Tuning Letter 2000, N0.5 - Focus: VTS Measurements Cheryl Watson’s Tuning Letter 2001, N0.6 - Focus: StorageTek VSM Introduction to Virtual Storage Manager, StorageTek StorageTek Virtual Tape Control System - Installation, Configuration & Administration Guide, StorageTek PerfMan for Tape Libraries, ISM PerfMan for OS/390, ISM CMG Proceedings Share Proceedings Vendor WEB sites