1. WAVV 2007, Green Bay, WI VSE/VSAM – Inside & Out John Mycroft, Software Developer CSI International www.e-vse.comjohnm@e-vse.com

2. WAVV 2007, Green Bay, WI Acknowledgement With grateful thanks to Dan Janda, The Swami of VSAM, from whom most of this presentation was stolen With grateful thanks to Dan Janda, The Swami of VSAM, from whom most of this presentation was stolen

3. WAVV 2007, Green Bay, WI Abstract This presentation gives an overview of VSAM & its components. We take a look at what a VSAM file really looks like and how to soup up its performance. This presentation gives an overview of VSAM & its components. We take a look at what a VSAM file really looks like and how to soup up its performance. We also look at some common mistakes and how to avoid them. We also look at some common mistakes and how to avoid them. This presentation and its materials are copyrighted and developed by John Mycroft from a presentation originally copyrighted by Dan Janda. Permission is granted for WAVV to reproduce this presentation for distribution to its members at no charge. This presentation and its materials are copyrighted and developed by John Mycroft from a presentation originally copyrighted by Dan Janda. Permission is granted for WAVV to reproduce this presentation for distribution to its members at no charge. Trademarks: Trademarks: IBM, VSE, VSE/ESA, zVSE, CICS & DL/I are trademarks or registered trademarks of the IBM Corporation IBM, VSE, VSE/ESA, zVSE, CICS & DL/I are trademarks or registered trademarks of the IBM Corporation The Swami of VSAM is a trademark of Dan Janda. The Swami of VSAM is a trademark of Dan Janda.

4. WAVV 2007, Green Bay, WI VSE/VSAM Overview Virtual Storage Access Method Virtual Storage Access Method For disk files For disk files Sequential – “Entry Sequence Dataset” or ESDS Sequential – “Entry Sequence Dataset” or ESDS Begin at the beginning, go on til you get to the end and then stop Begin at the beginning, go on til you get to the end and then stop Indexed – “Keyed Sequence Dataset” or KSDS Indexed – “Keyed Sequence Dataset” or KSDS Process by key or sequentially or a mixture Process by key or sequentially or a mixture Direct – “Relative Record Dataset” or RRDS (fixed) or VRDS (variable) Direct – “Relative Record Dataset” or RRDS (fixed) or VRDS (variable) Calculate a record’s location in the file to access it Calculate a record’s location in the file to access it Alternate index (AIX) – gives an alternative route to a KSDS Alternate index (AIX) – gives an alternative route to a KSDS Allows unique & non-unique keys Allows unique & non-unique keys

5. WAVV 2007, Green Bay, WI VSE/VSAM Functional areas Catalog Catalog Volume & file information Volume & file information Usage statistics Usage statistics Disk space management Disk space management Space allocation including secondary allocations Space allocation including secondary allocations VSAM and VSAM/SAM files VSAM and VSAM/SAM files System files System files Libraries Libraries

6. WAVV 2007, Green Bay, WI VSE/VSAM Functional areas Integrity Integrity Performance Performance Data transfer size Data transfer size Buffering Buffering Backup / restore Backup / restore File sharing between jobs and systems File sharing between jobs and systems

7. WAVV 2007, Green Bay, WI Processing a VSAM file Sequentially (ESDS) Sequentially (ESDS) Forward or backward Forward or backward Keyed access (KSDS) Keyed access (KSDS) Direct by full or partial (generic) key Direct by full or partial (generic) key Sequentially, forward or backward Sequentially, forward or backward Skip sequential, forward or backward Skip sequential, forward or backward Addressed access (RRDS, VRDS) Addressed access (RRDS, VRDS) Direct, by record address Direct, by record address Sequential & skip sequential Sequential & skip sequential Alternate Index Access Alternate Index Access Same as keyed access Same as keyed access Also direct access by non-unique key Also direct access by non-unique key

8. WAVV 2007, Green Bay, WI How VSAM stores data We’re going to look at We’re going to look at How VSAM stores records logically on disk How VSAM stores records logically on disk Performance considerations Performance considerations How VSAM physically stores data on disk How VSAM physically stores data on disk Disk space usage calculations Disk space usage calculations Optimizing disk capacity Optimizing disk capacity Performance considerations Performance considerations VSAM jargon VSAM jargon Control Interval Control Interval Control Area Control Area CI & CA splits CI & CA splits Freespace Freespace RDF, CIDF RDF, CIDF

9. WAVV 2007, Green Bay, WI VSAM Jargon Control Interval (CI) Control Interval (CI) “Smallest unit of data transfer between main & disk storage” “Smallest unit of data transfer between main & disk storage” In other words, when you read a record, VSAM reads the whole CI that contains that record In other words, when you read a record, VSAM reads the whole CI that contains that record Think of it as the same as a block of records in a sequential file if you like (though it’s laid out differently) Think of it as the same as a block of records in a sequential file if you like (though it’s laid out differently) A CI can initially contain 1 or more records A CI can initially contain 1 or more records More can be inserted More can be inserted Some or all can be deleted Some or all can be deleted When you try to add a new record to a CI with no room, a “CI split” takes place – more about that later When you try to add a new record to a CI with no room, a “CI split” takes place – more about that later

10. WAVV 2007, Green Bay, WI Layout of a control interval ALL VSAM FILES ARE VARIABLE LENGTH ALL VSAM FILES ARE VARIABLE LENGTH Even if all the records are the same size Even if all the records are the same size Rec 1 – Rec n 1 to n logical records of any length Rec 1 – Rec n 1 to n logical records of any length FreespaceUnused space in CI for inserting records or making existing records longer FreespaceUnused space in CI for inserting records or making existing records longer RDFs3 byte record descriptor field RDFs3 byte record descriptor field ESDS/KSDSone per record length, one for all consecutive records of same length ESDS/KSDSone per record length, one for all consecutive records of same length RRDSone per numbered record slot RRDSone per numbered record slot CIDF4 byte Control Interval Descriptor Field CIDF4 byte Control Interval Descriptor Field Rec 1 Rec 2 Rec 3 Rec … FreespaceRDFsCIDF

11. WAVV 2007, Green Bay, WI Control Area (CA) CA size is the smallest of : CA size is the smallest of : One cylinder or The size of the primary allocation The size of the secondary allocation The number of CIs per CA depends on the device and the CI and CA sizes It is generally a good idea to go for the biggest CA possible A CA is a group of CIs. In a KSDS, all the data CIs in a CA are indexed by one index CI CI 0 CI 1 CI 2 CI 3 CI 4 CI 5 CI 6 CI 7 CI 8 CI 9 CI10CI11CI12CI13CI14CI15CI16CI17CI18CI19 CI20CI21CI22CI23CI24CI25CI26CI27CI28CI29

12. WAVV 2007, Green Bay, WI Index Control Interval (Index CI) CI 0 CI 1 CI 2 CI 3 CI 4 CI 5 CI 6 CI 7 CI 8 CI 9 CI10CI11CI12CI13CI14CI15CI16CI17CI18CI19 CI20CI21CI22CI23CI24CI25CI26CI27CI28CI29 A CI in an index containing pointers to The next level in the index or The Data CI in the CA – this is referred to as a Sequence Set CI Index CI

13. WAVV 2007, Green Bay, WI Index and data structure Balanced tree Balanced tree Sparse index Sparse index Always just 1 high-level index CI Always just 1 high-level index CI There can be 0 to many intermediate level index CIs There can be 0 to many intermediate level index CIs There can be one or more low-level (sequence set) index CIs. There can be one or more low-level (sequence set) index CIs. If there is only 1 sequence set CI, it is also the high-level index CI If there is only 1 sequence set CI, it is also the high-level index CI

14. WAVV 2007, Green Bay, WI And now the bit you’ve all been waiting for……

15. WAVV 2007, Green Bay, WI Performance rules of thumb Use largest data CI possible, especially for sequential work Use largest data CI possible, especially for sequential work Use as small an index CI as you can (but not too small!) Use as small an index CI as you can (but not too small!) Use large data CA – allocate primary and secondary as at least 1 cylinder Use large data CA – allocate primary and secondary as at least 1 cylinder Avoid too many extents / allocations Avoid too many extents / allocations

16. WAVV 2007, Green Bay, WI Allocation calculations CI freespace = CI freespace = CI Size * Freespace % Number of records per CI Number of records per CI “Fixed” length: “Fixed” length: (CI Size -10 –Freespace) / LRECL (CI Size -10 –Freespace) / LRECL Variable length: Variable length: (CI Size -7 –Freespace) / (Average LRECL +3) (CI Size -7 –Freespace) / (Average LRECL +3)

17. WAVV 2007, Green Bay, WI Allocation calculations Calculate Freespace in each CA Calculate Freespace in each CA Get number of CIs per CA from LISTCAT or device characteristics (3390, 12 x 4K CIs/track, 180/cyl) Get number of CIs per CA from LISTCAT or device characteristics (3390, 12 x 4K CIs/track, 180/cyl) CA freespace = No of CIs per CA * CA Freespace %, rounded up CA freespace = No of CIs per CA * CA Freespace %, rounded up Number of CIs loaded per CA = Number of CIs loaded per CA = CIs per CA – CA freespace Number of records loaded per CA = Number of records loaded per CA = Loaded CIs in CA * No of recs in CI

18. WAVV 2007, Green Bay, WI VSAM Catalogs Exactly one master catalog Exactly one master catalog Assigned at IPL with DEF CAT or Assigned at IPL with DEF CAT or DEFINE MCAT IDCAMS command DEFINE MCAT IDCAMS command User catalogs – 0 to many User catalogs – 0 to many No more than 1 per volume No more than 1 per volume Catalog can own multiple spaces on a volume Catalog can own multiple spaces on a volume Many catalogs can own space on a volume Many catalogs can own space on a volume

19. WAVV 2007, Green Bay, WI VSAM Catalogs Catalog contains :- Catalog contains :- Self-describing records Self-describing records User catalog pointers User catalog pointers Volume definitions Volume definitions Space definitions Space definitions Cluster (file) definitions Cluster (file) definitions Component (data, index) definitions Component (data, index) definitions AIX & Path definitions AIX & Path definitions

20. WAVV 2007, Green Bay, WI Catalog recommendations Use naming conventions Use naming conventions Name Cluster, Data and Index components explicitly Name Cluster, Data and Index components explicitly Use partition / system independent names where applicable Use partition / system independent names where applicable Separate Separate Files seldom defined or deleted Files seldom defined or deleted Files often defined or deleted Files often defined or deleted Online critical files Online critical files Batch files Batch files Multiple baskets – all the eggs won’t get broken Multiple baskets – all the eggs won’t get broken

21. WAVV 2007, Green Bay, WI More recommendations Don’t use recoverable catalogs Don’t use recoverable catalogs Hangover from 2314 / 3330 Hangover from 2314 / 3330 Backup is vastly better Backup is vastly better IDCAMS, Faver, Maxback, Dr D, user-written … IDCAMS, Faver, Maxback, Dr D, user-written …

22. WAVV 2007, Green Bay, WI CI & CA splits and freespace You try to insert a record in a CI or extend a record already there You try to insert a record in a CI or extend a record already there If there is enough free space in the CI, everyone moves up, record is inserted and CI rewritten If there is enough free space in the CI, everyone moves up, record is inserted and CI rewritten BUT what if there isn’t enough free space???? BUT what if there isn’t enough free space????

23. WAVV 2007, Green Bay, WI CI & CA splits CI split – 4 physical IOs CI split – 4 physical IOs Set “Split in progress”, write CI Set “Split in progress”, write CI Move half of records to new CI & write it Move half of records to new CI & write it Update sequence set, write index CI Update sequence set, write index CI Erase moved records from old CI, turn off “Split in progress”, wite old CI Erase moved records from old CI, turn off “Split in progress”, wite old CI BUT….. BUT…..

24. WAVV 2007, Green Bay, WI Failure in CI split System failure System failure Corrected next time CI is updated Corrected next time CI is updated No free CI in the CA No free CI in the CA CA split is needed CA split is needed Remember – 1 physical IO = 30,000 – 40,000 CPU instructions… Remember – 1 physical IO = 30,000 – 40,000 CPU instructions…

25. WAVV 2007, Green Bay, WI CA Split MANY physical reads and writes MANY physical reads and writes Set “Split in progress”, write sequence set CI Set “Split in progress”, write sequence set CI Maybe get new extent Maybe get new extent Format new CA at HURBA position Format new CA at HURBA position Read / write half of CIs to new CA Read / write half of CIs to new CA Write new sequence set CI for new CA Write new sequence set CI for new CA Update higher level index CIs Update higher level index CIs Erase moved CIs from old CA, write empty CIs Erase moved CIs from old CA, write empty CIs Write updated original sequence set CI Write updated original sequence set CI

26. WAVV 2007, Green Bay, WI Recommendations Don’t worry about CI splits Don’t worry about CI splits Avoid excessive CA splits by defining CA freespace Avoid excessive CA splits by defining CA freespace Don’t do a reorg just because you have done n CI / CA splits Don’t do a reorg just because you have done n CI / CA splits

27. WAVV 2007, Green Bay, WI To reorg or not to reorg? “We’ve done 1000 CA splits – better reorg!” “We’ve done 1000 CA splits – better reorg!” Inserts tend to be clustered Inserts tend to be clustered CI / CA split creates freespace where it is needed, allows faster inserts CI / CA split creates freespace where it is needed, allows faster inserts Reorg gets rid of freespace, causing more CI / CA splits Reorg gets rid of freespace, causing more CI / CA splits

28. WAVV 2007, Green Bay, WI Recommendations Avoid frequent reorgs Avoid frequent reorgs Once a split has occurred, the processing cost has been paid Once a split has occurred, the processing cost has been paid Understand your application Understand your application 1 “hot spot” 1 “hot spot” Little distributed freespace – let it split Little distributed freespace – let it split Many hot spots Many hot spots Little distributed freespace – let it split Little distributed freespace – let it split Even distribution – no hot spots Even distribution – no hot spots Use distributed freespace Use distributed freespace

29. WAVV 2007, Green Bay, WI Freespace 3% of each CI is empty 5% of CIs in each CA are empty 3% of 2048 = 61 bytes = 0 records (or, at most, 1) 5% of 315 CIs per CA = 16 CIs

30. WAVV 2007, Green Bay, WI Freespace 3% CI freespace where CISZ=2048 and average LRECL=120 No room in this CI for an average length record

31. WAVV 2007, Green Bay, WI Altering freespace Initial freespace set via DEFINE eg 10% of CI and 5% of CA Initial freespace set via DEFINE eg 10% of CI and 5% of CA If inserts are clustered, consider If inserts are clustered, consider DEFINE with 0% freespace, then DEFINE with 0% freespace, then Load the file then Load the file then ALTER freespace to non-zero ALTER freespace to non-zero

32. WAVV 2007, Green Bay, WI Strings VSAM allows multiple concurrent processing e.g. VSAM allows multiple concurrent processing e.g. CICS transactions CICS transactions Browsing Browsing Updating Updating Placeholders (“strings”) hold file location info Placeholders (“strings”) hold file location info

33. WAVV 2007, Green Bay, WI Shared / non-shared resources Non-shared resources (NSR) Non-shared resources (NSR) Each string has its own buffers Each string has its own buffers Multiple copies of a CI may be in memory Multiple copies of a CI may be in memory Works well for batch Works well for batch Local Shared Resources (LSR) Local Shared Resources (LSR) Many strings share a pool of buffers Many strings share a pool of buffers Only 1 copy of a CI in the pool Only 1 copy of a CI in the pool Ideal for online Ideal for online

34. WAVV 2007, Green Bay, WI Recommendations - NSR Non-shared resources Non-shared resources Each string must have enough index buffers Each string must have enough index buffers Bad – 1 buffer (old default) Bad – 1 buffer (old default) OK – 1 buffer per index level (new default) OK – 1 buffer per index level (new default) Good – enough buffers for all high level indexes + 1 more Good – enough buffers for all high level indexes + 1 more Best – enough buffers to hold entire index Best – enough buffers to hold entire index

35. WAVV 2007, Green Bay, WI Recommendations - LSR Local Shared Resource buffers Local Shared Resource buffers Same index buffer needs as NSR (buffers are per pool, not per string) Same index buffer needs as NSR (buffers are per pool, not per string) Monitor VSAM LSR stats to make sure BUFNI keeps up with index growth Monitor VSAM LSR stats to make sure BUFNI keeps up with index growth Monitor data buffers for high hit rates Monitor data buffers for high hit rates

36. WAVV 2007, Green Bay, WI IO with NSR VSAM uses chained IO to read ahead and write behind VSAM uses chained IO to read ahead and write behind Better to read many CIs in one IO Better to read many CIs in one IO Block big Block big Large CI sizes Large CI sizes Be aware that VSAM will split CIs into smaller blocks to save space Be aware that VSAM will split CIs into smaller blocks to save space Eg 3390 with 32K CI gets written as 2 x 16K blocks giving 1.5 CIs = 48K/track Eg 3390 with 32K CI gets written as 2 x 16K blocks giving 1.5 CIs = 48K/track Buffer big Buffer big ½ to 1 cyl of BUFND to minimize IO ½ to 1 cyl of BUFND to minimize IO

37. WAVV 2007, Green Bay, WI IO with LSR VSAM reads 1 CI at a time, even for sequential processing VSAM reads 1 CI at a time, even for sequential processing

38. WAVV 2007, Green Bay, WI Monitor your stats LISTCAT before and after critical job LISTCAT before and after critical job Data & Index EXCPs – the fewer the better. Index EXCPs should be close to number of index CIs. Data & Index EXCPs – the fewer the better. Index EXCPs should be close to number of index CIs. Job Accounting data Job Accounting data IO count by device IO count by device Overal CPU & IO activity Overal CPU & IO activity CICS stats CICS stats Shows logical / physical IO counts by file Shows logical / physical IO counts by file LSR pool hits and misses LSR pool hits and misses VSAM buffer stats – in VSE/ESA examples doc VSAM buffer stats – in VSE/ESA examples doc LSR is in 31 bit – use LOTS but don’t page LSR is in 31 bit – use LOTS but don’t page

39. WAVV 2007, Green Bay, WI Sharing VSAM datasets VSAM can share files among partitions VSAM can share files among partitions And among VSE systems And among VSE systems BUT BUT TANSTAAFL (Robert Heinlein) TANSTAAFL (Robert Heinlein) Sharing is not a performance option (Dan Janda) Sharing is not a performance option (Dan Janda) It’s your gun and your foot (Steve Huggins) It’s your gun and your foot (Steve Huggins)

40. WAVV 2007, Green Bay, WI Sharing VSAM datasets Sharing is based on Sharing is based on The type of sharing you ask for (SHAREOPTIONS) The type of sharing you ask for (SHAREOPTIONS) VSE Lock Table within a single VSE system VSE Lock Table within a single VSE system VSE Lock File when sharing across VSE systems VSE Lock File when sharing across VSE systems VSE sharing mechanism is not compatible with zOS or zVM VSE sharing mechanism is not compatible with zOS or zVM

41. WAVV 2007, Green Bay, WI Sharing VSAM datasets Sharing at OPEN / CLOSE time Sharing at OPEN / CLOSE time Entries checked and placed in / removed from lock table Entries checked and placed in / removed from lock table If DASD volume is added as shared (ADD cuu,SHR), it is added to lock file If DASD volume is added as shared (ADD cuu,SHR), it is added to lock file VSE & VSAM allow concurrent processing to protect against concurrent updates messing up the file VSE & VSAM allow concurrent processing to protect against concurrent updates messing up the file

42. WAVV 2007, Green Bay, WI Sharing VSAM datasets Integrity classes – your choice Integrity classes – your choice NO INTEGRITY – VSE & VSAM provide no data protection: it’s all up to you. Your data can be messed up. NO INTEGRITY – VSE & VSAM provide no data protection: it’s all up to you. Your data can be messed up. WRITE INTEGRITY – VSE & VSAM protect against concurrent updates WRITE INTEGRITY – VSE & VSAM protect against concurrent updates READ INTEGRITY – VSE & VSAM make sure your programs always see the latest version of a record READ INTEGRITY – VSE & VSAM make sure your programs always see the latest version of a record The price The price Higher levels & broader scopes of integrity lead to more CPU and IO activity Higher levels & broader scopes of integrity lead to more CPU and IO activity

43. WAVV 2007, Green Bay, WI SHAREOPTIONS Ready – Fire – Aim Ready – Fire – Aim Set in DEFINE CLUSTER Set in DEFINE CLUSTER Get it wrong & be prepared to suffer Get it wrong & be prepared to suffer If a disk drive isn’t shared between VSEs, don’t ADD it with SHR as this causes lock file IO If a disk drive isn’t shared between VSEs, don’t ADD it with SHR as this causes lock file IO

44. WAVV 2007, Green Bay, WI SHAREOPTIONS & Locking SHR(1) 1 output OR many input SHR(1) 1 output OR many input External lock at OPEN, unlock at CLOSE External lock at OPEN, unlock at CLOSE SHR(2) 1 output AND many input SHR(2) 1 output AND many input External lock at OPEN, unlock at CLOSE External lock at OPEN, unlock at CLOSE SHR(3) No checking or locking SHR(3) No checking or locking Prepare for garbage data Prepare for garbage data SHR(4) Many output in one VSE & many input OPENs across all VSEs SHR(4) Many output in one VSE & many input OPENs across all VSEs External lock at OPEN, unlock at CLOSE External lock at OPEN, unlock at CLOSE External lock at access, unlock at release External lock at access, unlock at release SHR(4 4) Many output OPENs across all VSEs + many input OPENs SHR(4 4) Many output OPENs across all VSEs + many input OPENs Locks same as SHR(4) Locks same as SHR(4)

45. WAVV 2007, Green Bay, WI Alternate indexes (AIX) An AIX is a VSAM KSDS, acting as a “pointer file” for another file An AIX is a VSAM KSDS, acting as a “pointer file” for another file Target file (“Base Cluster”) can be Target file (“Base Cluster”) can be KSDS – pointers are KSDS key values KSDS – pointers are KSDS key values ESDS – pointers are Relative Byte Addrs ESDS – pointers are Relative Byte Addrs Great for multiple or non-unique keys Great for multiple or non-unique keys BUT BUT Processing via an AIX needs IO to both the AIX and to the base cluster Processing via an AIX needs IO to both the AIX and to the base cluster

46. WAVV 2007, Green Bay, WI Setting up an AIX DEFINE CLUSTER for base cluster DEFINE CLUSTER for base cluster DEFINE AIX for the alternate index DEFINE AIX for the alternate index Give base cluster’s name & alternate key Give base cluster’s name & alternate key Data & Index CI sizes Data & Index CI sizes DEFINE PATH DEFINE PATH Allows specifying of NOUPGRADE paths Allows specifying of NOUPGRADE paths BLDINDEX BLDINDEX Reads primary & alternate key info from base cluster Reads primary & alternate key info from base cluster Sorts into alternate key sequence Sorts into alternate key sequence Loads alternate index Loads alternate index

47. WAVV 2007, Green Bay, WI AIX recommendations To process the base cluster in AIX order, it is better to sort it and use the SORTOUT file To process the base cluster in AIX order, it is better to sort it and use the SORTOUT file Remember VSAM processes base clusters directly based on AIX values Remember VSAM processes base clusters directly based on AIX values Base cluster will need lots of index buffers for batch processing. Give Base cluster large BUFFERSPACE on DEFINE or ALTER Base cluster will need lots of index buffers for batch processing. Give Base cluster large BUFFERSPACE on DEFINE or ALTER

48. WAVV 2007, Green Bay, WI AIX and CICS “SPHERE” – a base cluster and all its AIXs related to it “SPHERE” – a base cluster and all its AIXs related to it Requirements Requirements Each sphere must be wholly within one LSR pool Each sphere must be wholly within one LSR pool Use Dataset Name Sharing Use Dataset Name Sharing In CICS 2.3, add BASE= to FCT entry for In CICS 2.3, add BASE= to FCT entry for Base cluster file entry Base cluster file entry Each related path file entry Each related path file entry This is automatic in CICS TS This is automatic in CICS TS SHR(2) is usually best SHR(2) is usually best Make sure your CICS and VSAM service is current! Make sure your CICS and VSAM service is current!

49. WAVV 2007, Green Bay, WI Contacting the presenter You can contact me by email at johnm@e-vse.com You can contact me by email at johnm@e-vse.com johnm@e-vse.com Dan Janda’s website has much of this info – http://business.epix.net/~theswami Dan Janda’s website has much of this info – http://business.epix.net/~theswami http://business.epix.net/~theswami And, if you want to find me this evening… And, if you want to find me this evening…

50. WAVV 2007, Green Bay, WI You’ll find me here