Express Reorg: How to save your weekend Laura Rochon Ajilon Consulting

Laura Rochon Started working with CA-IDMS in 1983 Joined Cullinet in 1985 Left CA in 1992, has been consulting ever since Joined Ajilon Consulting in January 2005 IUA Board member since 1998 CQUI President

Agenda Overview of Unload/Reload Overview of Express Reorg Express Reorg Execution Considerations Test results Conclusion

Overview of Unload/Reload UnloadReload DB

Overview of Unload/Reload UnloadSORT1DBL2 SORT2DBLX SORT3DBL3SORT4DBL4 ReloadUnload DB

Overview of Express Reorg DB UnloadReload DB Unload Reload Control file

Overview of Express Reorg New functionality: – Add or remove fields from index key – Change order of set

Overview of Express Reorg 4 major phases: – SETUP Allocates control file Specifies what is to be done – UNLOAD Unloads data – RELOAD Reloads data – CLEANUP Delete work files (but not control file) Unlock areas

Overview of Express Reorg Unload: – Area sweep for most records, except: Record stored VIA system-owned index VIA record CALC records with duplicate option

Overview of Express Reorg Unload (cont’d) – Records assigned target page CALC records VIA records Direct records

Overview of Express Reorg REORG slicing: – Based on DIVIDE PROCESSING n WAYS – Generally, each area is divided symmetrically – If total size of all areas is less than 200 pages, 1 slice – Areas and subareas that contain only system indexes not divided and assigned to a slice

Overview of Express Reorg REORG slicing (cont’d) – User-owned indexes are processed with the slice of the owner record – System-owned indexes assigned to index group and processed indepently of slices

Overview of Express Reorg Reorg processing is divided into tasks and grouped into phases RELOAD phase is divided into – Reload phases 1 thru 6 Each reload task in a given task must successfully complete before processing can move to next Reload phase – Rebuild phases 1 thru 4

Overview of Express Reorg Unload Rebuild 1 REORG: Reload 1 Reload 2 Reload 3 Reload 4 Reload 5 Reload 6 Rebuild 2 Rebuild 3 Rebuild 4 Cleanup Reload Unload Cleanup DB

Express Reorg Execution Parameters JCL Work Files Output Processing Job submission

Express Reorg Parameters SEGMENT source-segment USING source-ssc AREA area RELOAD INTO target-db USING target-ssc DMCL dmcl-name DIVIDE PROCESSING n WAYS REUSE WORKFILES AS SORTEXIT SHARE NOTIFY notify-interval REORG setup options :

Express Reorg Parameters REORG execution options: CLEANUP STOP AFTER SETUP UNLOAD RELOAD NO CREATE UNLOAD WORKFILES RELOAD ALL SUBMIT

Express Reorg Parameters REORG other parameters : CLEANUP STATUS REPORT ONLY

Express Reorg JCL //BCF EXEC PGM=IDMSBCF,REGION=0M //STEPLIB DD DSN=idms.loadlib,DISP=SHR //SYSIDMS DD * Sysidms parameters //SYSLST DD SYSOUT=* //SYSPRINT DD SYSOUT=* //SORTMSG DD SYSOUT=* //SORTWRKnn DD UNIT=unit,SPACE=(CYL,(mmm,nnn)) //SYSIPT DD * Reorg parameters //RORGCTL DD DSN=rorgctl-file,DISP=SHR //RORGJCL DD DSN=jcl-file,DISP=SHR

Express Reorg JCL RORGCTL file: – Describes the specifics of the REORG operation – Contains status information – Fixed-length, LRECL=4096 – Initialized during setup – Should not have DISP=OLD or DISP=NEW

Express Reorg JCL RORGJCL file: – Contains JCL for parallel jobs – Job name should be short enough to append 1+ characters – 1 step: IDMSBCF with following syntax: REORG; – Should have DB files and RORGCTL file

Express Reorg Work Files The number of work files proportional to number of jobs submitted Work files can be manually allocated Work files allocated automatically by using new command CREATE DSMODEL Can specify several CREATE DSMODEL

Express Reorg Work Files REORG uses following DDNAME prefixes: – WU -- unload/reload work files – WI -- Index Rebuild work files – WS-- Sort output work files – WD-- DBKEYS work files

Express Reorg Work Files CREATE DSMODEL creates a temporary model of data set attributes Must include information : – Where to create work file – Space – Blocksize

CREATE DSMODEL

Express Reorg Work Files Sizing work files: – Very hard to estimate – Run REORG STOP AFTER UNLOAD with dummied work files.

Express Reorg Work Files Number of files grows exponentially: example of WU* files: Divide n waysNo reuseReuse workfiles

Output

Output

Output

Output

Output

Output

Output

Output

Output

Express Reorg Processing If Target files same as Source files – REORG STOP AFTER UNLOAD – If not SHARE, UNLOCK AREAS – FORMAT files – REORG with STOP AFTER CLEANUP & SUBMIT – Backup – IDMSDBAN (optionally) – PRINT SPACE (optionally)

Express Reorg Processing If Target files with different DDnames – Format new files – REORG with STOP AFTER CLEANUP – Backup – PRINT SPACE – IDMSDBAN (optionally)

Express Reorg Processing If Target files different but same DDnames – Format new files – REORG with STOP AFTER UNLOAD (REORG jcl and jcl in RORGJCL have old DDnames) – REORG with STOP AFTER CLEANUP and SUBMIT (REORG jcl and jcl in RORGJCL have new files)

Job Submission Automatic job submission Manual job submission Restarting REORG

Automatic Job Submission Jobs automatically submitted – at end of SETUP phase – If SUBMIT parameter is specified REORG will submit – 1 job for each slice – 1 job for each index group Maximum jobs submitted = 2x no slices

Manual Job Submission Can manually submit REORG job(s) Job will examine control file and determine if there is work to be done If no work, waits until some other job completes a task If work is needed, it performs that work

Restarting REORG If a REORG job fails, other jobs will try to restart the task that failed If problem is not temporary (work file too small), problem needs to be corrected, then REORG can be submitted with SUBMIT. In some severe cases, you might need to restart REORG from the beginning

Considerations Relative REORG performance Efficiency of the reorganized database Work files

Relative REORG performance Usually runs in less time than normal UNLOAD/RELOAD, however….. Best to try it out first

DB Efficiency DB might not be as efficient as reorganized by UNLOAD/RELOAD REORG reloads in forwards fashion whereas UNLOAD/RELOAD reloads in backwards fashion REORG can result in more CALC & VIA overflow especially if long sets

Work Files Work files HAVE to be on disk Hard to estimate their size

Tests Results Unload/reload Reorg 2 waysReorg 3 waysReorg 4 waysReorg 5 ways 50.5% 58.4% 63.8% 64.5% TSTDML (46,116 pages)

Tests Results Unload/reload Reorg 2 waysReorg 3 ways 27.2% 44.0% 1 Area unloaded (120,000 pages going to 147,000 pages) 1 Area affected (45,000 pages)

Tests Results 1 Area unloaded (427,720 going to 428, ) 3 Areas affected (39,000 pages 160,000 pages 294,000 pages) % 51.1% Unload/reload Reorg 3 ways Reorg 5 ways

Conclusion Can reduce the time to reorganize a database up to 50-60% Easy to use Needs lots of disk space