1. St. Petersburg 1703 Philadelphia 1682 George Potemkin Russia Dmitri Levin alphabroder DBAnalys To The Rescue! Disclaimer: The presentation is highly technical and dull. Listener need to know the database structure well. This is a complex topic and it requires a serious effort. ftp://ftp.progress-tech.ru/pub/Users/george/DBAnalys1_Boston.pptx ftp://ftp.progress-tech.ru/pub/Users/george/DBAnalys2_Copenhagen.pptx

2. Contents DBAnalys To The Rescue! Theory Optimization Synergy Practice History DBAnalys To The Rescue2

3. PROGRESS Database Analysis Database: %s Options: ixanalys chanalys Date: %s FREE CHAIN ANALYSIS =================== %l block(s) found in the free chain. RM CHAIN ANALYSIS ================= %l block(s) found in the RM chain. BLOCK ANALYSIS ============== %l RM block(s) found in the database. %l.%l% of the RM block space is used. %l index block(s) found in the database. %l.%l% of the index block space is used. INDEX BLOCK SUMMARY % % of all Index Blocks Bytes Utiliz. Indexes %s %d %l %l %l.%l %l.%l V5 1988 No tabanalys option Manning Road in Billerica, Massachusetts DBAnalys To The Rescue3

4. PROGRESS Database Analysis Database: Options: tabanalys Date: Mon Nov 6 18:26:53 1995 BLOCK ANALYSIS (3906) ============== 1 master block(s) found in the database. RECORD BLOCK SUMMARY (6129) ---Record Size--- ---Fragments--- Scatter Table Records Bytes Min Max Mean Count Index Index ----------------------------------------------------------- Totals: 15021963 642199k 6 1675 43 15275513 1.0 5.3 802392 RM block(s) found in the database. 86.10% of the RM block space is used. 270404 index block(s) found in the database. 400571 free block(s) found in the database. 5 index table block(s) found in the database. 1 sequence block(s) found in the database. 1473374 total blocks found in the database. tabanalys option V7.3A Nov, 9 1994 14 Oak Park in Bedford, Massachusetts DBAnalys To The Rescue4

5. msg 3867: 1) PROGRESS Database Analysis msg 3868: Database: %s msg 8361: Blocksize: %i msg 14178: RecSpaceSearchDepth: %d msg 6172: Options: %s %s %s msg 3870: Date: %s%r msg 14179: 2) CHAIN ANALYSIS FOR AREA "%s" : %l msg 14163: 2.1) FREE CLUSTER CHAIN ANALYSIS msg 14157: LIST OF FREE CLUSTER CHAIN BLOCKS msg 14161: %J cluster(s) found in the free cluster chain. msg 3873: 2.2) FREE CHAIN ANALYSIS msg 3880: LIST OF FREE CHAIN BLOCKS msg 14155: %J block(s) found in the free chain of %s object %i msg 16573: %rNumber of Object Object Partition/Tenant/Group msg 16574: Blocks Type msg 3892: 2.3) RM CHAIN ANALYSIS msg 3897: LIST OF RM CHAIN BLOCKS msg 14172: %J block(s) found in the RM chain of %s object %i msg 7269: 2.4) INDEX DELETE CHAIN ANALYSIS msg 7283: LIST OF INDEX DELETE CHAIN BLOCKS msg 14176: %J block(s) found in the Index Delete chain of %s object %i msg 10097: 3) AREA "%s" : %d BLOCK ANALYSIS msg 3906: 4) BLOCK ANALYSIS (moved to the end at V11) DBAnalys To The Rescue5 Segments of dbanalys output

6. msg 6129: 5) RECORD BLOCK SUMMARY msg 14212: RECORD BLOCK SUMMARY FOR AREA "%s" : %l msg 16667: RECORD BLOCK SUMMARY FOR SHARED TABLES msg 16668: RECORD BLOCK SUMMARY FOR %s %s: %l msg 16656: Summary for AREA "%s": %l msg 3921: %J RM block(s) found in the database. msg 3925: 6) INDEX BLOCK SUMMARY msg 14225: INDEX BLOCK SUMMARY FOR AREA "%s" : %l msg 16657: INDEX BLOCK SUMMARY FOR SHARED OBJECTS: msg 16658: INDEX BLOCK SUMMARY FOR %s: %s msg 3923: %J index block(s) found in the database. msg 2314: 7) DATABASE SUMMARY msg 16666: SUMMARY FOR AREA "%s": %l msg 16659: SUMMARY FOR SHARED OBJECTS: msg 16579: 8) DATABASE BLOCK ANALYSIS: msg 3937: database analysis complete %s%r msgs 0000-6130 before 8.2A msgs 7270-9999 8.3A msgs 9894-10130 9.1D msgs 14155-14225 10.2B msgs 16641-17035 11.1 DBAnalys To The Rescue6 Segments of dbanalys output

7. History of changes proutil db-name -C dbanalys [ [area] area-name ] V9.0A Nov 21, 1998 V9.1A Jan 21, 2000 Index Delete Chains / LOCKCHN (bk_frchn 2) V10.0A Dec 13, 2003 Free Cluster Chains / CLIST_FREECHN (bk_frchn 3) proutil db-name -C dbanalys –Bp 64V9.0A Nov 21, 1998 DBAnalys To The Rescue7

8. History of changes Algorithm for calculation of Scatter Factor for SAT II tables has changed V10.1A02 Oct 27, 2006 Scatter for Type II is reported as a number of blocks occupied divided by the minimum number of blocks that table could occupy. Scatter for Type I is reported as log10 of sum of distances between records in a table, where distance is calculated as the difference between the RECID values for the two records, divided by minimum distance possible. In V11 that formula lost the meaning again when the database is multi-tenant or has partitioning. DBAnalys To The Rescue8

9. History of changes V10.1B02 Aug 3, 2007 Full report on chains when running chanalys online. The following lists will be reported: LIST OF FREE CLUSTER CHAIN BLOCKS LIST OF FREE CHAIN BLOCKS LIST OF RM CHAIN BLOCKS LIST OF INDEX DELETE CHAIN BLOCKS V10.1B02 Aug 3, 2007 -recspacesearchdepth The number of RM blocks to move from the front to the back of the RM chain before allocating a new cluster for data. Default is 5 * (cluster size / 10). Max value is 100. Old algorithm: No more than 3 blocks at a time will be examined and moved to the tail end. DBAnalys To The Rescue9

10. History of changes V10.1B03 Nov 2, 2007 proutil -C chanalys [ -scan ] -scan: fewer locks, less consistent It instructs chanalys to use minimal locking. This may cause the chains report to be incomplete, but it will avoid turning on table locks. Offline chanalys does not use a locking mechanism, the -scan parameter will be ignored. [Warning] Unable to acquire table lock on object 2. Chain analysis continuing with -scan option. (14167) LIST OF FREE CHAIN BLOCKS next dbkey free 51 52 CTRL-C pressed - exit free chain analysis DBAnalys To The Rescue10

11. Dance of the broken lines (the i xanalys width is 86 chars) INDEX BLOCK SUMMARY FOR SHARED OBJECTS: -------------------------------------------- Table Index Fields Levels Blocks Size % Util Factor PUB.theTable IntField IntField901234567890123456789012 10 1 IntField-1:3 2 215 1.6M 98.2 1.0 IntField-10:2 2 215 1.6M 98.2 1.0 History of changes V11 10.2B06 March, 2012 Database Analysis (dbanalys) output has changed. In the past customer's complained that the DBANALYS report widths were not wide enough resulting in a CR after the table name when the table name exceeded the Tablename column width. Now the tabanalys width is 110 chars. DBAnalys To The Rescue11

12. History of changes V11.4 Jul 25, 2014 proutil db-name -C dbanalys [ [area] area-name ] [ tenant tenant-name| group group-name| shared ] [-csoutput [ -fieldsep sep-value][ -csfilePrefix pref-name ] [ -verbose ]] proutil sports –C dbanalys -csoutput -verbose -csoutput -fieldsep sep-value (default is space ) -csfilePrefix pref-name (default db-name) -verbose DBAnalys To The Rescue12

13. Output files DBAnalys To The Rescue13 sports.block.txt sports.ch.txt sports.ix.txt sports.tbl.txt "Time Stamp" "DBUID" "DB name" "Area" "Area #" "Partition Type" "Tenancy" "Tenancy #" "Partition Policy" "Partition" "Partition #" "Owner" "Table" "Table #" "Records" "Size" "Min" "Max" "Mean“ 2015-04-06T14:59:00.000-4:00 "njVjvK5a8qfvEyE+7KM4WA" "sports" "Customer/Order Area" 8 "-" "-" 0 "-" "-" 0 "PUB" "Customer" 2 83 12377 118 222 149 83 1.0 1.0 2015-04-06T14:59:00.000-4:00 "njVjvK5a8qfvEyE+7KM4WA" "sports" "Customer/Order Area" 8 "-" "-" 0 "-" "-" 0 "PUB" "Order" 4 207 11730 53 61 56 207 1.0 1.0 -csfilePrefix -fieldsep -verbose sports.lob.txt V11.6

14. Loading sports.tbl.txt in Excel DBAnalys To The Rescue14

15. Loading sports.ix.txt in Excel DBAnalys To The Rescue15

16. History of changes 11.6 Oct 26, 2015 LOB support in PROUTIL DBANALYS The PROUTIL database analysis utilities are enhanced to include analysis of LOB data. When LOB data is found in an area, a “LOB Summary” follows the “Record Block Summary” in the output, and LOB data is included in the combined summary. LOB data is also written to a new *lob.txt file when the -cscoutput parameter is used on the analysis command line. DBAnalys To The Rescue16

17. Contents DBAnalys To The Rescue! Theory Optimization Synergy Practice History DBAnalys To The Rescue17

18. Space: The Final Frontier… 20 Years After Disk Space Allocation in the Progress RDBMS G. Bjorklund, posted on PEG: Aug 15, 1995 Space: The Final Frontier, Engine Crew Monograph No. 2 Gus Bjorklund, Aug 10, 1997 http://www.fast4gl.com/downloads/monographs/space/space.html Space: the final frontier, OpenEdge RDBMS Internals Americas PUG Challenge 2012, Westford, MA Gus Bjorklund, May 9, 2012 https://community.progress.com/community_groups/openedge_rdbms/w/openedgerdbms/791.space-the- final-frontier OpenEdge 10 RDBMS Advanced Storage Architecture Richard Banville, May, 2004 https://community.progress.com/community_groups/user_conferences/w/exchange/2138.exchange-2004- breakout-sessions DBAnalys To The Rescue18

19. Space: The Final Frontier… 20 Years After 1. Master Block 2. Index Block 3. Data Block 4. Free Block 5. Index Dbkey List Block 6. Sequence Block 7. Empty Block 8. Parameter Block (Not Used) 9. Area Block 10. Object Directory Block (Not Used) 11. Extent List Block (Not Used) 12. Object Block 13. Control Block 14. Area Object List Block 15. Area Cluster Allocation Block 16. Object Cluster List Block 17. Object Block Allocation Block 18. RM bit Map List Block 19. Extent Header Block (bk_type 254) 20. Reserved Block Block Types (dbrpr) There are 19 known block types. 5 block types do not (should not) exist in database. 4 block types do exist in database but they are not used. 16 block types are mentioned by dbanalys before V11 and 14 block types by V11 dbanalys. DBAnalys To The Rescue19

20. Block types in dbanalys AREA "misc128_dat": 16 BLOCK ANALYSIS ------------------------------------------------- 4629499 block(s) found in the area. Current high water mark: 4629439 56 free block(s) found in the area 4627892 record block(s) found in the area 0 index block(s) found in the area 60 empty block(s) found in the area 372 object block(s) found in the area 372 cluster list block(s) found in the area 2 object list block(s) found in the area 372 cluster map block(s) found in the area 0 row allocation block(s) found in the storage area. DATABASE BLOCK ANALYSIS: ----------------------- 66432530 RM block(s) found in the database. 17906763 index block(s) found in the database. 1 master block(s) found in the database. 38 area block(s) found in the database. 1 control block(s) found in the database. 2 object block(s) found in the database. 2905 cluster list block(s) found in the database. 2905 cluster allocation block(s) found in the database. 2905 object block(s) found in the database. 42 object list block(s) found in the database. 2905 object allocation block(s) found in the database. 91506 free block(s) found in the database. 0 index table block(s) found in the database. 1 sequence block(s) found in the database. 692370 empty block(s) found in the database. DBAnalys To The Rescue20

21. Types of Storage Areas 1. Storage Area Type I 2. Storage Area Type II 3. Storage Area Type III I + II Hybrid (V10.0B) Blocks could be in the areas of the following types: Used for temp-tables ( DBI file ) DBAnalys To The Rescue21

22. Data Clusters Storage Area Type II Three possible cluster sizes: 8, 64, or 512 adjacent blocks Clusters are allocated from the area and given to a particular object. They are maintained by a doubly linked list. First and last blocks in the cluster have the extended headers (80 bytes instead of 64). First block stores Owner Info (transactionId, serialNumber) Last block stores Cluster Link (firstFreeCluster, lastFreeCluster) Database extend: MAX(64, cluster size) – that means that database will be extended for 1 cluster if the cluster size is 64 or 512 and 8 clusters if cluster size is 8. DBAnalys To The Rescue22

23. Chain Types 1. Cluster chains 2. Block chains 3. F-chains DBAnalys To The Rescue23 ;-) Cluster Chains 1. Area Control Clusters 2. Data Object Clusters 3. Free Cluster Chains (CLIST_FREECHN) Block Chains 1. Free Chains (FREECHN, bk_frchn 0) 2. RM Chains (RMCHN, bk_frchn 1) 3. Index Delete Chains (LOCKCHN, bk_frchn 2) F-Chains  unofficial name ;-)

24. Free Cluster Chains DBAnalys To The Rescue24 proutil sports –C dbrpr 15. Display Cluster Chain (Type II Area) Cluster Link of object 1/4 Total Clusters: 2 First cluster: 768 Last cluster: 1536 768 1536 1536 0 2 cluster(s) found Cluster Link of object 2/21 Total Clusters: 1 First cluster: 1024 Last cluster: 1024 1024 0 1 cluster(s) found Clusters of the“Order” table (object 1/4) and its “Cust-Order” index (object 2/21). Fast Object Deletion Area Free Cluster Link First cluster: 1024 Last cluster: 1536 1024 768 768 1536 1536 0 3 cluster(s) found Free cluster chain after the “Order” table and its “Cust-Order” index were deleted.

25. Free Cluster Chains CHAIN ANALYSIS FOR AREA "Customer/Order Area" : 8 ---------------------------------------------------------- FREE CLUSTER CHAIN ANALYSIS --------------------------- LIST OF FREE CLUSTER CHAIN BLOCKS next dbkey free 1024 768 768 1536 1536 0 3 cluster(s) found in the free cluster chain. DBAnalys To The Rescue25

26. Free Cluster Chains DBAnalys To The Rescue26

27. Block Chains proutil db –C chanalys [–scan] DBAnalys To The Rescue27 LIST OF FREE CHAIN BLOCKS next dbkey free 7026440 7026439 2046268 block(s) found in the free chain of Index object 22 LIST OF INDEX DELETE CHAIN BLOCKS dbkey Block 722 2499 3 block(s) found in the Index Delete chain of Index object 33 LIST OF RM CHAIN BLOCKS next dbkey Block 804512704 2372 49 0 8534342 block(s) found in the RM chain of Table object 4

28. Index Delete chain DBAnalys To The Rescue28 20731 block(s) found in the Index Delete chain of Index object 622 0 block(s) found in the Index Delete chain of Index object 622 proutil db-name -C idxcompact item-tax.item-group Index Object Block Free Block Free Block Free Block Free Block Ixdel Block Ixdel Block Ixdel Block Ixdel Block (Unique Indexes only)

29. Contents DBAnalys To The Rescue! Theory Optimization Synergy Practice History DBAnalys To The Rescue29

30. Dbanalys Processing Phases [2015/10/19@12:49:02.602-0400] CHAIN ANALYSIS FOR AREA "Order Area" : 9 [2015/10/19@13:18:16.944-0400] 1402490 block(s) found in the RM chain of Table object 362 [2015/10/19@13:18:16.960-0400] AREA "Order Area" : 9 BLOCK ANALYSIS [2015/10/19@13:26:11.385-0400] 4221951 block(s) found in the area. Scan 4 types of the chains: FREE CLUSTER CHAIN ANALYSIS FREE CHAIN ANALYSIS RM CHAIN ANALYSIS INDEX DELETE CHAIN ANALYSIS ) Chanalys on/offline Sequential scan of all blocks in the area Scan the Fifth type of the chains: “F-chains” Dbanalys offline

31. Timing of dbanalys run proutil dbname -C dbanalys | \ while IFS="" read Line do echo `date '+%H:%M:%S'` "$Line" done > dbname.dbanalys.log 10:19:52 INDEX DELETE CHAIN ANALYSIS 10:19:52 ----------------- 10:19:52 AREA "Order-line Area" : 7 BLOCK ANALYSIS 10:19:52 ------------------------------------------------- 10:19:52 11:55:37 13212671 block(s) found in the area. 11:55:37 11:55:37 Current high water mark: 13212671 11:55:37 DBAnalys To The Rescue31

32. Timing of dbanalys run PROMSGS=$DLC/promsgs.dba export PROMSGS DB=dbname Offset=`ls -l $DB.lg | awk '{print $5}’` (proutil $DB -C dbanalys && tail +${Offset}c $DB.lg) >$DB.dbanalys.log Use ModPromsgs.p to create the promsgs.dba file: ftp://ftp.progress-tech.ru/pub/Users/george/Programs/ModPromsgs.p ftp://ftp.progress-tech.ru/pub/Users/george/Programs/ModPromsgs.p DBAnalys To The Rescue32 [2015/10/06@17:59:12.425-0400] P-10944600 T-1 I : (-----) AREA "Order Area" : 9 BLOCK ANALYSIS [2015/10/06@18:01:20.312-0400] P-10944600 T-1 I : (-----) 2997247 block(s) found in the area. 127.887 sec = 00:02:07.887

33. Free Cluster Chain Scan Time [2015/09/29@13:13:48.269-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "Misc1 Area" : 21 [2015/09/29@13:13:48.269-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS [2015/09/29@13:17:24.353-0400] P-37552166 T-1 I Usr 19: (-----) 56325 cluster(s) found in the free cluster chain. [2015/09/29@14:55:51.512-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "ic-ledger index Area" : 26 [2015/09/29@14:55:51.512-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS [2015/09/29@14:56:26.651-0400] P-37552166 T-1 I Usr 19: (-----) 5726 cluster(s) found in the free cluster chain. [2015/09/29@17:06:05.698-0400] P-37552166 T-1 I Usr 19: (-----) CHAIN ANALYSIS FOR AREA "item-ledger Index Area" : 38 [2015/09/29@17:06:05.698-0400] P-37552166 T-1 I Usr 19: (-----) FREE CLUSTER CHAIN ANALYSIS [2015/09/29@17:19:24.490-0400] P-37552166 T-1 I Usr 19: (-----) 257338 cluster(s) found in the free cluster chain. DBAnalys To The Rescue33

34. Free Chain Scan Time DBAnalys To The Rescue34

35. Free Chain Scan Time DBAnalys To The Rescue35

36. RM Chains Scan Time DBAnalys To The Rescue36 [2015/10/19@10:25:01.825-0400] P-14155958 T-1 I Usr 19: (-----) RM CHAIN ANALYSIS [2015/10/19@11:43:01.389-0400] P-14155958 T-1 I Usr 19: (-----) 3755255 block(s) found in the RM chain of Table object 370 [2015/10/19@11:56:02.210-0400] P-27394110 T-1 I Usr 19: (-----) RM CHAIN ANALYSIS [2015/10/19@12:39:30.437-0400] P-27394110 T-1 I Usr 19: (-----) 2136418 block(s) found in the RM chain of Table object 63 [2015/10/19@12:49:03.141-0400] P-11010220 T-1 I Usr 19: (-----) RM CHAIN ANALYSIS [2015/10/19@13:18:16.944-0400] P-11010220 T-1 I Usr 19: (-----) 1402490 block(s) found in the RM chain of Table object 362 [2015/10/19@13:26:12.032-0400] P-7995430 T-1 I Usr 19: (-----) RM CHAIN ANALYSIS [2015/10/19@13:35:16.730-0400] P-7995430 T-1 I Usr 19: (-----) 185946 block(s) found in the RM chain of Table object 520 [2015/10/19@13:38:06.351-0400] P-37028030 T-1 I Usr 19: (-----) RM CHAIN ANALYSIS [2015/10/19@13:44:57.849-0400] P-37028030 T-1 I Usr 19: (-----) 177890 block(s) found in the RM chain of Table object 267 [2015/10/19@13:45:48.519-0400] P-31916152 T-1 I Usr 19: (-----) RM CHAIN ANALYSIS [2015/10/19@13:53:49.885-0400] P-31916152 T-1 I Usr 19: (-----) 150138 block(s) found in the RM chain of Table object 287

37. AREA BLOCK ANALYSIS Scan Time DBAnalys To The Rescue37 “F-chains” = the fragmented records proutil db-name –C ixanalys proutil db-name –C tabanalys

38. Table Fragmentation Affects the Reading

40. DBAnalys To The Rescue40 Table Fragmentation Affects the Reading

41. Block’s DBKEYTypeChainBackup Ctr Next DBKEY in ChainBlock Update Counter Free Space Free Dirs. Rec 0 OffsetRec 1 Offset Rec 2 OffsetRec n Offset Num Dirs. Free Space Used Data Space Record 0 Record 1 Block’s DBKEYTypeChainBackup Ctr Next DBKEY in ChainBlock Update Counter Free Space Free Dirs. Rec 0 OffsetRec 1 Offset Rec 2 OffsetRec n Offset Num Dirs. Free Space Used Data Space Record 0 Record 1 Language Record Buffer (used by 4GL clients but not by dbanalys) Rowid 9794 Rowid 2828 Block’s DBKEYTypeChainBackup Ctr Next DBKEY in ChainBlock Update Counter Free Space Free Dirs. Rec 0 OffsetRec 1 Offset Rec 2 OffsetRec n Offset Num Dirs. Free Space Used Data Space Record 0 Record 1 Rowid 10240 Connect All Fragments DB-08: A Day in the Life of a Type II Record Richard Banville Exchange 2005 DBAnalys To The Rescue41

42. Contents DBAnalys To The Rescue! Theory Optimization Synergy Practice History DBAnalys To The Rescue42

43. Record Packing Factor The minimum amount of free space needed in a block for it to remain on the RM Chain. Default: 300 bytes (150 bytes for 1K block size) Range: from 1 (from Create Limit) to (blocksize – 128) The minimum amount of free space required in a block to allow creation of a new record starting in the block. Default: 150 bytes (75 bytes for 1K block size) Range: from 32 to (blocksize – 128) The number of RM blocks to move from the front to the back of the RM chain before allocating a new cluster for data. Default: 5 (clustersize / 10) Range: from 0 to 100 Create Limit -recspacesearchdepth RPB One value per area is set in.st file. It is powers of 2 from 1 to 256. DBAnalys To The Rescue43 Toss Limit

44. Create/Toss limits proutil db -C dispTos[sCreateLimits] area* proutil db -C setAreaCr[eateLimit] area* limit proutil db -C setBlobCr[eateLimit] lob* limit (deprecated in V11) proutil db -C setLobCre[ateLimit] lob* limit proutil db -C setTableCr[eateLimit] table limit proutil db -C setAreaTo[ssLimit] area* limit proutil db -C setBlobTo[ssLimit] lob* limit (deprecated in V11) proutil db -C setLobTos[sLimit] lob* limit proutil db -C setTableTo[ssLimit] table limit [ tenant tenant-name| group group-name| shared ] * - For an area in V11 you can specify either the name or the number. And in V10 only the area number could be specified. And the same for LOBs. For a table only the name could be specified in any version. DBAnalys To The Rescue44 Proutil updates: _StorageObject._Create-Limit _StorageObject._Toss-Limit

45. Toss Limit proutil db -C chanalys “Order Area” -scan RM CHAIN ANALYSIS ----------------- LIST OF RM CHAIN BLOCKS free # free dbkey space slots hold 58719520 1337 5 0 58719536 8124 16 0 58719552 8124 16 0... 58718960 682 5 0 58719456 669 5 0 1313186 block(s) found in the RM chain of Table object 362 DBAnalys To The Rescue45

46. Number of blocks on the RM chain with free space above “X” Toss Limit DBAnalys To The Rescue46

47. Number of blocks on the RM chain with free space above “X” Toss Limit What value to choose? DBAnalys To The Rescue47

48. Toss Limit Rule of Thumb Set toss limit = average record length + 10-20% DBAnalys To The Rescue48

49. Distribution of Free Space in RM Chain Inside the Area DBAnalys To The Rescue49

50. DBAnalys To The Rescue50 Distribution of Free Space in RM Chain Inside the Area

51. DBAnalys To The Rescue51 Distribution of RM Chain Blocks Inside the Area

52. Chains in dbrpr DBAnalys To The Rescue52 DATABASE SCAN MENU ------------------ 1. Report Bad Blocks 3. Fix Bad Blocks 4. Report Bad Records 5. Delete Bad Records 6. Dump Records to RM File ON 7. Rebuild Free Chain ON 8. Rebuild RM Chain ON 9. Rebuild Index Delete Chain 10. Change Current Working Area ON 11. Fix Cluster Chains in Type II Area A. Apply scan to all areas G. Go Q. Return to Main Menu (Current Working Area: Customer/Order Area) DATABASE REPAIR MENU -------------------- 1. Database Scan Menu 2. Test One or More Indexes 3. Remove Bad Record Fragment 4. Dump Block 5. Load Block 6. Copy Bytes Between Files 7. Load RM Dump File 8. Reformat Block to a Free Block 9. Change Current Working Area 10. Display the Free Chain 11. Display the RM Chain 12. Display the Index Delete Chain 13. Display Block Contents 14. Display Record Contents 15. Display Cluster Chain (Type II Area) 16. Scan/Fix block checksum (Type II Area) A. Try to read additional Index/Table info P. Print Info Menu Q. Quit

53. Long RM chain DBAnalys To The Rescue53 proutil db-name -C dbrpr dbanalys before dbrpr 1313186 block(s) found in the RM chain of Table object 362 Scan Backward (Yes/No)? Scan of Order Area extent 1 10% complete Scan of Order Area extent 1 20% complete Scan of Order Area extent 1 30% complete Scan of Order Area extent 1 40% complete Scan of Order Area extent 1 50% complete Scan of Order Area extent 1 60% complete Scan of Order Area extent 1 70% complete Scan of Order Area extent 1 80% complete Scan of Order Area extent 1 90% complete Scan of Order Area extent 1 100% complete dbanalys after dbrpr 120813 block(s) found in the RM chain of Table object 362

54. Create Limit: For now we see through a glass, darkly promon db-name Activity: Space Allocation USA KZ RU1 RU2 LOAD Per Sec Per Sec Per Sec Per Sec Per Sec Database extends 0.29 4.94 7.02 4.99 120.13 Take free block 0.30 1.66 1.99 1.79 33.11 Return free block 0.01 0.04 0.18 0.06 0.00 Alloc rm space 53.69 196.27 130.62 387.88 2083.33 Alloc from rm 53.69 196.26 130.62 387.88 2070.93 Alloc from free 0.00 0.00 0.00 0.00 12.40 Bytes allocated 8538.09 17411.29 22496.94 38779.66 372915.41 rm blocks examined 55.69 198.46 131.42 389.71 2070.93 Remove from rm 2.02 4.23 12.44 12.88 99.21 Add to rm, front 0.0016 0.0129 0.0105 0.0065 12.40 Add to rm, back 0.47 1.11 7.14 9.66 0.00 Move rm front to back 1.18 1.55 0.05 1.42 0.00 Remove locked rm entry 14.54 18.50 20.60 180.48 0.00 Add to rm, front (blocks) 0.82 6.59 5.37 3.34 99.21 DBAnalys To The Rescue54

55. Create Limit a.k.a. Expansion Space Constant proutil db -C setAreaCreateLimit area limit proutil db -C setTableCreateLimit table limit DBAnalys To The Rescue55

56. DBAnalys To The Rescue56 Create Limit. The “Light Breathing” Space RECORD BLOCK SUMMARY FOR AREA "item-summary Area" : 35 ------------------------------------------------------- -Record Size (B)- ---Fragments--- Scatter Table Records Size Min Max Mean Count Factor Factor PUB.item-summary 34301938 15.2G 419 1066 474 35430542 1.0 1.0 1128604 records (3%) are fragmented. proutil db -C chanalys "item-summary Area" -scan CHAIN ANALYSIS FOR AREA "item-summary Area" : 35 Date: Mon Oct 19 13:45:48 2015 Date: Tue Oct 20 12:35:22 2015 150138 block(s) found in the RM chain 150460 block(s) found in the RM chain 2288127 block(s) found in the area. 15378 blocks were changed (10% of all blocks) 15258 blocks were updated 234 blocks were removed from the chain 0 blocks were added to the chain’s head 556 blocks were added or moved to the bottom 322 blocks = chain's increase

57. Create Limit. The “Light Breathing” Space proutil db -C chanalys "item-summary Area" –scan RPB 16 LIST OF RM CHAIN BLOCKS free free # free free free # free dbkey space1 space2 slots 36597840 8036 8035 -1 15 35558848 7678 7669 -9 14 36597872 8037 8037 0 15 35558864 6879 6882 +3 12 36597968 7705 7675 -30 12 36235728 7674 7674 0 15 36597984 8124 8124 0 16 36235744 7975 7976 +1 15 36598016 8000 8000 0 14 35558880 7436 7453 +17 13 36598064 8054 8054 0 15 36235760 7223 7223 0 12 36598160 7934 7919 -15 14 35558896 6539 6540 +1 10 36598192 8124 8124 0 16 36235776 7213 7206 -7 13 36598208 7990 7960 0 15 35558912 6597 6664 +67 10 36598336 7935 7874 -61 14 36235792 6506 6498 -8 11 36598432 8124 8124 0 16 35558928 7535 7558 +23 13 36598448 6711 6711 0 13 36235808 7247 7244 -3 14 DBAnalys To The Rescue57

58. Create Limit. The “Heavy Breathing” Space proutil db -C tabanalys "Order Area” RECORD BLOCK SUMMARY FOR AREA "Order Area" : 9 ------------------------------------------------------- -Record Size (B)- ---Fragments--- Scatter Table Records Size Min Max Mean Count Factor Factor PUB.order 40840212 29.1G 450 13519 764 49388337 1.2 1.0 8548125 records (21%) are fragmented. 1396614 block(s) found in the RM chain of Table object 362 4174847 block(s) found in the area. 33% of data blocks are on RM chain. DBAnalys To The Rescue58 Chain reduced by 786 blocks (from 1,403,276 to 1,402,490). HWM increased by 4096 blocks or by 8 clusters (from 4,221,951 to 4,226,047). 1329 blocks (0.1%) on RM chains were updated. 9 blocks were moved to the chain’s back. 22 records were deleted from blocks on RM chain.

59. proutil db -C chanalys "Order Area" –scan RPB 16 DBAnalys To The Rescue59 Create Limit. The “Heavy Breathing” Space

60. Create Limit. Table’s Updates vs. Creates Updates/Creates 0 : 1 ? 1 : 1 ? many to any ? DBAnalys To The Rescue60

61. Create Limit. Updates Immediately After Creates DO TRANSACTION: CREATE Customer. ASSIGN Customer.Name = “George”. END. _TableStat-create: 1 _TableStat-update: 0 ??? DO TRANSACTION: CREATE Customer. TRIGGER PROCEDURE FOR Create OF Customer. /* Automatically Increment Customer Number using Next-Cust-Num Sequence */ ASSIGN Customer.Cust-Num = NEXT-VALUE(Next-Cust-Num). ASSIGN Customer.Name = “George”. END. _TableStat-create: 1 _TableStat-update: 1 DBAnalys To The Rescue61

62. Min Record Length DBAnalys To The Rescue62

63. Mean Record Length DBAnalys To The Rescue63

64. Create/Toss Limits "Philosophy" 1.No fragmentation, no long chains –> don’t change the create and toss limits. Your efforts will be worthless. 2.First decide on the value of the create limit. Treat it as a "breathing" space for all records stored in the same block rather then an expansion of an individual record. Identify a type of record updates: frequent, only-after-create, almost- never. 3.The toss limit must be larger than the create limit. Use it to find a balance between the efficient space usage inside a database and the keeping RM chains relatively short. If it is impossible to reach the balance than just leave it and relax.

65. Defragmentati on DBAnalys To The Rescue65

66. AreaDefrag.p Version 2.0 (“Precision weapon”) 1. Works online (4GL) 2. Defrags only records that are fragmented 3. Defrags one area at a time 4. Allows to specify a range of blocks 5. Could be multi-threaded 6. Could be interrupted DBAnalys To The Rescue66

67. AreaDefrag.p Syntax: RUN ScanArea( type, area-number, position, max-blocks) Type: TRUE – with defragmentation FALSE – report only Position: Either from block-number or to block-number Unknown value “?” Means HWM and then max-blocks should be negative DBAnalys To The Rescue67

68. AreaDefrag.p Examples: RUN ScanArea( FALSE, 7, 1, 100000) scan area 7 without defragmentation from the first block forward 100 thousand blocks RUN ScanArea( TRUE, 7, ?, -100000) scan area 7 with defragmentation from HWM backwards 100 thousand blocks DBAnalys To The Rescue68

69. Testing results DBAnalys To The Rescue69 for each no-lock use-index : end. Before defrag.p : 2 hr 19 min or 139 min After defrag.p : 22 min or more then 6 times faster proutil -C dump index 0 Before defrag.p : 102 min After defrag.p : 10 min or more then 10 times faster

70. Contents DBAnalys To The Rescue! Theory Optimization Synergy Practice History DBAnalys To The Rescue70

71. _TableStat _IndexStat _TableStat _IndexStat Dbanalys DBAnalys To The Rescue71

72. synergy DBAnalys To The Rescue72

73. synergy DBAnalys To The Rescue73

74. synergy DBAnalys To The Rescue74 Dbanalys

75. synergy DBAnalys To The Rescue75

76. synergy

77. DBAnalys To The Rescue77

78. synergy DBAnalys To The Rescue78

79. synergy DBAnalys To The Rescue79

80. George potemkin@progress-tech.rupotemkin@progress-tech.ru Dmitri d_levin@hotmail.comd_levin@hotmail.com DBAnalys To The Rescue80

81. Thank you for your time!