1. Ingres Configuration and Performance Or How to get Ingres to perform as well as the English Team by Rob Acraman Dept Natural Resources 30 th May 2007

2. SQL performance Database Design Database Design –Over-Normalisation –Table Structure (B-Tree vs ISAM vs Hash) –Secondary Indexes Application Behaviour Application Behaviour –Query Structure eg. avoid Sub-SELECTs eg. avoid Sub-SELECTs –Check QEPs –Locking Contention

3. Ingres Configuration $II_SYSTEM/ingres/files/config.dat $II_SYSTEM/ingres/files/config.dat Maintain by text editor (eg. vi) Maintain by text editor (eg. vi) Maintain by ‘cbf’ Maintain by ‘cbf’ Changes take effect at next startup Changes take effect at next startup

4. Ingres Health Check Lockstat Lockstat –Locking Statistics Logstat Logstat –Logging Statistics DM420 DM420 –DMF Cache Statistics Ima Ima –Query interface –SELECT what you want All produce statistics SINCE THE LAST STARTUP.

5. /home/somedba> iishowres -d Current lock/log Allocations (not DMF) !Retrieving configuration data for the logging system. ! !Display the configuration data entered. ! !The log file size: 2096128KB !The number of log file partitions: 2 !The number of log buffers in the memory : 20 !The maximum number of transactions in the logging system : 1700 !The maximum number of databases in the logging system: 160 !The max consistency point interval for invoking archiver : 7 !The block size of the log file : 4 Kbytes !The log-full-limit in percentage : 90 !The percentage of log file for each consistency point : 5.00 !The force-abort-limit in percentage : 72 !The size of the lock hash table : 18199 !The size of the resource hash table : 18199 !The maximum number of locks in the locking system : 136548 !The maximum number of resources in the locking system: 91032 !The maximum number of lock lists in the locking system : 1700 !The maximum number of locks allowed per transaction : 500 Memory allocation for 1700 lock lists = 564400 bytes Memory allocation for 136548 locks = 11652096 bytes Memory allocation for 91032 resource blocks = 46244256 bytes Memory allocation for hash tables (18199 lock hash table,18199 resource hash table) = 6406048 bytes. Memory allocation for 20 (4096-sized) log buffers = 10196 bytes Memory allocation for 1700 transactions = 1013200 bytes Memory allocation for 160 databases = 85760 bytes Memory allocation for internal data structures = 51564 bytes Total allocation for logging/locking resources = 66027520 bytes /home/somedba>

6. lockstat /home/somedba> lockstat -statistics ==============Tue May 1 16:41:56 2007 Total Locks 50000 Total Locks 50000 Lock hash table 67819 Lock hash table 67819 Resource hash table 67819 Resource hash table 67819 Total lock lists 5000 Total lock lists 5000 =============Tue May 1 16:41:56 2007 Create lock list 3260256 Create lock list 3260256 Request lock 333614341 Request lock 333614341 Convert lock 4621641 Convert lock 4621641 Escalate 2 Escalate 2 Convert wait 0 Convert wait 0 Deadlock Wakeups 13 Deadlock Wakeups 13 Deadlock Search 13 Deadlock Search 13 Cancel 5 Cancel 5 Allocate CB 336767602 Allocate CB 336767602 LBK Highwater 4 LBK Highwater 4 SBK Highwater 3 SBK Highwater 3 RBK Highwater 10 RBK Highwater 10 Max Local dlk srch 0 Max Local dlk srch 0 Max rsrc chain len 16 Max rsrc chain len 16 Max locks per txn 2633 Max locks per txn 2633 Callbacks Invoked 0 Callbacks Invoked 0 Locking System Summary============= Total Resources 50000 Locks per transaction 5000 Locks in use 8271 Resources in use 7611 Lock lists in use 154 Locking System Statistics=========== Release lock list 3259965 Re-request lock 18057160 Release lock 318460770 Lock wait 22348 Convert Deadlock 0 Max dlk queue len 1 Deadlock 0 Convert Search 0 Deallocate CB 668207774 LLB Highwater 200 LKB Highwater 5760 RSB Highwater 10936 Dlk locks examined 13 Max lock chain len 16 Callback Wakeups 0 Callbacks Ignored 0

7. CP Thread WriteBack Threads Logging and Cache Transaction Log File Database 1 Files Journal Files (database 1) Journal Files (database 2) DMF Cache Log Buffers Ingres Server dmfacp (Archiver Process) LogWriter Threads

8. logstat ======================Mon May 28 17:08:31 Database add 225111 Database add 225111 Transaction begins 6146608 Transaction begins 6146608 Log read i/o's 499109 Log read i/o's 499109 Log writes 2925788 Log writes 2925788 Log waits 620508 Log waits 620508 Log free waits 39 Log free waits 39 Log BCP waits 18 Log BCP waits 18 Log group commit 6486 Log group commit 6486 Check commit timer 3709 Check commit timer 3709 Timer write, time 0 Timer write, time 0 Inconsistent db 0 Inconsistent db 0 ii_log_file read 499109 ii_log_file read 499109 write complete 605350 write complete 605350 All logwriters busy 11442 All logwriters busy 11442 ----Current log file header----------------------------------------------------- 2007 Logging System Summary=========== Database removes 224813 Transaction ends 6146178 Log write i/o's 605483 Log forces 12388 Log split waits 1212 Log stall waits 18 Logfull stall waits 0 Log group count 6502 Timer write 3709 Timer write, idle 3709 Kbytes written 357648 ii_dual_log read 0 dual write complete 0 Max write queue len 20 Max write queue cnt 48 Block size: 4096 Block count: 1310720 Partitions: 1 Buffer count: 20 Increase number of Log Buffers (maybe double number with half the size) If All Logwriters Busy > 10% of all Log Waits, increase number of Log Writers (if half the size, don’t forget to also double Block count = number of blocks in Transaction Log File)

9. Buffer Utilization ----Buffer utilization profile-------------------------------------------------- <10% **************************************** <10% **************************************** 10-19% **** 10-19% **** 20-29% ***** 20-29% ***** 30-39% * 30-39% * 40-49% ** 40-49% ** 50-59% * 50-59% * 60-69% * 60-69% * 70-79% * 70-79% * 80-89% * 80-89% * >90% ************* >90% ************* ----Buffer utilization profile-------------------------------------------------- <10% **************** <10% **************** 10-19% * 10-19% * 20-29% *** 20-29% *** 30-39% * 30-39% * 40-49% * 40-49% * 50-59% * 50-59% * 60-69% * 60-69% * 70-79% * 70-79% * 80-89% * 80-89% * >90% ********************************************** >90% ********************************************** Increase number of Log Buffers (maybe double number with half the size) Good: BAD:

10. logstat ======================Mon May 28 17:08:31 Database add 225111 Database add 225111 Transaction begins 6146608 Transaction begins 6146608 Log read i/o's 499109 Log read i/o's 499109 Log writes 2925788 Log writes 2925788 Log waits 620508 Log waits 620508 Log free waits 39 Log free waits 39 Log BCP waits 18 Log BCP waits 18 Log group commit 6486 Log group commit 6486 Check commit timer 3709 Check commit timer 3709 Timer write, time 0 Timer write, time 0 Inconsistent db 0 Inconsistent db 0 ii_log_file read 499109 ii_log_file read 499109 write complete 605350 write complete 605350 All logwriters busy 11442 All logwriters busy 11442 ----Current log file header----------------------------------------------------- 2007 Logging System Summary=========== Database removes 224813 Transaction ends 6146178 Log write i/o's 605483 Log forces 12388 Log split waits 1212 Log stall waits 18 Logfull stall waits 0 Log group count 6502 Timer write 3709 Timer write, idle 3709 Kbytes written 357648 ii_dual_log read 0 dual write complete 0 Max write queue len 20 Max write queue cnt 48 Block size: 4096 Block count: 1310720 Partitions: 1 Buffer count: 20 Fill Factor% = “Kbytes written”,000 * 100 “Log write i/o’s” * “Block size” = 35764800000 605483 * 4096 = 14% Aim for 60%

11. CPs and WriteBack Both : Both : –Write from Transaction Log File –To Database Files –Freeing pages in DMF CPs triggered by percentage of Transaction Log File usage CPs triggered by percentage of Transaction Log File usage –Default 1% - Don’t exceed 5% –ii.myserver.rcp.log.cp_interval: 5 –Logstat (26201 = 5% of 524032) ----Current log file header----------------------------------------------- Block size: 4096 Block count: 524032 Partitions: 1 Buffer count: 20 Block size: 4096 Block count: 524032 Partitions: 1 Buffer count: 20 CP interval: 26201 Logfull interval: 471628 Abort interval: 377303 CP interval: 26201 Logfull interval: 471628 Abort interval: 377303 CPs also trigger Archiver CPs also trigger Archiver –Archiver process (dmfacp) moves pages from Transaction Log File to Journals –Freeing Transaction Log File pages in the process –This happens every CPs –ii.myserver.rcp.log.archiver_interval: 7 –Typically, you want it 1 or 2 (to avoid large updates)

12. Writeback Threads WriteBack Threads triggered by number of modified pages in DMF WriteBack Threads triggered by number of modified pages in DMF –ii.myserver.dbms.private.*.dmf_cache_size: 928 –ii.myserver.dbms.private.*.dmf_wb_end: 464 –ii.myserver.dbms.private.*.dmf_wb_start: 557 –ii.myserver.dbms.private.*.p16k.dmf_cache_size: 928 –ii.myserver.dbms.private.*.p16k.dmf_wb_end: 464 –ii.myserver.dbms.private.*.p16k.dmf_wb_start: 557 –ii.myserver.dbms.private.*.p4k.dmf_cache_size: 928 –ii.myserver.dbms.private.*.p4k.dmf_wb_end: 464 –ii.myserver.dbms.private.*.p4k.dmf_wb_start: 557 ……… CBF Defaults : CBF Defaults : –dmf_wb_start = dmf_modify_limit - 15% * dmf_cache_size –dmf_wb_end = 50% * dmf_cache_size

13. DM420 Displays DMF Cache Statistics $ sql iidbdb * set trace output ‘dm420.log’ \p\g * set trace point dm420\p\g \q $ ls –l $II_SYSTEM/ingres/files/dm420.log -rw-r--r-- 1 ingres dba 5335 Apr 27 16:06 dm420.log DM420 APPENDS to the file, not Overwrites ! DM420 APPENDS to the file, not Overwrites ! (so make sure you delete first). (so make sure you delete first).

14. DM420 !Begin Trace Log at Fri Apr 27 16:06:40 2007 !Fri Apr 27 16:06:50 2007 [29097,01A6B4A0] iidbdb >>>>> set trace point 'dm420' ! Buffer Status: MT ! Buffer Manager Id: 12 Connected servers 1 ! CP count: 725 CP index : 0 CP check : 0 ! Database cache size: 40 Table cache size: 40 ! Statistics----------------------------------------------- ! Lock reclaim: 0 CP flushes: 724 ! Buffer Cache Configuration ( 2K) ------------------------ ! Buffer count: 1064 Bucket count: 1063 Group count: 42 Size: 8 ! Free count: 327 Limit: 23 Modify count: 401 Limit: 546 ! Free group count: 41 Modify group count: 1 ! Fixed count: 0 Group fixed count: 0 ! WB start limit: 437 WB end limit: 364 ! WB flushes: 5519, WB threads created: 0 ! WB threads active: 0, WB thread hwm: 1 ! WB pages flushed: 73, WB groups flushed: 0 WB Flushes > CP Flushed System can’t keep pages in memory long enough. Increase dmf_wb_start (or reduce CP_Interval)

15. DM420 by Page Size ! Summary Statistics ( 2K) -------------------------------------- ! GWAIT GSYNC FREEWAIT GW PAGES GW I/O ! 5 0 0 570318 57072 ! FIX CALLS HITS CHECK REFRESH READ TOSS ! 414658001 332355910 82787709 63 50301045 74932 ! UNFIX CALLS DIRTY FORCE WRITE IOWAIT SYNC ! 217670172 1907841 25639 919761 1961 0 ! MWAIT FCWAIT RECLAIM REPLACE GREADS GWRITES ! 9 2 47251 50824104 247568079 0 FREEWAIT MUST be zero – FREEWAIT MUST be zero – otherwise SERIOUSLY increase dmf_cache_size for the pagesize otherwise SERIOUSLY increase dmf_cache_size for the pagesize HitRate = “HITS” / “FIX CALLS” Should be >= 95% HitRate = “HITS” / “FIX CALLS” Should be >= 95% Increase dmf_cache_size and dmf_group_count Increase dmf_cache_size and dmf_group_count Check EACH Page Size Check EACH Page Size “READ” should be less than “GREADs” (Group Reads = Table Scans) “READ” should be less than “GREADs” (Group Reads = Table Scans) Perform Optimize, Missing Indexes, Check SQL Perform Optimize, Missing Indexes, Check SQL

16. Modifying Configuration MEASURE weekly MEASURE weekly –Keep history of snapshots –Take snapshot before weekly bounce –If no bounce, then subtract values DON’T make changes blindly DON’T make changes blindly CHANGE and bounce CHANGE and bounce MEASURE the following week(s) MEASURE the following week(s)

17. ?