1. Ten Clicks to Breakthrough DB2 UDB Performance Results
DB2 for Linux, UNIX, and Windows
Ten Clicks to Breakthrough DB2 UDB Performance Results
Philip K. Gunning
Performance Consultant
DGI
Session: D9
September 26, 2002
Copyright 1999

2. ABSTRACT
This session will present key DB2 UDB Performance Measurements and a path to navigate through the numbers such that huge performance gains can be achieved very rapidly, often in just minutes. Discover the top warning signs and symptoms of poor database health, and learn how to drill down to find the real underlying problems 2

3. OUTLINE 1. 10 Key Performance Tips 2. Key Performance Knobs
3. Memory Utilization -- Agent parameters
4. Evaluating Sort Performance
5. Monitoring and Measuring results 2

4. DISCLAIMERS
DB2 ™ is a registered trademark of IBM Corporation. Oracle is a registered trademark of Oracle Corporation and SQL Server is a registered trademark of Microsoft.
DB2 Universal Database (DB2 UDB) is a registered trademark of International Business Machines Corp.
Other logos and product/trade names are registered trademarks or trademarks of their respective companies. 2

5. BREAKTHROUGH DEFINITION:
NOUN. “ MAJOR ADVANCE OR DISCOVERY”
NOUN.
“ ACT OF BREAKING THROUGH AN OBSTACLE” 2

6. The Ten Clicks SQL Indexes Bufferpools Prefetching Parallelism SORT
Logging
AGENTS
Memory Structures
Monitoring
Continue 2

7. SQL
Experience has shown that poor performing SQL is the cause of most performance problems. Too many organizations spend most of their time changing DB or DBM CFG parameters without first looking at the SQL!
Use explain and snapshots and event monitors for capturing and analyzing SQL. This is an iterative approach. Fix one poorly performing SQL statement and then find the next one!
Use INDEX ADVISOR! Still, many DBAs not using it. 2

8. SQL
Poor performing SQL is usually % of most performance problems
Using the rule, 20% of this SQL will be what you need to focus your efforts on
Just because it is a Third Party Vendor Package doesn’t mean YOU CAN’T TUNE IT!
Index changes
Clustering indexes*
Include columns
Use the rule when analyzing poorly performing SQL. Look for SQL executed with high frequency. This type of SQL can also be identified by identifying your most active tables and tablespaces.
Many actions can be taken to tune SQL with Third Party Vendor packages. Identify the poorly performing SQL and fix it by adding or changing indexes. 2

9. SQL db2 get snapshot for dynamic sql on hrproddb
select STMT_TEXT from table(SNAPSHOT_STATEMENT)’ATGW_D’, -2)) as D;
Returns statement information from application snapshot
db2 create event monitor dgi_han for statements write to table buffersize 4 blocked manualstart ;
db2advis -d atgw_p -i inputfile.txt
When supporting Third Party Vendor packages such as ERP or CRM, dynamic SQL statements can be captured via a dynamic SQL snapshot and much information gleaned from them such as prepare time, # of times compiled, and elapsed time, plus the SQL statement. This can be redirected to a file and then used with explain and index advisor.
Event monitoring will capture all SQL executed upon completion of the event. db2evmon, db2evma can be used to format and review the data. This can also be helpful in identifying and resolving deadlock conditions. V8 has enhanced the deadlock snapshot to include detailed lock information and the SQL statement involved.
The Index advisor can be used to tune SQL and workloads. It displays results of analysis and provides index recommendations and indicates associated improvement percentages in timerons. 2

10. Find the SQL! - The DGI Method -1

11. Find the SQL! - The DGI Method -2

12. SQL
TIP: Identify SQL executed with high frequency and tune. Review, monitor, and tune dynamic and static SQL on a regular basis
Identify SQL executed with high frequency and tune. This will typically be associated with your worst performing tablespace. By reviewing it as part of the application development process, it can be found and tuned before it gets into production!

13. INDEXES 2

14. Indexes New Type-2 Index in V8.1
Eliminates most next key share locking problems
Pseudo delete of index entries
REQUIRED for MDC
Migration Considerations
Migrate tables at-a-time
Type-2 required for New V8.1 Online Reorg and Online Load
Cannot mix and match type-1 and type-2 on same table
New Type-2 index improve database concurrency because the key is marked deleted instead of physically removed from the index page. And an index can now be created on columns with more than 255 bytes. Indexes can be converted to Type-2 by using the REORG INDEX command. The new INSPECT command can be used to find out what type of indexes exist on a table. 2

15. Indexes Clustering Maintains data rows in order of index entries
Greatly improves prefetching and sequential scans
Every table should have a clustering index
Clustering reduces or eliminates sorts
Most Leading ERP and CRM Third Party Vendor packages do not use any clustering indexes
Must reorg table using clustering index for clustering to take effect
One clustering index per table
New type of Clustering VX.
Many vendor packages do not specify clustering indexes. Significant performance can be gained by specifying clustering indexes based on the most common type of access is used for a table. 2

16. Indexes
Read-only objects can have more indexes than heavily inserted and updated tables
Review application access patterns with developers before indiscriminately adding indexes
Consider “include columns” for index-only access
Reorg indexes
Index Anding and Index Oring
DB2 can use index Oring to convert an “in list” to OR predicates and also use index Oring to eliminate duplicates. DB2 can use index ANDing when a table has multiple indexes and a query is run with multiple AND predicates. DB2 can use bitmaps to identify qualifying rows.
Include columns can be used to obtain index-only access. Include columns can only be used with a unique index. 2

17. Indexes Multidimensional Clustering (MDC)
can create indexes on multiple dimensions
no reorg required, automatically kept in clustering order
stored in blocks
MDC enables tables to be clustered on multiple keys or dimensions, simultaneously
primarily designed to benefit DW/BI type queries but can also be used for OLTP
DB2 can use index Oring to convert an “in list” to OR predicates and also use index Oring to eliminate duplicates. DB2 can use index ANDing when a table has multiple indexes and a query is run with multiple AND predicates. DB2 can use bitmaps to identify qualifying rows.
Include columns can be used to obtain index-only access. Include columns can only be used with a unique index. 2

18. Indexes Create appropriate indexes
Consider include columns for index-only access
Use clustering indexes to improve performance
Analyze using Index Advisor and look at local, order by and join predicates

19. BUFFERPOOLS 2

20. Bufferpools Asilomar Report
1998 meeting of Relational Database Industry and Academic Professionals
Predict main memories and addressing techniques will improve to hold vast amounts of data
Indicate Significant improvement needed in Optimization
Technology
Sizing
75% of RAM available if dedicated database server
Better to over-allocate than to under-allocate
monitor OS paging and reduce bufferpool size if necessary
Breakout Strategy
The Asilomar Report was published by the ACM in It was developed by industry experts from IBM and other corporations along with academia. A similar study is conducted every few years to identify database research to be conducted over the next ten years. It also identifies technology areas that should be included with database management systems.
One of the projections from the report is the availability of very large memories that will be able to accommodate most of the main memory needs of databases. The use of XML and Web services were also identified and we are seeing them now in current products.
A dedicated database server should allocate up to 75% of available memory to exploit the efficiencies of maintaining data in memory. As with any recommendation this should be monitored to ensure no adverse effects are seen such as paging. 2

21. Bufferpools Breakout Strategy Random versus sequential
Index versus data
Prioritization
Large Randomly Accessed Tables
DW versus OLTP
One large bufferpool
Reduces overhead of managing multiple pools
DW tables typically very large involving large scans 2

22. Bufferpools Allocate up to 75% of memory to bufferpools
Use block bufferpools for large sequential access type applications
Use one bufferpool for DW/BI
Use multiple bufferpools and associated breakout strategy for OLTP

23. PREFETCHING 2

24. Prefetching
Num_ioservers should be set to number of disks plus 10% as a starting point
monitor and adjust as necessary
Exentsize of 32 is generally good
If set too high can overwhelm bufferpool
Prefetchsize should be set to (extentsize x number of containers)
This will enable parallel prefetch request to be issued to each container
Efficient prefetching is critical to DW/BI type environments where large scans may be performed. By adequately identifying the prefetchsize, this can be accomplished.
Use Asynchronous pages read per request to determine the efficiency of prefetching. If prefetchsize is less than extent size the bufferpool is overheated or undersized. 2

25. Prefetching Parallel IO DB2_PARALLEL_IO=* (1,2,3)
DB2_STRIPED_CONTAINERS
RAW DEVICES ARE BECOMING THE NORM!
Prefetchsize= (extent size x # of containers)
Spread containers over multiple physical disks
Set Operating System parameters to maximize Asynchronous IO performance (AIX - minservers and maxservers)
Block-based bufferpools in V8.1 provide improved performance by prefetching sequential pages that are contiguous in block bufferpools
Use the DB2_PARALEL_IO registry setting to enable parallel IO on single containers defined on raid devices. Can enable for all tablespaces or just for selected ones.
DB2_STRIPED_CONTAINERS should be set to YES when using DMS Raw Logical Devices as this lines up the container tag with the RAID stripe and can provide a significant IO boost. 2

26. Prefetching

27. Prefetching

28. Prefetching
Set prefetchsize to a multiple of extentsize x number of containers
Don’t overestimate prefetchsize as this can overwhelm bufferpool
Monitor Asynchronous Pages Read per Minute to evaluate efficiency of prefetching

29. PARALLELISM 2

30. Parallelism OLTP versus DW OLTP set intra_parallel to NO
DW set intra_parallel to YES
How to disable for OLTP
set MAX_QUERYDEGREE=1
set DFT_DEGREE=1
How to Exploit for BI/DW
set DFT_DEGREE=1 and let DB2 decide
set MAX_QUERYDEGREE equal to number of CPUs but never more
Generally, OLTP applications do not lend themselves to query parallelism so intra_parallel should be set to NO. DW environments should always use query parallelism with intra_parallel set to yes. DW queries lend themselves to parallel task processing with data ranges over large amounts of data.
MAX_QUERYDEGREE limits the amount of parallelism at the database level.
DFT_DEGREE sets current value for the CURRENT DEGREE special register
and the DEGREE bind option. 2

31. Parallelism Utility Parallelism
Utilities use parallelism independently of DB or DBM CFG settings
New “throttle” capability in V8.1
Fast Communication Manager (FCM)
used to manage communication of messages between subagents
used with intra_parallel enabled or if using ESE Partitioning Option
Monitor using “Get snapshot for fcm on all nodes”
Utilities use available CPUs to accomplish work. This is done through the use of IO parallelism when multiple containers and disks are present and through the use of multiple processors in an SMP environment. Both can be used together to provide even faster loading.
FCM is used to manage communication between partitions and subagents in intra-partition parallelism. Monitor to ensure number of buffers is adequate. FCM errors written to db2diag.log file.
New throttle capability in V8.1 enables DB2 to restrict amount of CPU utilities use base on your definitions! 2

32. Parallelism Set intra_parallel to no in OLTP
Use utility parallelism with load, reorg,backup and restore (Ensure adequate buffer size)

33. SORT 2

34. Sort Private Sortheap allocated out of agent private memory Shared
Sortheap allocated out of Database Global Memory
SHEAPTHRES
Private Sort – Soft Limit
Shared Sort – Hard Limit
SORT is a FOUR Letter WORD! Identify SQL performing the most sorts and try and eliminate sorts through indexing or clustering. Private sorts are allocated out of agent private memory (sortheap) and Database Global Memory for parallel sorts. SHEAPTHRES controls number of concurrent sorts across an instance. Post threshold sorts occur when this limit is exceeded for private sorts. Soft limit for private sorts which means the database manager allocates smaller and smaller sortheaps while SHEAPTHRES limit exceeded. For shared sorts, applications will receive errors and additional sort will not be allocated. 2

35. Sort Piped Non-Piped Sort Overflows Entire sort overflows
Overflows to Bufferpool and System Temporary Space Work Tables
How to Avoid
Sort Heap (sortheap) (DB CFG) specifies the maximum number of private memory pages to be used for each private sort or the maximum number of shared memory pages to be used for shared sorts. The default is 256 4k pages with a range of 16 – 524,288 4k pages. It is allocated as needed to perform sorts and freed when sorting is complete. DB2 uses two types of memory for sorts. Private memory is used for non-parallel sort techniques (using EE with intra_parallel set to no). Private sorts are allocated out of agent private memory as needed by the Database Manager. Data is sorted in the sort heap. A smaller sort heap can be used if determined by the optimizer. Sort heap is used by the optimizer in determining access paths and packages should be rebound if the value of sort heap is changed. If the sort heap is not big enough, sorting can overflow to temporary tables in Tempspace1 and this should be avoided in OLTP systems if possible. In large database environments or OLAP/BI, sort overflows can probably not be avoided and then effort should be placed on tuning the I/O paths (disks, containers, bufferpools). Of course, the best sort is NO SORT but we humans like things in order so it is difficult to avoid. The most effective method for eliminating sorts is proper indexing on the sorted columns. Sorting is very costly and probably makes up more than 70% of most queries elapsed times, so there can be big payoffs by tuning sorts and sort memory areas. Be sensitive to OS paging when sizing sortheap and SHEAPTHRES and do not size them so big as to cause paging. Piped sort not freed until cursor closed. 2

36. Sort Post Threshold Sorts Piped Sorts Rejected Hash Joins Binary Sort
DB2_BIN_SORT in V6.1 enables new sort algorithm that optimizes sort
Monitor post threshold sorts and piped sorts rejected to evaluate efficiency of sort operations. Adjust SHEAPTHRES or SORTHEAP to reduce or eliminate post threshold sorts. If HASH JOINS is being used, understand implications for SORTHEAP. 2

37. SORT Harsh Reality
SQL True Cost = Individual Execution Cost multiplied by Frequency of Execution in the application workload mix.
The previous 2 slides have demonstrated how a seemingly fast (elapsed time < .5 second) SQL statement consumes over 1/5th (21%) of CPU time in the application, and 48%, almost half, of all sort time in the database.
It is extremely unlikely that a 1/2 second duration statement would be captured by a snapshot, and, if it were, based on the “low” estimated optimizer cost, such a statement might miss the DBA attention “clip level”

38. Sort db2advis -d peiprd -I wildsortsql.txt > wildsqlixadvout.txt
execution started at timestamp
found [1] SQL statements from the input file
Calculating initial cost (without recommmended indexes) [ ] timerons
Initial set of proposed indexes is ready.
Found maximum set of [1] recommended indexes
Cost of workload with all indexes included [ ] timerons
total disk space needed for initial set [ ] MB
total disk space constrained to [ ] MB
1 indexes in current solution
[ ] timerons (without indexes)
[ ] timerons (with current solution)
[%99.69] improvement
Trying variations of the solution set.--
-- execution finished at timestamp
-- LIST OF RECOMMENDED INDEXES
-- ===========================
-- index[1], MB
CREATE INDEX WIZ1 ON "PUSER "."T_FILE" ("FILE_STATUS_NUM" DESC) ;
-- ===========================--
Index Advisor tool is finished. 2

39. Sort Eliminate through Clustering Indexing Efficient SQL
Cannot exterminate because we humans need things returned in an orderly fashion, but we can eliminate the unnecessary ones! 2

40. Sort SORT IS A FOUR LETTER WORD
Find the SQL that is performing sorts and eliminate through indexing, clustering, changing the SQL
Use Snapshots, Event Monitors, Third Party Vendor tools, DB2 Explain
Index Advisor
MDC

41. LOGGING 2

42. Logging
Primary Logs formatted at first connect or activate database command
Have enough primary and plenty of secondary
Determine via benchmarking and monitoring
Maximize use of log buffer by setting logbufsz to at least 64 4k pages (32-bit AIX)
Increases throughput and makes maximum use of log buffering 2

43. Logging Mincommit Used to group commits to increase throughput
Can increase throughput at expense of response time wait up to 1 second
Create separate file system for logs and don’t put any other data on volumes with logs
Consider mirroring logs or use mirrorlogpath DB CFG parameter

44. Logging In an OLTP environment set mincommit using following formula
(# of transactions per second / 10) and use the result for setting mincommit.
Example: 95 TX per sec / 10 = mincommit 9
Set logbufsz to at least a value of 64, larger if ample memory available

45. AGENTS

46. Agents
Coordinator Agent – Each application has one which does work on its behalf and in a parallel environment distributes work to subagents
Upon disconnect or detach from an instance the coordinating agent is freed and marked as idle if max number of pool agents not reached else it is terminated and storage freed if max number of pool agents reached
DBM CFG parameter max_coordagents

47. Can be used in resource constrained systems to limit memory usage
Agents
Maximum Number of Agents (maxagents)
specifies the maximum number of database manager agents, whether coordinating agents or subagents, available at any given time to accept application requests
Can be used in resource constrained systems to limit memory usage

48. Agents Maximum Number of Active Applications (maxappls)
Specifies the maximum number of concurrent
applications that can be connected to a database
When reached, an error is returned to the application and connection is not allowed
Can be used to throttle applications in a resource constrained system

49. Agents Maximum Number of Concurrent Agents (maxcagents)
Specifies the max number of database manager coordinating agents that can be concurrently executing a database manager transaction
Does not limit the number of applications connected but limits the number of database manager agents that can be processed concurrently
Can be used to throttle applications if resource constrained

50. Initial Number of Agents in Pool (Num_initagents)
Specifies the initial number of idle agents that are created in the agent pool at DB2START
By specifying a value, agents are available in the pool for initial requests and the overhead of repeated agent creation is avoided

51. Contains subagents and idle agents
Agent Pool Size (num_poolagents)
Specifies how large the agent pool can get
Contains subagents and idle agents
Idle agents can be used as coordinating agents or subagents
If more agents created than this parameter they will be terminated when the current request is completed rather than returned to the pool

52. Agents Ensure that there are no Stolen Agents!
Ensure that no Agents are waiting for a Token (maxcagents)
Use connection pooling in OLTP and hybrid OLTP/DW environments
Connection concentration
Reduces amount of total memory used by agents and maps many connections to one logical agent

53. MEMORY

54. Memory Locklist Used to store memory for locks
Over-allocate versus under-allocate
num_locks default of 22% is a good starting point
Use lock_timeout to set timeout for lock waiters
Default -1 means applications wait for lock forever
not normally a good thing
Catalog Cache
Used to store table descriptors
Eliminates read to catalog if used
efficiently
Size and monitor to ensure consistent hit ratio > 90%

55. Memory Package Cache How it really works
Must be large enough in the first place to store all executing dynamic and static SQL before caching occurs
Used to store dynamic and static SQL
Size and monitor appropriately and make sure inserts are low
If package cache overflows, it overflows into locklist and other dbheap areas!
Size and monitor to ensure hit ratio > 90%
New V8.1 capability to flush package cache!

56. Memory
Over-allocate locklist by 50% or ensure that no more than 50% of locklist is in use
Make catalogcache_sz large enough so that a 90% hit ratio is obtained
Sufficiently size bufferpools!
Analyze memory usage over time

57. MONITORING

58. Monitoring New SQL Function in V8.1enables taking snapshots via SQL
select STMT_TEXT from table(SNAPSHOT_STATEMENT)’ATGW_D’, -1)) as T;
Can issue from command line or API
Event monitors can write to tables!

59. Monitoring
Snapshot monitoring enabled via DBM CFG monitor switches or application can set switches
Snapshot capability added to SQL as function
Snapshot output on LOCKS improved to show lock details and SQL involved!

60. Monitoring
Event Monitoring can now be created in EEE and write to a table on each node!

61. Monitoring New V8.1 Health Monitor enables alerts by exception
send to predefined list of DBAs
send alert to a pager
Exception monitoring fairly common requirement with many organizations having 100’s or thousands of databases to monitor
many companies had written in-house exception monitoring scripts but were still looking for a better solution
Health monitor is configured via the Health Center

62. HEALTH CENTER

63. HEALTH MONTOR
Server side code that monitors thresholds and generated alerts via Health Center, Beacons, , page to pager
Configured via the Health Center
Implements “Management by Exception” alerting
It addresses the need to monitor 100s or 1,000s of databases without “eyes on”

64. MEMORY VISUALIZER

65. Monitoring Have a monitoring plan and monitor on a regular basis
Take advantage of new monitoring tools in V8.1 and from Third Party Vendors
If you are not monitoring, you are driving blind

66. Summary Now you have 10 ways to improve your database performance
Remember, make one change at a time and monitor and measure the effects of all changes

67. References SC09-4848-00, What’s New, V8
SG , Redbook, DB2 UDB V7.1 Performance Tuning Guide
SC , DB2 UDB V8.1 Administration Guide: Performance
Asilomer Report, ACM Proceedings, 1998
SC , DB2 UDB V7.2 Administration Guide: Planning
AIX Performance and Tuning Guide

68. Philip K. Gunning DGI THANK YOU! pgunning@breakthroughdb2.com
10 Clicks to Breakthrough DB2 UDB Results
Session: D9
Philip K. Gunning DGI THANK YOU!
Copyright 1999