Monitoring the Essentials in DB2 10.5 Philip K. Gunning Gunning Technology Solutions, LLC Session Code: C03 May 13, 2014 3:15 PM – 4:15 PM| Platform: DB2 LUW

DISCLAIMER DB2™ is a registered trademark of IBM Corporation. DB2 BLU, BLU Acceleration are registered trademarks or copyrights of IBM Corporation Other logos and product/trade names are registered trademarks or copyrights of their respective companies NOTES:

The Warp Speed Achieved with BLU Acceleration “For data analytics the volumes involved are usually dramatically larger than cache sizes, which highly limits exploitation of CPU caches. The secret to the warp speed-up achieved in BLU Acceleration is that IBM has designed every algorithm to minimize access to RAM, and maximize processing time in L3 and L2 caches. For the processor shown that provides a whopping 15x-41x speed-up for memory access” (Sam Lightstone, IBM, www.Softwaretradecraft.com) NOTES:

Overview Overview of BLU Acceleration DB2 in the New Galaxy Concepts Terminology DB2 in the New Galaxy When RAM is too slow Learning to speak and think “Column organized” in BLU Acceleration Rules of Thumb for column organized data in terms workload, concurrency, bufferpools, sort heap threshold, sortheap NOTES:

Overview Important role of UTIL_HEAP_SZ New “Column Organized” monitoring metrics How efficient are my column-organized tables Are you getting what you think? Using MONREPORT reporting module Tuning Recommendations Best Practices NOTES:

Quick Look at Prerequisites DB2 Workgroup Advanced Enterprise Server Edition DB2 Advanced Enterprise Server Edtion OS and Hardware prerequisites NOTES:

New Registry Variable Setting for Analytical Workload Before creating a new database, set the DB2_WORKLOAD registry variable to ANAYLTICS This will result in the following: Makes column organized tables the default table type DFT_TABLE_ORG=COLUMN Page and extent size configured for analytical processing PAGESIZE=32 DFT_EXTENT_SZ=4 DFT_DEGREE=ANY Memory for sorting, caching and hashing will be set based on the amount of RAM and server size Enables automatic space reclamation Compression is always on NOTES: Set the DB2_WORKLOAD registry variable to ANALYTICS before creating any database.

Setting the DB2_WORKLOAD Registry Variable Continued Memory for sorting, caching and hashing will be set based on the amount of RAM and server size CATALOGCACHE_SZ, SHEAPTHRES_SHR and SORTHEAP set to a value that is optimized for the hardware you are using Intraquery Parallelism enabled for any workload even if DBM CFG parameter INTRA_PARALLEL=OFF The default concurrency threshold on the SYSDEFAULTMANAGEDSUBCLASS service subclass is enabled to ensure maximum efficiency and utilization of the server Automatic Space Reclamation is enabled AUTO_MAINT=ON AUTON_REORG=ON Page and extent size configured for analytical processing Enables automatic space reclamation Compression is always on NOTES:

Setting the DB2_WORKLOAD Registry Variable Continued Automatic Space Reclamation is enabled AUTO_MAINT=ON AUTO_REORG=ON Space reclamation performed for column-organized tables by default NOTES: Set the DB2_WORKLOAD registry variable to ANALYTICS before creating any database.

Use of AUTOCONFIGURE For an existing database that you want to configure for BLU Acceleration Set the DB2_WORKLOAD=ANALYTICS registry variable and restart the instance Connect to the database you want to run autoconfigure on Run “db2 autoconfigure apply dbname” After running autoconfigure, review the bufferpool. SHEAPTHRES_SHR, SORTHEAP , UTIL_HEAP_SZ and adjust as necessary NOTES:

Use of AUTOCONFIGURE In an instance with multiple databases some of which are not using BLU, set the registry variable before you create the databases that will be using BLU, then unset the registry variable after creating the databases that will use BLU NOTES:

Setting SHEAPTHRES_SHR and SORTHEAP Parameters When BLU Acceleration is enabled, the self-tuning memory manager (STMM) does not adjust the values of the SHEAPTHRES_SHR and SORTHEAP database configuration parameters automatically. Set with static values by the AUTOCONFIGURE command when DB2_WORKLOAD=ANALYTICS registry setting is enabled NOTES:

Setting SHEAPTHRES_SHR and SORTHEAP Analytical workloads require larger sort memory areas due to the nature of the workload Optimal setting provides degree of concurrency required and associated SORTHEAP sizing Setting SHEAPTHRES_SHR to AUTOMATIC not supported for databases using column organized tables Tuning SHEAPTHRES_SHR and SORTHEAP as important as tuning BUFFERPOOLS Aim to prevent sort overflows to disk which can significantly slow down queries SET SEHAPTHRES_SHR to 40-50% of the memory available to the database DATABASE_MEMORY DB CFG Parameter NOTES: The SHEAPTHRES_SHR database configuration parameter represents a soft limit on the total amount of database shared memory that can be used by sort memory consumers at any one time. Set to handle degree of concurrent sort consumers (nbr of sort consumers x sortheap setting).

Setting SHEAPTHRES_SHR and SORTHEAP The SORTHEAP DB CFG parameter controls the amount of memory an application can use for SORTS SORTHEAP memory comes out of total SHEAPTHRES_SHR memory Size the SHEAPTHRES_SHR DB CFG parameter and SORTHEAP DB CFG parameter to provide the amount of concurrency desired and to prevent sort overflows Rule of thumb for sizing SORTHEAP is set it to 5-20% of SHEAPTHRES_SHR Monitor for sort overflows and post threshold sorts via MON_GET_DATABASE and MON_GET_UNIT_OF_WORK table functions NOTES: The higher the degree of sort concurrency requires, increase SHEAPTHRES_SHR and reduce the SORTHEAP DB CFG parameter. This should be done in small increments and through dedicated monitoring sessions.

Monitoring SHEAPTHRES_SHR and SORTHEAP Monitor for sort overflows and post threshold sorts via MON_GET_DATABASE and MON_GET_UNIT_OF_WORK table functions Adjust SHEAPTHRES_SHR upward and SORTHEAP upward gradually to eliminate sort overflows Don’t increase SORTHEAP without concomitant adjustment to SORTHEAP Can decrease SORTHEAP independently Mine the DYNAMIC SQL and target queries with high number of overflows Good way to train a DBA in tuning SQL or identifying proper indexes to eliminate SORTS NOTES:

Setting the STMTHEAP DB CFG Parameter Since analytic queries are more complex than OLTP type queries, they can benefit from an increased STMTHEAP DB CFG size The STMTHEAP DB CFG parameter specifies the size of the workspace used by the SQL compiler during compilation of an SQL statement NOTES: STMTHEAP is essentially a work space used by the SQL Compiler during compilation of an SQL statement. If you have queries that receive an SQL0437W with reason code 1, adjust the STMTHEAP size by 50% and continue to increment by same percentage as needed.

Importance of Setting UTIL_HEAP_SZ Correctly For LOAD operations and for building compression dictionaries UTIL_HEAP_SZ must be sufficiently sized Set to AUTOMATIC if DB2_WORKLOAD=ANALYTICS If DATABASE_MEMORY set to AUTOMATIC should be sufficient as it will grow as necessary If not sized sufficiently, can result in less than optimal compression NOTES:

Monitoring Adequacy of UTIL_HEAP_SZ setting Check the new PCTENCODED column in the SYSCAT.COLUMNS catalog view which is the the percentage of values that are encoded as a result of compression for a column in a column-organized table. If the overall compression ratio for you’re the table in is too low, check this statistic to see if values in specific columns were left uncompressed. Many columns with a very low value (or even 0) can be an indicator that utility heap might have been too small when the column compression dictionaries were created You might also see very low values for columns that were incrementally loaded with data that was outside of the scope of the column compression dictionary NOTES:

So, How are your Column-organized Tables Performing? Use monitor elements returned by the MON_GET_TABLE table function to determine the efficiency of column-organized tables and to identify candidate tables suitable to be organized by column NOTES:

New Column-organized Metrics from MON_GET_TABLE table function Use monitor elements returned by the MON_GET_TABLE table function to determine the efficiency of column-organized tables and to identify candidate tables suitable to be organized by column By dividing the NUM_COLUMNS_REFERENCED monitor element by the SECTION_EXEC_WITH_COL_REFERENCES monitor element, you can determine the average number of columns that a particular query accesses. If this average is much less than the number of columns in the table, the workload favors column organization. Controlled by the mon_obj_metrics DB CFG parameter setting Extended is the default and is required to return column-organized metrics NOTES:

Monitor Collection DB CFG Parameters (Review) Mon_act_metrics – controls collection of activity level monitor elements on the entire database (DEFAULT – BASE) MON_GET_ACTIVITY_DETAILS MON_GET_PKG_CACHE_STMT Activity event monitor (DETAILS_XML monitor element in the event_activity logical data groups) Mon_deadlock – controls generation of deadlock events on the entire database (DEFAULT- WITHOUT_HIST) Mon_locktimeout – controls generation of lock timeout events on the entire database (DEFAULT – NONE) Mon_lockwait – controls generation of lock wait events for the lock event monitor (DEFAULT – NONE) Mon_lw_thresh – the amount of time spent in lock wait before an event for mon_lockwait is generated (DEFAULT - 5000000) Mon_obj_metrics – controls collection of data object monitor elements on the entire database (DEFAULT- EXTENDED *formerly BASE) MON_GET_TABLE NOTES:

Monitor Collection DB CFG Parameters (Review) Mon_req_metrics – controls the collection of request monitor elements on the entire database (DEFAULT – BASE) MON_GET_UNIT_OF_WORK MON_GET_UNIT_OF_WORK_DETAILS MON_GET_CONNECTION MON_GET_CONNECTION_DETAILS MON_GET_SERVICE_SUBCLASS MON_GET_SERVICE_SUBCLASS_DETAILS MON_GET_WORKLOAD MON_GET_WORKLOAD_DETAILS Statistics event monitor (DETAILS_XML monitor element in the event_wlstats and event_scstats logical data groups) Unit of work event monitor Mon_uow_data – controls the generation of UOW events at the database level for the UOW event monitor (DEFAULT – NONE) NOTES:

Monitoring and MONREPORT Reporting Enhancements MONREPORT.DBSUMMARY and MONREPORT.CONNECTION reporting modules enhanced to support on column processing New monitor elements to assess bufferpool efficiency Monitor elements contain COL in the monitoring element name New monitor elements to monitor prefetch performance for data in column organized tables New monitor elements to measure column data size ADMIN_GET_TAB_INFO New monitor elements to measure time spent New monitor elements for the HASHED GROUP BY operator New monitor interfaces to get request metrics that are aggregated along different dimensions New monitor interfaces to get activity metrics Monitor interfaces to get database object metrics Refer to Database Monitoring Guide and Reference for complete details NOTES:

New CTQ Optimization Operator Use to denote operations on column organized tables Most processing should be below the CTQ operator in the access path (db2exfmt, db2explain) Optim Performance Manager Optim Query Tuner If many operations above this operator indicator of inefficient use of column organized table Re-look how table is being used NOTES:

Reviewing time spent in column processing versus row processing with new monitor elements New monitoring elements returned from MON_GET_ACTIVITY_DETAILS tabe function to show where time is being spent Total_app_section_excutions Total_col_executions Total_col_proc_time Total_col_time Total_cpu_time NOTES:

Understanding Automatic Space Reclamation Space reclamation for column-organized tables Space reclamation after deletions After a delete operation the row is logically deleted Space that was used by the deleted row is not available to subsequent transactions and remains unusable until space reclamation occurs REORG TABLE is required to reclaim the space NOTES:

Understanding Automatic Space Reclamation Space reclamation after updates After a delete operation the row is first deleted and a new copy of the row inserted at the end of the table Updated row uses space in proportion to the number of times that the row is updated until space reclamation occurs All rows in the extent where the update took place must be deleted before any space reclamation can occur REORG TABLE is required to reclaim the space NOTES:

Understanding Automatic Space Reclamation If DB2_WORKLOAD=ANALYTICS then AUTO_REORG enabled Define an automatic reorg online and/or offline policy using the AUTOMAINT_SET_POLICY or AUTOMAINT_SET_POLICYFILE stored procedures with the appropriate parameters Offline reorg considerations Need appropriate window Reclaims space Faster than online REORG Sample POLICYFILE in SAMPLE DB2 directory NOTES:

Automatic Space Reclamation Online REORG considerations Still need appropriate window Schedule ONLINE REORG window during period of low activity Reclaims space by specifying Specify RECLAIM EXTENTS option of REORG command to reclaim space To determine when a table REORG is necessary, use the ADMIN_GET_TAB_INFO table function to review an estimate of the amount of reclaimable space to reclaim Monitor REORG status with db2pd or SNAPTAB_REORG administrative view NOTES:

Summary DB2 BLU New Column-organized capabilities Use of AUTOCONFIGURE Review of DB2_WORKLOAD=ANALYTICS registry setting SHEAPTHRES_SHR and SORTHEAP setting discussion and recommendations Table monitoring metrics Monitoring Enhancements and some review Activity detail monitoring elements New Monitoring Elements Importance of adequate UTIL_HEAP_SZ REORG Considerations Space Usage and Reclamation NOTES:

References Tuning and Monitoring System Performance, located at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Tuning%20and%20Monitoring%20Database%20System%20Performance DB2 10.5 Database Monitor Guide and Reference, http://public.dhe.ibm.com/ps/products/db2/info/vr105/pdf/en_US/DB2Monitoring-db2f0e1050.pdf Best Practices, Optimizing Analytical Workloads using DB2 10.5 with BLU Acceleration, https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Optimizing%20analytic%20workloads%20using%20DB2%2010.5%20with%20BLU%20Acceleration/comment/1dc59ee4-8c70-43f9-9d36-fba334b95891 NOTES:

References DB2 BLU Acceleration, Rapid Adoption Guide, http://www.ibm.com/developerworks/data/library/techarticle/dm-1309db2bluaccel/ When RAM is too Slow, http://softwaretradecraft.com/when-ram-is-too-slow-how-dynamic-in-memory-processing-changes-the-game-for-data-analytics/ DB2 10.5, What’s New, http://public.dhe.ibm.com/ps/products/db2/info/vr105/pdf/en_US/DB2WhatsNew-db2q0e1050.pdf NOTES:

Monitoring the Essentials in DB2 10.5 DB2 LUW Philip K. Gunning Gunning Technology Solutions, LLC pgunning@gts1consulting.com Monitoring the Essentials in DB2 10.5 DB2 LUW Please fill out your session evaluation before leaving!