Waits Defined Kyle Hailey

#.2 Copyright 2006 Kyle Hailey Top 36 Foreground Waits write complete waits library cache lock SQL*Net more data from dblink log file switch (checkpoint incomplete) library cache load lock row cache lock local write wait sort segment request process startup unread message file identify pipe put switch logfile command SQL*Net break/reset to dblink log file switch (archiving needed) Wait for a undo record direct path write (lob) undo segment extension 1. 1.db file sequential read 2. 2.log file sync 3. 3.db file scattered read 4. 4.latch free 5. 5.enqueue 6. 6.SQL*Net more data from client 7. 7.direct path read 8. 8.direct path write 9. 9.buffer busy waits SQL*Net more data to client log buffer space log file switch completion library cache pin SQL*Net break/reset to client io done file open free buffer waits db file parallel read minus OPS, RAC, PQ, Resource Mgr

#.3 Copyright 2006 Kyle Hailey Tuning Methodology  Machine  Run queue (CPU)  reduce CPU usage or add CPUs  Paging  Reduce memory usage or add memory  Oracle  Waits >> CPU  Tune waits We are going to concentrate here on WAITS

#.4 Copyright 2006 Kyle Hailey For Analysis  CPU Time  For comparision with Wait Time  Wait Time  Top Waits  If Wait Time >> CPU Time, need top waits  Wait Arguments p1, p2, p3  For detailed analysis of some waits  SQL that caused waits  For detailed analysis of some waits

#.5 Copyright 2006 Kyle Hailey Wait and CPU Data Sources I Statspack Top 5 Timed Events II 10g ASH OEM ASH Report : ashrpt.sql Custom queries on v$active_session_history III V$session & v$session_wait Custom queries

#.6 Copyright 2006 Kyle Hailey I Statspack  Look at Top 5 Timed Events Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space

#.7 Copyright 2006 Kyle Hailey II OEM 10g Performance Tab

#.8 Copyright 2006 Kyle Hailey II OEM 10g Zoom-In

#.9 Copyright 2006 Kyle Hailey II Top Activity Page

#.10 Copyright 2006 Kyle Hailey II 10g v$active_session_history  The best source  Only in 10g  Can simulate in any version since v7

#.11 Copyright 2006 Kyle Hailey II Querying ASHselect event as "TOP 5 Timed Events", event as "TOP 5 Timed Events", cnt as "Count" from ( cnt as "Count" from ( select decode(ash.session_state, select decode(ash.session_state, 'ON CPU','CPU',en.name) event, 'ON CPU','CPU',en.name) event, count(decode(ash.session_state, count(decode(ash.session_state, 'ON CPU','CPU',en.name)) cnt 'ON CPU','CPU',en.name)) cntfrom v$active_session_history ash, v$active_session_history ash, v$event_name en v$event_name en where en.event# (+) = ash.event# and SAMPLE_TIME > sysdate - (5/(24*60)) and SAMPLE_TIME > sysdate - (5/(24*60)) group by decode(ash.session_state,'ON CPU','CPU',en.name) order by count(decode(ash.session_state, 'ON CPU','CPU',en.name)) desc 'ON CPU','CPU',en.name)) desc ) where rownum < 6; select event as "TOP 5 Timed Events", event as "TOP 5 Timed Events", cnt as "Count" from ( cnt as "Count" from ( select decode(ash.session_state, select decode(ash.session_state, 'ON CPU','CPU',en.name) event, 'ON CPU','CPU',en.name) event, count(decode(ash.session_state, count(decode(ash.session_state, 'ON CPU','CPU',en.name)) cnt 'ON CPU','CPU',en.name)) cntfrom v$active_session_history ash, v$active_session_history ash, v$event_name en v$event_name en where en.event# (+) = ash.event# and SAMPLE_TIME > sysdate - (5/(24*60)) and SAMPLE_TIME > sysdate - (5/(24*60)) group by decode(ash.session_state,'ON CPU','CPU',en.name) order by count(decode(ash.session_state, 'ON CPU','CPU',en.name)) desc 'ON CPU','CPU',en.name)) desc ) where rownum < 6; Top 5 Timed Events Count log file parallel write 8 CPU 7 control file parallel write 3 null event 3 log file sync 2 Top 5 Timed Events Count log file parallel write 8 CPU 7 control file parallel write 3 null event 3 log file sync 2

#.12 Copyright 2006 Kyle Hailey II Querying ASH for P1,P2,P3 SID EVENT P1 P2 P db file scattered read db file scattered read read by other session db file scattered read db file scattered read db file sequential read SID EVENT P1 P2 P db file scattered read db file scattered read read by other session db file scattered read db file scattered read db file sequential read selectsession_id,event, p1, p2, p3 from v$ash selectsession_id,event, p1, p2, p3 from v$ash

#.13 Copyright 2006 Kyle Hailey II ASH Report  ASH report  Pick interval over last 7 days ! 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***

#.14 Copyright 2006 Kyle Hailey III V$session_wait select decode(w.wait_time, 0, w.event, 'CPU') as "TOP 5 Timed Events”, count(*) from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply') group by decode(w.wait_time, 0, w.event, 'CPU') order by count(*) desc; select decode(w.wait_time, 0, w.event, 'CPU') as "TOP 5 Timed Events”, count(*) from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply') group by decode(w.wait_time, 0, w.event, 'CPU') order by count(*) desc; Moment in Time data

#.15 Copyright 2006 Kyle Hailey III V$session_wait Top 5 Timed Events Top 5 Timed Events COUNT enq: US – contention 4 CPU 3 buffer busy waits 1 latch: row cache objects 1 Top 5 Timed Events Top 5 Timed Events COUNT enq: US – contention 4 CPU 3 buffer busy waits 1 latch: row cache objects 1

#.16 Copyright 2006 Kyle Hailey III V$session_wait col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event, 'CPU') as status, w.p1, w.p2, w.p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply'); col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event, 'CPU') as status, w.p1, w.p2, w.p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply');

#.17 Copyright 2006 Kyle Hailey III V$session_wait SID SQL_HASH STATUS P1 P2 P CPU enq: US - contention enq: US - contention enq: US - contention enq: US - contention enq: US - contention CPU enq: US - contention CPU SID SQL_HASH STATUS P1 P2 P CPU enq: US - contention enq: US - contention enq: US - contention enq: US - contention enq: US - contention CPU enq: US - contention CPU Moment in Time data

#.18 Copyright 2006 Kyle Hailey Waits we will Ignore  Background  Idle  RAC  Resource Manager

#.19 Copyright 2006 Kyle Hailey Background Waits Filter Out Background Waits  Statspack  ASH : SESSION_TYPE='FOREGROUND‘  V$session_wait : type='USER'

#.20 Copyright 2006 Kyle Hailey Background Waits  ASH  Avoid Background waits in ASH with  V$session_wait joined to v$session Select …from v$active_session_history where SESSION_TYPE='FOREGROUND' select … from v$session s, v$session_wait w where w.sid=s.sid and s.type='USER' select … from v$session s, v$session_wait w where w.sid=s.sid and s.type='USER'

#.21 Copyright 2006 Kyle Hailey Idle Waits  Filter Out  10g  where wait_class != ‘Idle’  Create a list  9i  Create a list with  Documentation  List created from 10g Select name from v$event_name where wait_class=‘Idle’;

#.22 Copyright 2006 Kyle Hailey Parallel Query Waits Filter Out  Parallel Query Wait events are unusable  Save waits are both idle and waits  Parallel Query Waits start with ‘PX’ or ‘KX’  PX Deq: Par Recov Reply  PX Deq: Parse Reply

#.23 Copyright 2006 Kyle Hailey RAC Waits  You are on your own  Check documentation  If you are not using RAC then no worries  10g  9i  RAC and OPS waits usually contain the word “global” Select event from v$event_name where wait_class=‘Cluster’;

#.24 Copyright 2006 Kyle Hailey Resource Manager Waits  Resource manager throttles user  Creates wait  Obfuscates problems  10g select name from v$event_name where wait_class='Scheduler';

#.25 Copyright 2006 Kyle Hailey Wait Arguments: P1,P2,P3  Each Wait has a 3 parameters P1,P2,P3  Give detailed information  Meaning different for each wait  Meaning definitions in V$event_name Select name, parameter1, parameter2, parameter3 from v$event_name;

#.26 Copyright 2006 Kyle Hailey Wait Arguments Example NAME PARAMETER1 PARAMETER2 PARAMETER latch: cache buffers chains address number tries free buffer waits file# block# set-id# buffer busy waits file# block# class# latch: redo copy address number tries log buffer space switch logfile command log file sync buffer# db file sequential read file# block# blocks enq: TM - contention name|mode object # table/partition undo segment extension segment# enq: TX - row lock contention name|mode usn<<16 | slot sequence row cache lock cache id mode request library cache pin handle address pin address 100*mode+namesp library cache load lock object address lock address 100*mask+namesp pipe put handle address record length timeout

#.27 Copyright 2006 Kyle Hailey Waits I/O Library Cache Locks Undo Redo Buffer Cache SQL*Net Wait Tree

#.28 Copyright 2006 Kyle Hailey Wait Tree - Heirarchy Waits Disk I/O Buffer Busy Library Cache Log Buffer Enqueue Log File Sync Log File Undo Free Buffer Hot Blocks Free lists IO Read Library Cache Latch Library Cache Pin Shared Pool Latch TX Row Lock TX ITL Lock HW Lock Redo Latches Redo Buffer Cache SQL*Net

#.29 Copyright 2006 Kyle Hailey Top 12 Waits NAME Count % Total 1.db file sequential read 23, % 2.log file sync 20, % 3.db file scattered read 15, % 4.latch free 11, % 5.enqueue 7, % 6.SQL*Net more data from client 7, % 7.direct path read 5, % 8.direct path write 4, % 9.buffer busy waits 4, % 10.SQL*Net more data to client 3, % 11.log buffer space 2, % 12.log file switch completion 2, % Above is over 80% of wait times reported Anjo Kolk

#.30 Copyright 2006 Kyle Hailey Top 36 Foreground Waits write complete waits library cache lock SQL*Net more data from dblink log file switch (checkpoint incomplete) library cache load lock row cache lock local write wait sort segment request process startup file identify pipe put switch logfile command SQL*Net break/reset to dblink log file switch (archiving needed) Wait for a undo record direct path write (lob) undo segment extension undo segment tx slot 1. 1.db file sequential read 2. 2.log file sync 3. 3.db file scattered read 4. 4.latch free 5. 5.enqueue 6. 6.SQL*Net more data from client 7. 7.direct path read 8. 8.direct path write 9. 9.buffer busy waits SQL*Net more data to client log buffer space log file switch completion library cache pin SQL*Net break/reset to client io done file open free buffer waits db file parallel read minus OPS, RAC, PQ, Resource Mgr

#.31 Copyright 2006 Kyle Hailey Waits db file sequential read Tune SQL, speed up disks (5-15ms), increase buffer cache 2.log file sync Commit less, put redo logs on faster disks 3.db file scattered read FTS - Tune SQL, add indexes, speed up disks (5-15ms) 4.latch free – need p1,p2,p3 5.enqueue – need p1,p2,p3 and SQL 6.SQL*Net more data from client Usually OK, reduce data transferred, possible Network problems 7.direct path read sorts or PQO - tune IO, sort less 8.direct path write direct path load or temp io, improve disk speed 9.buffer busy waits – need p1,p2,p3 and SQL

#.32 Copyright 2006 Kyle Hailey Waits SQL*Net more data to client Usually OK, reduce amount of data transferred, possible Network tuning needed 11.log buffer space Increase log buffer 12.log file switch completion Increase log file sizes 13.library cache pin Reduce parsing the same cursor concurently 14.SQL*Net break/reset to client Check for errors in sql statement 15.io done Io done by IO slaves, oracleIxxx could also be an idle wait – sort of worthless 16.file open Reduce logons/logoffs or keep log files open 17.free buffer waits Increase buffer cache size 18.db file parallel read Tune sql, tune io subsystem, increase buffer cache

#.33 Copyright 2006 Kyle Hailey Waits write complete waits Increase buffer cache (or tune DBWR) 20.library cache lock Avoid hard parsing same cursor at same time 21.SQL*Net more data from dblink Reduce data transfer, check net response 22.log file switch (checkpoint incomplete) Add log files (or increase log file size) 23.library cache load lock Avoid hard parsing same cursor at same time 24.row cache lock need p1 25.local write wait truncates / reduce cache size 26.sort segment request SMON busy, process rollback problem 27.process startup If MTS, increase min servers and dispatchers in init.ora

#.34 Copyright 2006 Kyle Hailey Waits file identify Keep log files open, reduce checkpoints 29.pipe put Speed up pipe readers 30.switch logfile command Avoid switching log files 31.SQL*Net break/reset to dblink Check for errors in sql statement sent 32.log file switch (archiving needed) Archive log running out of space 33.Wait for a undo record ?? 34.direct path write (lob) Improve IO, reduce lob write size 35.undo segment extension Use UNDO or with RBS, increase RBS size, avoid OPTIMAL 36.undo segment tx slot Use UNDO, increae # of RBS segs

#.35 Copyright 2006 Kyle Hailey Difficult Waits  Latches  Locks  Buffer Busy  Row Cache Lock

#.36 Copyright 2006 Kyle Hailey Concepts: Latches  Protect areas of memory from concurrent use  Light weight locks  Usually a held in a bit in memory  Often supported by atomic processor call  Fast and cheap  Gone if memory is lost  Often used in cache coherency management  Changes to a datablock  Exclusive Generally  Sharing reading has been introduced for some latches

#.37 Copyright 2006 Kyle Hailey Finding Latches  “latch free”  Covers many latches, find the problem latch by 1. select name from v$latchname where latch# = p1; OR 2. Find highest sleeps in Statspack latch section  In 10g, important latches have a wait event  latch: cache buffers chains  latch: shared pool  latch: library cache

#.38 Copyright 2006 Kyle Hailey Latches : Find Highest Sleeps  Use Statspack (or AWR report) Sleeps Latch Name Get Requests Misses Sleeps Spin Gets Sleep ,098 shared pool 3,044, ,517 19, , ,198 library cache 2,242,805 79,554 15,198 66, ,774 kks stats 263,464 2,842 1,774 1, row cache object 434, library cache lock 666, library cache pin 741, enqueues 133, enqueue hash chains 133, Sleeps Latch Name Get Requests Misses Sleeps Spin Gets Sleep ,098 shared pool 3,044, ,517 19, , ,198 library cache 2,242,805 79,554 15,198 66, ,774 kks stats 263,464 2,842 1,774 1, row cache object 434, library cache lock 666, library cache pin 741, enqueues 133, enqueue hash chains 133,

#.39 Copyright 2006 Kyle Hailey Important Latches  Cache Buffers Chains  Hot blocks, distribute activity, reduce update time, avoid “select for update”  Cache Buffers LRU Chain  Too much activity, uses multiple buffer caches and/or increse LRU latches  Shared Pool  Too much hard parsing, too small a shared pool  Library Cache Latch  Hard Parsing  Use bind variables  Uses cursor_sharing=force  Increase shared pool size if reloads are high  Avoid invalidations  Soft Parsing  Hold cursors open  Use session_cached_cursors  Use cursor_space_for_time

#.40 Copyright 2006 Kyle Hailey Concepts: Locks  “Enqueue” wait – covers all locks pre 10  Protect data against concurrent changes  Lock info written into data structures  Block headers  Data blocks  Written in cache structures  Shareable in compatible modes

#.41 Copyright 2006 Kyle Hailey Finding Locks  Statspack not much help  V$session_wait needs lots of decoding  P1 tells Lock Type and Mode  P2,P3 give more data  Usually Need SQL to solve select sid, event, chr(bitand(P1, )/ )|| chr(bitand(P1, )/65535) as "Type", mod(p1,16) as "mode" from v$session_wait where event = 'enqueue‘; select sid, event, chr(bitand(P1, )/ )|| chr(bitand(P1, )/65535) as "Type", mod(p1,16) as "mode" from v$session_wait where event = 'enqueue‘; SID EVENT Ty mode P2 P enqueue TX SID EVENT Ty mode P2 P enqueue TX

#.42 Copyright 2006 Kyle Hailey Locks 10g  10g breaks Enqueues out  enq: HW - contention Configuration  enq: TM - contention Application  enq: TX - allocate ITL entry Configuration  enq: TX - index contention Concurrency  enq: TX - row lock contention Application  enq: UL - contention Application

#.43 Copyright 2006 Kyle Hailey Locks : TM & TX select * from v$lock where type in ('TX', 'TM'); SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK TX TM TM TX select * from v$lock where type in ('TX', 'TM'); SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK TX TM TM TX

#.44 Copyright 2006 Kyle Hailey Buffer Busy Cases  Undo Header - Not enough rollback segments  Undo Block – hot spot in RBS  Data Block  Table Block w/ DML : Table lacks Free lists  Table I/O : Multiple sessions reading same block into data cache  Note: in 10g, becomes “read by other session”  Index : has hot block, partition or change SQL  Segment Header – add free Lists (with Datablock)  Free List – add free lists groups

#.45 Copyright 2006 Kyle Hailey Buffer Busy Waits  Statspack Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: > ordered by wait time desc, waits desc Tot Wait Avg Class Waits Time (s) Time (ms) data block 1, undo header Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: > ordered by wait time desc, waits desc Tot Wait Avg Class Waits Time (s) Time (ms) data block 1, undo header 2 0 0

#.46 Copyright 2006 Kyle Hailey Buffer Busy Solutions  IF Buffer Busy Waits high  First look at v$waitstat  Class  Time/Count SQL> select * from v$waitstat; CLASS COUNT TIME data block 1 1 segment header 0 0 free list 0 0 undo header 0 0 undo block 0 0 SQL> select * from v$waitstat; CLASS COUNT TIME data block 1 1 segment header 0 0 free list 0 0 undo header 0 0 undo block 0 0

#.47 Copyright 2006 Kyle Hailey Row Cache Lock  Need p1 to see the cache type SQL> select cache#, parameter from v$rowcache; CACHE# PARAMETER dc_free_extents 4 dc_used_extents 2 dc_segments 0 dc_tablespaces 5 dc_tablespace_quotas 6 dc_files 7 dc_users 3 dc_rollback_segments 8 dc_objects 17 dc_global_oids 12 dc_constraints SQL> select cache#, parameter from v$rowcache; CACHE# PARAMETER dc_free_extents 4 dc_used_extents 2 dc_segments 0 dc_tablespaces 5 dc_tablespace_quotas 6 dc_files 7 dc_users 3 dc_rollback_segments 8 dc_objects 17 dc_global_oids 12 dc_constraints

#.48 Copyright 2006 Kyle Hailey Row Cache Lock  Statspack ^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: >"Pct Misses" should be very low (< 2% in most cases) ->"Cache Usage" is the number of cache entries being used ->"Pct SGA" is the ratio of usage to allocated size for that cache Get Pct Scan Pct Mod Final Cache Requests Miss Reqs Miss Reqs Usage dc_object_ids dc_objects ,129 dc_segments dc_tablespaces dc_usernames dc_sequences 120, ,003 5 ^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: >"Pct Misses" should be very low (< 2% in most cases) ->"Cache Usage" is the number of cache entries being used ->"Pct SGA" is the ratio of usage to allocated size for that cache Get Pct Scan Pct Mod Final Cache Requests Miss Reqs Miss Reqs Usage dc_object_ids dc_objects ,129 dc_segments dc_tablespaces dc_usernames dc_sequences 120, ,003 5

#.49 Copyright 2006 Kyle Hailey Areas of Contention  Buffer Cache  Disk I/O  Library Cache  Redo  SQL*Net  Undo We will go through these areas and the wait events that are relevant to them.