1. 1 3/21/2016 Average Active Sessions (AAS) The Golden Metric ? Kyle Hailey perfvision.com

2. Copyright 2006 Kyle Hailey Who is Kyle Hailey 1990 Oracle 90 support 92 Ported v6 93 France 95 Benchmarking 98 ST 2000 Dot.Com 2001 Quest 2002 Oracle OEM 10g 2005 Embarcadero 2006 Independent When not being being a Geek writing for the Oaktable etc - Just Married and new home owner in Portland - photographer - and wonder how I missed the dot.com millions

3. 3 3/21/2016 Copyright 2006 Kyle Hailey Goal of this Presentation Cut out unnecessary Hone in on essential Simplify the data and empower the DBA

4. Graphics “The human nervous system … is exceptionally good at parsing visual information, especially when that information is coded by color and/or motion.” Knowledge representation in cognitive science. Westbury, C. & Wilensky, U. (1998)

5. 5 3/21/2016 Database Performance When it comes to Database Performance How quick can you find Bottleneck in DB If DB is idle Current DB Load what is DB Load ? What do you use? Statspack/AWR V$active_session_history Alerts what do you alert on ? Or as Tanel said “ what the *!####!*!*? is the database doing ?!

6. 6 3/21/2016 Copyright 2006 Kyle Hailey In this Session 1. AAS Single Metric Shows DB Performance 2. Yardstick Max CPU CPU Count To measure AAS against 3. Subcomponents CPU Waits Time series +Geek stuff on ASH & SASH for Hotsos

7. 7 3/21/2016 Statspack / AWR 1000 lines of data / 30 pages What do you look at ? Top 5 Timed Events? Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time ------------------------ ------------ ----------- -------- CPU time 11,330 81.95 log file sync 163,725 1,012 7.32 enqueue 4,486 615 4.45 latch free 28,786 236 1.71 db file sequential read 1,420,355 149 1.08 ----------------------------------------- Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html 188 minutes CPU Elapsed 15 minutes 48 Processors or 74% idle

8. 8 3/21/2016 Copyright 2006 Kyle Hailey What’s the DB Doing?! It’s 2am … your manager calls Whip out the stethoscope: AAS As Tanel said “ what the *!####!*!*? is the database doing ?!

9. 9 3/21/2016 Copyright 2006 Kyle Hailey Welcome to … The Cult of AAS Once you’ve been initiated … there is no going back For those of you who are already members, welcome back

10. 10 3/21/2016 Copyright 2006 Kyle Hailey Average Active Sessions Centi-seconds per second In the dark ages waits were often as Centi-secs per sec Sometimes called secs/sec Usually didn’t include CPU time, which is essential Average Active Sessions (AAS) OEM 10g Graphs ASH Report Session Load I often refer to it this way

11. 11 3/21/2016 Copyright 2006 Kyle Hailey AAS Average load of sessions on the database Average over 15 secs in OEM 10g Period varies in other tools measured in number of sessions Active Sessions Only Active sessions put load the database Inactive Sessions don’t put load ( Except for memory usage ) The Golden Metric Powerful Multidimensional Indispensable

12. 12 3/21/2016 Copyright 2006 Kyle Hailey AAS Calculation DB TIME AAS = DB TIME / Elapsed Time DB Time = sum over all sessions of time spent in a call ex:Session executing SQL Background process working

13. 13 3/21/2016 Copyright 2006 Kyle Hailey AAS Sources 1. Manually from v$sysstat (9i ) v$system_event 2. Statspack Need several calculations 3. AWR One calculation 4. OEM 10g Directly displayed

14. 14 3/21/2016 Copyright 2006 Kyle Hailey 1. Manually DB Time = sum active session time DB TIME (10g) DB TIME = (9i) Select sum(time_waited) from v$system_event where event not in (... idle events …); + Select value from v$sysstat where name = ‘CPU used by this session’; Select sum(time_waited) from v$system_event where event not in (... idle events …); + Select value from v$sysstat where name = ‘CPU used by this session’; select value from v$sysstat where name = ‘DB time’; select value from v$sysstat where name = ‘DB time’; ‘DB time’

15. 15 3/21/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS Look for Elapsed Time Top 5 Timed Events Start at line 52 of about 1300!Start at line 52 of about 1300!

16. 16 3/21/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS Elapsed Time Look at Top 5 Timed Events Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time --------------------- --------- --------- ------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time --------------------- --------- --------- ------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host ------- ----------- -------- -------- ---------- --- ------- LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr Snap Id Snap Time Sessions Curs/Sess --------- ------------------ -------- --------- Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins) STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host ------- ----------- -------- -------- ---------- --- ------- LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr Snap Id Snap Time Sessions Curs/Sess --------- ------------------ -------- --------- Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins)

17. 17 3/21/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS DBTIME= CPU + WAITS CPU = 32 WAITS = 250+15+8+5 = 278 secs DBTIME=320 Elapsed Time = 60 secs 320 secs / 60 secs AAS = 5.1 Top 5 Timed Events Event Time (s) ----------------- ----- buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5 Top 5 Timed Events Event Time (s) ----------------- ----- buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5

18. 18 3/21/2016 3. AWR Report AAS = DB Time/Elapsed Time 23.56/59.66 = 0.39 AAS= 0.39

19. 19 3/21/2016 4. OEM 10g AAS = ~0.75 OEM 10g

20. 20 3/21/2016 Copyright 2006 Kyle Hailey Got AAS, Now What ? We Need one more item: CPU Count # of CPUs available on System Shared with other applications Need to track CPU used on the system as well On dual & quad cores, lower the CPU count Represents max active sessions that can do work

21. 21 3/21/2016 Copyright 2006 Kyle Hailey CPU Count # of CPUs available in Statspack 10g AWR report OEM 10g Statspack 9i # of CPUs missing # of CPUs SQLPLUS> show parameters cpu_count NAME VALUE ------------------ ---------- cpu_count 2 SQLPLUS> show parameters cpu_count NAME VALUE ------------------ ---------- cpu_count 2

22. 22 3/21/2016 Copyright 2006 Kyle Hailey AAS Formulas Use CPU count as yardstick: < 1 Database is not blocked AAS ~= 0 Database basically idle Problems are in the APP not DB AAS < # of CPUs CPU available Database is probably not blocked Are any single sessions 100% active? AAS > # of CPUs Could have performance problems  AAS >> # of CPUS There is a bottleneck Ideal world – one database solution track CPU at OS AAS > 1 still want to know if a single user is 100% active

23. 23 3/21/2016 Copyright 2006 Kyle Hailey Available CPU vs AAS AAS far above available CPU => problem AAS = 5.1 # of CPU = 2 AAS = 0.39 # of CPU = 2 AAS < 1, database is fine

24. 24 3/21/2016 Copyright 2006 Kyle Hailey Going Farther with AAS AAS can tell you a lot But it’s components tell you much more To go farther need the components of AAS 1.CPU 2.Wait 3.Value over time Only OEM 10g has all 3 (Statspack and AWR are aggregated over the snapshot period)

25. 25 3/21/2016 AAS Components : OEM 10g OEM 10g Performance Page Available CPU AAS: CPU + WAIT Real CPU available: Max CPU - non instance CPU

26. 26 3/21/2016 OEM 10g Relax Get to Work! Looks OK But …

27. 27 3/21/2016

28. 28 3/21/2016 Limited Analysis What if you find a problem ? Of the 800 waits most need SQL and/or P1,P2,P3 to solve What if there is a wait bottleneck ? What SQL Which sessions Values of P1, P2 and P3 Statspack and AWR fail Example “easy” waits free buffer waits log buffer space log file switch (archiving needed) log file switch (checkpoint incomplete) log file switch completion Example “hard” waits Buffer busy wait Row cache lock Latch free row lock contention Latch: cache buffers chains

29. 29 3/21/2016 Copyright 2006 Kyle Hailey Alternative AAS Calc AAS = DB TIME / Elapsed Time But there is another way …

30. 30 3/21/2016 Copyright 2006 Kyle Hailey AAS based on ASH ASH - Active Session History v$active_session_history AAS = count(*) / elapsed_seconds A statistical approximation, but surprisingly close ASH data source empowers drilldowns Top Sql Top Waits Details p1,p2,p3 and more

31. 31 3/21/2016 Copyright 2006 Kyle Hailey AAS from ASH 1. ASHRPT Based entirely on v$active_session_history @?/rdbms/admin/ashrpt Exec ASH_REPORT_TEXT/HTML 2. OEM 10g Top Activity page Displays it directly Enables Drilldowns 3. Custom queries on v$active_session_history select * from table (dbms_workload_repository.ash_report_text( (select dbid from v$database), (select dbid from v$database), 1, 1, sysdate – 1/24, sysdate – 1/24, sysdate )) ; sysdate )) ;

32. 32 3/21/2016 Copyright 2006 Kyle Hailey 1. ASHRPT ASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host ------------ ----------- ------------ -------- ----------- --- ------------ TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ---- ------------------ ------------------ ------------------ ------------------ 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: 12.39 Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class % Activity Sessions ----------------------------------- --------------- ---------- ---------- CPU + Wait for CPU CPU 67.98 8.42 enq: TX - row lock contention Application 23.98 2.97 buffer busy waits Concurrency 4.66 0.58 latch: cache buffers chains Concurrency 2.26 0.28 ASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host ------------ ----------- ------------ -------- ----------- --- ------------ TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ---- ------------------ ------------------ ------------------ ------------------ 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: 12.39 Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class % Activity Sessions ----------------------------------- --------------- ---------- ---------- CPU + Wait for CPU CPU 67.98 8.42 enq: TX - row lock contention Application 23.98 2.97 buffer busy waits Concurrency 4.66 0.58 latch: cache buffers chains Concurrency 2.26 0.28

33. 33 3/21/2016 Copyright 2006 Kyle Hailey 1. ASH RPT 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***

34. 34 3/21/2016 1. ASHRPT over Time Waits over Time Not in AAS Difficult but better than nothing Compare to …

35. 35 3/21/2016 2. OEM 10g : Top Activity Top Activity Based on ASH Enables Drilldowns Top SQL Top Session Drill into a session Stats Raw waits Open cursors General info Drill into a SQL Stats and text Users executing Explain plan Tuning options

36. 36 3/21/2016 3. Custom Scripts Hate Graphics ? Query v$active_session_history directly Join to dba_hist_active_sess_history for week of data act.sql Like top 5 timed events Aveact.sql Charts with text AAS by hour (15 minute, minute, etc) Aveactn.sql Ditto, with top 2 wait events per bucket Following Scripts Available on http://perfvision.com/ashscripts.php

37. 37 3/21/2016 Copyright 2006 Kyle Hailey 3. Custom Scripts @act Analysis Begin Time : 2007-07-24 11:04:48 Analysis End Time : 2007-07-24 11:19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess -------------------------------------- ---------- ------------ latch free 10.3.02 log buffer space 13.39.02 buffer busy waits 14.42.03 db file scattered read 15.45.03 library cache pin 78 2.34.15 log file sync 213 6.40.40 ON CPU 726 21.82 1.38 enqueue 855 25.70 1.62 db file sequential read 1399 42.05 2.65 ------------ sum 6.30 @act Analysis Begin Time : 2007-07-24 11:04:48 Analysis End Time : 2007-07-24 11:19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess -------------------------------------- ---------- ------------ latch free 10.3.02 log buffer space 13.39.02 buffer busy waits 14.42.03 db file scattered read 15.45.03 library cache pin 78 2.34.15 log file sync 213 6.40.40 ON CPU 726 21.82 1.38 enqueue 855 25.70 1.62 db file sequential read 1399 42.05 2.65 ------------ sum 6.30

38. 38 3/21/2016 Copyright 2006 Kyle Hailey 3. Custom Scripts @aveact TM NPTS AVEACT GRAPH CPU WAITS ---------------- ------ ------- ---------------------- ---- ----- 06-AUG 13:00:00 270.33 +- 2 29 59 06-AUG 14:00:00 1040 2.24 ++--------2--- 341 1984 06-AUG 15:00:00 623 6.67 ++++------2---------- 438 3718 06-AUG 16:00:00 1088 2.59 ++--------2---- 335 2486 06-AUG 17:00:00 1104 1.26 ++----- 2 349 1043 06-AUG 18:00:00 1093 1.38 +++---- 2 663 842 06-AUG 19:00:00 1012 1.74 ++------- 2 373 1388 06-AUG 20:00:00 1131.99 +---- 2 304 820 06-AUG 21:00:00 1111 1.22 ++----- 2 344 1012 06-AUG 22:00:00 1010 1.66 ++------ 2 414 1259 06-AUG 23:00:00 1120 1.08 +---- 2 298 913 07-AUG 00:00:00 1024.83 +--- 2 273 576 07-AUG 01:00:00 1006 1.74 ++------- 2 319 1428 07-AUG 02:00:00 1090 2.47 ++--------2---- 347 2345 07-AUG 03:00:00 687 6.59 +++-------2---------- 382 4142 07-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 659 07-AUG 05:00:00 1104 3.08 +++++-----2------ 1170 2226 07-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 558 @aveact TM NPTS AVEACT GRAPH CPU WAITS ---------------- ------ ------- ---------------------- ---- ----- 06-AUG 13:00:00 270.33 +- 2 29 59 06-AUG 14:00:00 1040 2.24 ++--------2--- 341 1984 06-AUG 15:00:00 623 6.67 ++++------2---------- 438 3718 06-AUG 16:00:00 1088 2.59 ++--------2---- 335 2486 06-AUG 17:00:00 1104 1.26 ++----- 2 349 1043 06-AUG 18:00:00 1093 1.38 +++---- 2 663 842 06-AUG 19:00:00 1012 1.74 ++------- 2 373 1388 06-AUG 20:00:00 1131.99 +---- 2 304 820 06-AUG 21:00:00 1111 1.22 ++----- 2 344 1012 06-AUG 22:00:00 1010 1.66 ++------ 2 414 1259 06-AUG 23:00:00 1120 1.08 +---- 2 298 913 07-AUG 00:00:00 1024.83 +--- 2 273 576 07-AUG 01:00:00 1006 1.74 ++------- 2 319 1428 07-AUG 02:00:00 1090 2.47 ++--------2---- 347 2345 07-AUG 03:00:00 687 6.59 +++-------2---------- 382 4142 07-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 659 07-AUG 05:00:00 1104 3.08 +++++-----2------ 1170 2226 07-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 558

39. 39 3/21/2016 Copyright 2006 Kyle Hailey 3. Custom Scripts @aveactn TO_CHAR(STA AAS AAS1 FIRST AAS2 SECOND GRAPH ----------- ------- ----- --------------- ----- --------------- ------------------------ 06 13:00:00.30.11 db file sequent.10 CPU - 2 06 14:00:00 2.24.87 direct path wri.49 log file sync ++--------2--- 06 15:00:00 6.67 2.07 direct path wri 1.93 log file sync ++++------2---------- 06 16:00:00 2.59 1.01 direct path wri.57 log file sync ++--------2---- 06 17:00:00 1.26.63 log file sync.32 CPU ++----- 2 06 18:00:00 1.38.61 CPU.45 log file sync +++---- 2 06 19:00:00 1.86.97 log file sync.39 CPU ++------- 2 06 20:00:00.99.38 log file sync.27 CPU +---- 2 06 21:00:00 1.22.57 log file sync.31 CPU ++----- 2 06 22:00:00 1.66.81 log file sync.41 CPU ++------ 2 06 23:00:00 1.08.49 log file sync.27 CPU +---- 2 07 00:00:00.83.37 log file sync.27 CPU +--- 2 07 01:00:00 1.74.96 log file sync.31 CPU ++------- 2 07 02:00:00 2.48 1.21 direct path wri.52 log file sync ++--------2---- 07 03:00:00 6.59 2.63 direct path wri 1.58 log file sync +++-------2---------- 07 04:00:00 1.95 1.29 CPU.27 log file sync ++++++--- 2 07 05:00:00 3.08 1.05 CPU.83 enqueue +++++-----2------ @aveactn TO_CHAR(STA AAS AAS1 FIRST AAS2 SECOND GRAPH ----------- ------- ----- --------------- ----- --------------- ------------------------ 06 13:00:00.30.11 db file sequent.10 CPU - 2 06 14:00:00 2.24.87 direct path wri.49 log file sync ++--------2--- 06 15:00:00 6.67 2.07 direct path wri 1.93 log file sync ++++------2---------- 06 16:00:00 2.59 1.01 direct path wri.57 log file sync ++--------2---- 06 17:00:00 1.26.63 log file sync.32 CPU ++----- 2 06 18:00:00 1.38.61 CPU.45 log file sync +++---- 2 06 19:00:00 1.86.97 log file sync.39 CPU ++------- 2 06 20:00:00.99.38 log file sync.27 CPU +---- 2 06 21:00:00 1.22.57 log file sync.31 CPU ++----- 2 06 22:00:00 1.66.81 log file sync.41 CPU ++------ 2 06 23:00:00 1.08.49 log file sync.27 CPU +---- 2 07 00:00:00.83.37 log file sync.27 CPU +--- 2 07 01:00:00 1.74.96 log file sync.31 CPU ++------- 2 07 02:00:00 2.48 1.21 direct path wri.52 log file sync ++--------2---- 07 03:00:00 6.59 2.63 direct path wri 1.58 log file sync +++-------2---------- 07 04:00:00 1.95 1.29 CPU.27 log file sync ++++++--- 2 07 05:00:00 3.08 1.05 CPU.83 enqueue +++++-----2------

40. 40 3/21/2016 Copyright 2006 Kyle Hailey In Review: Two Sources 1. v$system_event & v$sysstat Indirect Accurate Lags (especially CPU) Limits analysis 2. v$active_session_history Direct Real time Approximation ***Allows drilldowns***

41. 41 3/21/2016 Copyright 2006 Kyle Hailey Samples VS CountersCounters Samples Slight Lags v$system_event v$active_session_history

42. 42 3/21/2016 Copyright 2006 Kyle Hailey CPU in ASH vs Stats

43. 43 3/21/2016 Solutions ? AAS is good at finding problems. How do you find solutions? ADDM ? Includes machine info - not that complicated Paging & CPU Need license Some solutions missing – where do you look? ASH data (source for ASH ) Simulatable for free Time Slicing – no need for snap shot ids, down to the second Multi-dimensional SQL Session, Package, Procedure, Module, Action, Service, Client Object, File, Block Wait Details – P1, P2, P3

44. Copyright 2006 Kyle Hailey v$active_session_history 10.1 SESSION_ID NUMBER SESSION_SERIAL# NUMBER USER_ID NUMBER SERVICE_HASH NUMBER SESSION_TYPE VARCHAR2(10) PROGRAM VARCHAR2(64) MODULE VARCHAR2(48) ACTION VARCHAR2(32) CLIENT_ID VARCHAR2(64) EVENT VARCHAR2(64) EVENT_ID NUMBER EVENT# NUMBER SEQ# NUMBER P1 NUMBER P2 NUMBER P3 NUMBER WAIT_TIME NUMBER TIME_WAITED NUMBER CURRENT_OBJ# NUMBER CURRENT_FILE# NUMBER CURRENT_BLOCK# NUMBER0 SQL_ID VARCHAR2(13) SQL_CHILD_NUMBER NUMBER SQL_PLAN_HASH_VALUE NUMBER SQL_OPCODE NUMBER QC_SESSION_ID NUMBER QC_INSTANCE_ID NUMBER SAMPLE_ID NUMBER SAMPLE_TIME TIMESTAMP(3) When Session SQL Wait SESSION_STATE VARCHAR2(7) WAIT_TIME NUMBER State TIME_WAITED NUMBER Duration

45. 45 3/21/2016 Simulating ASH Performance Package License? $3000/pack/CPU ? Not on 10g ? Do it yourself V$SESSION 10g most of ASH Before 10g, join v$session and v$session_wait Sample every second Store in a table Free PL/SQL package available on http://perfvision.com/ash.php 9i Centralize repository (need partitioning) v$active_session_history v$session SESSION_ID SID SESSION_SERIAL# SERIAL# USER_ID USER# SQL_OPCODE COMMAND PROGRAM SESSION_TYPE TYPE SQL_ID SQL_CHILD_NUMBER MODULE ACTION CURRENT_OBJ# ROW_WAIT_OBJ# CURRENT_FILE# ROW_WAIT_FILE# CURRENT_BLOCK# ROW_WAIT_BLOCK# CLIENT_ID CLIENT_IDENTIFIER BLOCKING_SESSION BLOCKING_SESSION_STATUS SEQ# EVENT# EVENT P1TEXT P1 P2TEXT P2 P3TEXT P3 WAIT_CLASS_ID WAIT_CLASS WAIT_TIME

46. 46 3/21/2016 Simulating – Key Ideas session_state = decode(WAIT_TIME, 0,'WAITING', 'ON CPU') v$session where ON CPU : status='ACTIVE' and wait_time > 0 or Waiting : wait_class != 'Idle‘ select sysdate sample_time, decode(s.WAIT_TIME, 0,'WAITING','ON CPU') "SESSION_STATE“ /* plus other fields */ from v$session where sid != ( select distinct sid from v$mystat where rownum < 2 ) and ( ( s.status='ACTIVE' and wait_time > 0 ) or s.wait_class!='Idle' ) ; 9i – join v$session and v$session_wait

47. Copyright 2006 Kyle Hailey S-ASH – All DBs @actallsum HOST ASL GRAPH --------------- ---------- ----------------------- # of cpus ----1----2----3----4--- bsn08 1.83 +++------2- control1 1.82 ++-------2- cont01 1.28 +----- 2 bsn02.68 ++-- 4 tsukuba10.53 -- 2 limerock.13 2 bsn03.06 4 devnode.02 1 @actallsum HOST ASL GRAPH --------------- ---------- ----------------------- # of cpus ----1----2----3----4--- bsn08 1.83 +++------2- control1 1.82 ++-------2- cont01 1.28 +----- 2 bsn02.68 ++-- 4 tsukuba10.53 -- 2 limerock.13 2 bsn03.06 4 devnode.02 1

48. 48 3/21/2016 ASH extra fields v$session missing V$session extras (added in 11g ASH) FORCE_MATCHING_SIGNATURE v$sql.sql_id = sql_id QC_SESSION_ID QCSID v$px_session.sid=sid, serial# QC_INSTANCE_ID QCINST_ID v$px_session.sid=sid, serial# SQL_PLAN_HASH_VALUE PLAN_HASH_VALUE v$sqlstats.sql_id v$sqlstats.sql_id = sql_id (PLAN_HASH_VALUE = x$ksuse.KSUSESPH) XID v$transaction.ADDR = taddr FIXED_TABLE_SEQUENCE - find sql executions ROW_WAIT_ROW# Not in v$sqlstats

49. 10.2 ASH Extras BLOCKING_SESSION BLOCKING_SESSION_STATUS BLOCKING_SESSION_SERIAL# P1TEXT P2TEXT P3TEXT WAIT_CLASS WAIT_CLASS_ID XID 49 3/21/2016

50. 10.2.0.3 ASH Extras FORCE_MATCHING_SIGNATURE PLSQL_ENTRY_OBJECT_ID PLSQL_ENTRY_SUBPROGRAM_ID PLSQL_OBJECT_ID PLSQL_SUBPROGRAM_ID 50 3/21/2016 ALL_PROCEDURES where object_id = plsql_object_id where object_id = plsql_object_id and subprogram_id = plsql_subprogram_id

51. Copyright 2006 Kyle Hailey 11g ASH extras SQL Elapsed SQL_EXEC_ID SQL_EXEC_START IN_CONNECTION_MGMT IN_PARSE IN_HARD_PARSE IN_SQL_EXECUTION IN_PLSQL_EXECUTION IN_PLSQL_RPC IN_PLSQL_COMPILATION IN_JAVA_EXECUTION IN_BIND IN_CURSOR_CLOSE SQL_PLAN_LINE_ID SQL_PLAN_OPERATION SQL_PLAN_OPTIONS CURRENT_ROW# EVENT# QC_SESSION_SERIAL# CONSUMER_GROUP_ID FLAGS ETC CPU Analysis – non Timed TOP_LEVEL_SQL_ID TOP_LEVEL_SQL_OPCODE Track Recursive SQL SQL Row Source REMOTE_INSTANCE# RAC – remote transfers

52. 52 3/21/2016 Wait Drill Downs latch: Cache Buffers Chains Find Top SQL Buffer Busy Wait Find Block Type, Object, Object Type and SQL IO Find SQL, file, block, object enq: HW – contention lock mode, sql enq: TX - row lock contention Get lock mode from P1 Get waiter and SQL, block, file, object and blocker Get transaction from log miner with XID from ASH IO db file sequential read db file scattered read db file parallel read read by other session direct path read direct path write direct path write temp Latches 9i need to translate p2 into latch name (still need to in 10g for some ) Locks 9i need to translate p1 into lock name

53. Cache Buffers Chains : Diag Hash Buckets s5 s4 s3 s2 s1 Sessions Contention if too many accesses on a bucket latches Block Headers Cache Buffer Chain

54. 54 3/21/2016 Latch: Cache Buffers Chains Select ash.sql_id, count(*), sql_text from v$active_session_history ash, v$sqlstats sql where event='latch: cache buffers chains' and sql.sql_id(+)=ash.sql_id group by ash.sql_id, sql_text; SQL_ID COUNT(*) SQL_TEXT ------------- -------- -------------------- 3vjxpmhhzngu4 52 SELECT sysdate FROM dual 3uz0s4srzcywr 21 SELECT ID FROM EMP_HASH WHERE EMPNO=6987

55. Buffer Busy Waits S1 S2 S3 S4 Index

56. Solutions depend on block type might get from statspack Object Not in statspack In AWR SQL Not in statspack Not in AWR 56 3/21/2016 Buffer Busy Waits Block Types:  Undo Header  use AUM (or add more RBS)  Undo Block  hot spot in UNDO, application issue  Data  index – hot spot, partition  table – free lists, ASSM, partition  Segment header  table data block -> freelists  Freelist blocks –  free lists groups  File Header Block  extent allocation too small / too many

57. 57 3/21/2016 Buffer Busy Waits P1 P2 OBJN OTYPE SQL_ID BLOCK_TYPE -- ----- --------------- ----- ------------- ----------- 1 92223 BBW_INDEX_VAL_I INDEX 635xhydd6fzgg data block 1 92223 BBW_INDEX_VAL_I INDEX 6avm49ys4k7t6 data block 1 92223 BBW_INDEX_VAL_I INDEX 1hsb81ypyrfs5 data block select ash.p1 p1, ash.p2 p2, nvl(o.object_name,ash.current_obj#) objn, o.object_type otype, ash.SQL_ID, nvl(w.class,to_char(ash.p3)) block_type from v$active_session_history ash, ( select rownum class#, class from v$waitstat ) w, all_objects o where event='buffer busy waits' and w.class#(+)=ash.p3 and o.object_id (+)= ash.CURRENT_OBJ#;

58. 58 3/21/2016 IO EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID ------------------------- --- ------------------ ----------------- ----- ------ ------------ db file sequential read 1 ORDER_ITEMS_UK INDEX PARTITION 7 855 0yas01u2p9ch4 db file scattered read 12 ORDERS TABLE PARTITION 6 10319 05s4vdwsf5802 db file scattered read 13 ORDERS TABLE PARTITION 6 12593 05s4vdwsf5802 db file sequential read 1 ITEM_PRODUCT_IX INDEX 7 55093 0yas01u2p9ch4 db file sequential read 1 CUSTOMERS TABLE PARTITION 6 22852 75621g9y3xmvd db file sequential read 1 CUST_EMAIL_IX INDEX 7 36597 0bzhqhhj9mpaa db file scattered read 13 ORDERS TABLE PARTITION 6 20016 05s4vdwsf5802 select event, ash.p3, o.object_name objn, o.object_type otype, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, ash.SQL_ID from v$active_session_history ash, all_objects o where event like 'db file s%' and o.object_id (+)= ash.CURRENT_OBJ# order by sample_time ; CNT AAS OBJN OTYPE -------- ---------- ------------------------- --------------- 143.01 52938 CUSTOMERS TABLE PARTITION 361.02 53043 ORDER_PK INDEX 506.02 -1 565.03 53042 ORDER_ITEMS_PK INDEX

59. Copyright 2006 Kyle Hailey HWData Empty High Water Mark Header Table Session 1 Session 2 Session 3

60. HW Locks 60 3/21/2016 EVENT SQL_ID OBJ FILE# BLOCK# -------------------- ------------- ----------- ----- ------ enq: HW - contention 49ch3jfkncnsp 53410 T1_I1 13 29734 select event, event, sql_id, sql_id, CURRENT_OBJ# || ' ' || name obj CURRENT_OBJ# || ' ' || name obj,CURRENT_FILE# file#,CURRENT_FILE# file#,CURRENT_BLOCK# block#,CURRENT_BLOCK# block# from v$active_session_history ash, from v$active_session_history ash, obj$ o obj$ o where where event like 'enq: HW%' event like 'enq: HW%' and o.obj# (+)= ash.current_obj# and o.obj# (+)= ash.current_obj# order by sample_time ; order by sample_time ;

61. 61 3/21/2016 Enq: TX – row lock contention SID LM P2 P3 OBJN OTYPE FN BLOCKN SQL_ID BSID XID --- --- ------ ---- ------ ----- -- ------ ------------- ---- ----------------- 152 6 589826 6509 TITI TABLE 1 85018 777z5h2z2qqw6 138 159 4 589864 6510 I1 INDEX 0 0 azav296xxqcjx 138 04001F00F3180000 140 4 524308 6404 -1 0 0 ahm7c9rupbz9r 138 0600070046190000 138 4 327725 6415 -1 0 0 bjvx94vnxtxgv 140 0A002300ED180000 P2=usn<<16 | slot P3=sequence XID logmnr row lock fk pk Bitmap index

62. 62 3/21/2016 PLSQL Tracking Package/Procedure/SQL tracking 10.2.0.3 calling_code SQL_ID COUNT(*) ----------------------------------------------- ---------- ORDERENTRY.NEWORDER 0uuqgjq7k12nf 258 ORDERENTRY.NEWCUSTOMER 0bzhqhhj9mpaa 262 ORDERENTRY.BROWSEANDUPDATEORDERS 41zu158rqf4kf 301 ORDERENTRY.NEWORDER 0yas01u2p9ch4 569 PARSE_SAME 3vjxpmhhzngu4 874 ORDERENTRY.BROWSEANDUPDATEORDERS 05s4vdwsf5802 1669 select object_name package||.|| procedure_name, sql_id, count(*) from v$active_session_history ash, all_procedures procs where ash.PLSQL_ENTRY_OBJECT_ID = procs.object_id and ash.PLSQL_ENTRY_SUBPROGRAM_ID = procs.SUBPROGRAM_ID

63. 63 3/21/2016 Top SQL Aggregate by SQL id Plan hash Force_matching_signature SQL_ID PLAN_HASH_ TYPE CPU WAIT IO TOTAL ------------- ---------- ---------- ----- ----- ----- ------ 3vjxpmhhzngu4 272002086 SELECT 457 52 1 510 3uz0s4srzcywr 1556932380 SELECT 220 21 0 241 967nxsran3qmq 0 PL/SQL EXE 158 14 3 175 7u3hm6xnq4k1n 0 PL/SQL EXE 85 0 0 85 5wqps1quuxqr4 3333492716 UPDATE 14 51 3 68 635xhydd6fzgg 3333492716 UPDATE 14 50 4 68 1hsb81ypyrfs5 3333492716 UPDATE 13 50 5 68 6avm49ys4k7t6 3333492716 UPDATE 13 32 5 50 2nhzyn560xyhs 735420252 SELECT 10 1 0 11

64. Time Slicing Previous examples were unbounded Need Time slice bounds Where sample_time > &begin and sample_time < &end Detail down to the seconds 64 3/21/2016 TM NPTS AAS GRAPH CPU WAITS ---------------- ------ ------- ------------------------------ ----- ----- 02-MAR 12:28:00 1.00 2 0 0 02-MAR 12:28:30 1.07 2 1 02-MAR 12:29:00 1.07 2 1 02-MAR 12:29:15 4 1.07 +++++ 2 16 02-MAR 12:29:30 15 7.93 ++++++++++2++++++++++ 112 02-MAR 12:29:45 15 8.07 ++++++++++2++++++++++ 109 12 02-MAR 12:30:00 15 8.00 ++++++++++2++++++++++ 113 02-MAR 12:30:15 15 8.13 ++++++++++2++++++++++ 120 2 02-MAR 12:30:30 14 5.47 ++++++++++2++++++++++ 81 02-MAR 12:30:45 13 3.73 ++++++++++2+++++++++ 53 02-MAR 12:32:00 1.07 2 1 02-MAR 12:33:15 1.07 2 1 02-MAR 12:34:45 1.07 2 1

65. Explain Plans DBMS_XPLAN.DISPLAY_CURSOR( sql_id ); 65 3/21/2016 set linesize 150 set pagesize 100 col PLAN_TABLE_OUTPUT for a100 select * from TABLE(DBMS_XPLAN.DISPLAY_CURSOR('&1')); SQL_ID 777z5h2z2qqw6, child number 0 ------------------------------------- delete from titi where dummy='X' Plan hash value: 3513264481 --------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | --------------------------------------------------------------------------- | 0 | DELETE STATEMENT | | | | 2 (100)| | | 1 | DELETE | TITI | | | | | |* 2 | TABLE ACCESS FULL| TITI | 1 | 2 | 2 (0)| 00:00:01 | --------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - filter("DUMMY"='X') Note ----- - dynamic sampling used for this statement 9i: SELECT * FROM TABLE( dbms_xplan.display('SQLPLANS','&sql_id'));

66. Graphics How do you keep track of Scripts for Types of wait events Aggregations by different dimensions Changing Time Slices 66 3/21/2016 Graphics

67. OEM / ASHMON Aggregates Drilldowns Time Slicing Guidance 67 3/21/2016

68. Copyright 2006 Kyle Hailey Test 3 Elapsed 1:23 14:44 14:59 Elapsed Time 1:23 10x Improvement ! Still more room for Improvement – a lot Of CPU left

69. Copyright 2006 Kyle Hailey ASHMON “OEM-lite” Written in TCL/TK Only Overview SQL Text

70. DEMO DEMO ASHMON? 70 3/21/2016

71. Parsability of Graphics Infocus – (overhead projectors) sited a study that humans can parse graphical information 400,000 times faster than textual data

72. What are the Trends?

73. Trends: With Graphics I II III IV

74. Trends: Waits Over Time date17:1317:1317:1417:1417:1517:1517:1617:1617:1717:17 17:1817:1817:1917:1917:2017:2017:2117:2117:22 17:2217:2317:2317:2417:2417:2517:2517:2617:26 17:2717:2717:2817:2817:2917:2917:3017:3017:31 17:3117:3217:3217:3317:3317:3417:3417:3517:35 17:3617:3617:3717:3717:3817:3817:3917:3917:40 17:4017:4117:4117:42 log_file_switch_completion_3000029829360000 000000000 000000000 000000000 00023764200000 000000021004190 000 log_buffer_space_3000000000 000000000 000000000 000000000 000000000 000000064782210 14945538521983600 QL*Net_break/reset_to_client_38501199113855397044359823469061832372333297777665961181986495648 75184891145532677028651108576276343001058283313260400854442213172524 850204112010958920853393841181020895773996766747457623161113441429 734461587618846642950725631570736238442915706039974856626601072 634901899294404902230607935957510219455790908786693412097664906 164991621023453510588829863801489187954312962805695749444087195059 333256531543485530740496531570113415248675684611667403785817539 901223874646569055304878131391159549858761 latch_free_3430649519313273327222218102317640233238736163787719223991786672 231250723083793109183518626666949805645308078131327673841164596 22496544403422076414406221424089592886800217772233326271237003 20982215986478828048257018321049432686288225944023883412044871 14622082306024163562421324617831022677123474363612926061499421 24395442753717190540730179331651176368629410882099224022319624 26455391260079431699713112422087917 rdbms_ipc_reply_3279201393188872229909917279033312793973629891410299011862995375627895419 298798282808622129906822279030672991587329896307318789182827193229883910 29885870299061132793320927794807298799972884204030138759121029540 000000000 0468156200101822981000 38300000000 0003079070006040 0422400005082702 log_file_sync_34897094598481887738596732708252869921640100466708893499376102384 56484181623855792025624214425129115558677281198244355491382978201 436191429246743969540460885144990435563593748045787361144087525 791662395317114204114400246542302774795701632444047404695123660 289104489313724254477617721851198364701245909418167510406969569 8678671106276098452116135051529052070919976269395913915373218974176 713059382500538867810953927165enqueue_339067875985842993754 38807270867646237615812519939092426739097418261878157 121925793008413930154100304260423003456430058775300621142706223927585615 30040241237417591390087511246778782463079493934850847 8396491959572765189382140371120439339251335880052 1219841483091683718737093333073894476811207243034138 40273882095127314609564924922979781508468089284755 248836 db_file_sequential_read_31070390511095665871270089429703361574615747707286541092316829375567686966703 1176380467606578888137834667594888970957080277614507626225820793861815450 656105805159940274260828781389654986700521069074176801591146049512467891 337860923321189640736046492123204116208445786601241222015565110034316957 117730086245523148897870672481196407746288899277896221753448352657402401 353197353061956261707668345143403950471437681062341653682495229630172023 255101444026033183215876784503525696381333846491288692345296497438161443 43878778310952952601874758239638197939177914975 itch_(checkpoint_incomplete)_30000285950959247915390000 000000000 000000000 000000000 000000000 00000000

75. Trends: Waits over Time

76. 76 3/21/2016 Counties in US 3101 Counties in US 50 pages

77. Copyright 2006 Kyle Hailey Launch: Pressure Midnight before January 28, 1986 Lives are on the line Thanks to Edward Tufte

78. Copyright 2006 Kyle Hailey 13 Pages Faxed

79. Copyright 2006 Kyle Hailey Correlated with Temperatures

80. Copyright 2006 Kyle Hailey Trying to Show data an Analysis

81. Copyright 2006 Kyle Hailey Congressional Hearings Evidence

82. Copyright 2006 Kyle Hailey Only Damage Engineering Document only Showed Damage

83. Copyright 2006 Kyle Hailey Clearer 1.Include successes 2.Remove Irrelevant 3.Normalize same temp 4.Scale known vs unknown ? ? ?

84. Copyright 2006 Kyle Hailey Do You Want? Engineering Data?

85. Copyright 2006 Kyle Hailey Do You Want? Pretty Pictures

86. Copyright 2006 Kyle Hailey Do You Want? Clean and Clear

87. Copyright 2006 Kyle Hailey Difficult NASA Engineers Fail Congressional Investigators Fail Data is Difficult But … Lack of Clarity can be devastating

88. Copyright 2006 Kyle Hailey Imagine Trying to Drive your Car And is updated once and hour Or would you like it to look … if you Dashboard looked like:

89. Copyright 2006 Kyle Hailey Or This And is updated once and hour

90. Conclusios AAS Identify Monitor, Alert ASH Drilldown and Solutions Top SQL/Session Module/Action Client/Service Package/procedure SQL, Wait, p1/2/3, Object, file, block, XID, Graphics Ease of Use Data Compression Parsability Guidance 90 3/21/2016 References Download at http://perfvision.com/ftp/hotsos/aas.ppt ASH Simulation http://perfvision.com/ftp/ash.php Graphical “OEM-lite” http://perfvision.com/ftp/ashmon.php ASH Scripts http://perfvision.com/ashscripts.php