1. ASH – Active Session History Feel the Power Kyle Hailey http://perfvision.com

2. #.2 Copyright 2006 Kyle Hailey ASH Whole New Paradigm in Technology The Power of ASH lies in Simplifying Performance Tuning  Totally new and exciting methodology  Cheaper, Quicker, Richer and better tasting too

3. #.3 Copyright 2006 Kyle Hailey Why should you care? Because ASH can Change your life …  10g immediately Accessible  Geeks: Via scripts in SQL  Managers : Graphical OEM  Stuck on Oracle 7,8,9 ? … my apologies because 10g rocks, but … no worries, the data is accessible for you too via scripts http://perfvision.com/sash.php

4. #.4 Copyright 2006 Kyle Hailey Here’s a piece of Personal history

5. #.5 Copyright 2006 Kyle Hailey This is the story of how ASH changed my Life Once upon a time I was stuck in dull tech support … Finally I was promoted to head DBA, thinking my life was getting better…

6. #.6 Copyright 2006 Kyle Hailey But … It was then that I learned I needed to know what the databases were doing ALL the time Because lives and more importantly my job depended on it

7. #.7 Copyright 2006 Kyle Hailey What could I do?! Clearly a job for Stats Pack Because stats pack showed me every hour  All the Stats!  All the top Bottlenecks  Almost all the top SQL

8. #.8 Copyright 2006 Kyle Hailey So why wasn’t I feeling better? Hmmm … but … wait  What if my database hits a bottleneck 15 minutes after my last stats pack?!  What about that rouge coder who writes bad SQL … can I find his module gone mad?  Ok I found the top SQL but … now what?  What do I do with the top WAIT?

9. #.9 Copyright 2006 Kyle Hailey If lives (and my job) depended on my databases  Was once an hour really enough ?  Could Stats Pack find that rogue coder?  Could Stats Pack tell me why an SQL bottlenecked?  Could Stats Pack tell me the root of a wait bottleneck?

10. #.10 Copyright 2006 Kyle Hailey Don’t tell anyone but … I couldn’t even decipher Stats Pack half the time Often, I didn’t understand statspack’s 1000+ lines

11. #.11 Copyright 2006 Kyle Hailey Oh no … it’s 3am my manager calls “why is the database hanging?!”  Who did it?  Where did the SQL get blocked?  What if the bottlenecked started 5 minutes after my last Stats Pack?

12. #.12 Copyright 2006 Kyle Hailey In that moment of crises Stats Pack couldn’t tell me … but ASH can

13. #.13 Copyright 2006 Kyle Hailey The crises of my day have left the “ASHes” of today. I didn’t have ASH but You do

14. #.14 Copyright 2006 Kyle Hailey It’s a Revolution and it’s an Evolution  Oracle 6 … ie the dark ages … there was once the Cache Buffer Hit Ratio  Oracle 7 … turned the lights on … Wait Events … hallelujah I can see the light  Oracle 10g … ASH has landed (like the wheels on the suitcases)

15. #.15 Copyright 2006 Kyle Hailey ASH uses Sampling New Paradigm  Clear Vision of the future  Use new sampling technology  Lets go of the need to know 100%  Statistical approximation  Cheaper  Richer Multidimensional Data  Identifies  SQL  Sessions  Objects  Machine  User

16. #.16 Copyright 2006 Kyle Hailey To identify Players before ASH  Sessions  # sessions x (# wait events + statistics)  Example (150 x (800+200) = 150,000 )  SQL  v$sql  Could be 10000s  Takes out latches that compete with other sql executions  Objects  V$segstat 9i+  Could be 1000s of objects Expensive !

17. #.17 Copyright 2006 Kyle Hailey ASH – Intelligence for the new Millennium  Intelligently Collects Data  It self adjusts for your needs  More activity, more data collected  Less activity, less data collected  Those old methods collected everything  Obfuscated the problem, too many statistics too late  Costly  Too Granular – once an hour ?! Give me a break

18. ASH Samples Session State TIME ? ? ??? Sessions change a lot quicker but can get the main picture via sampling by sampling faster Every second 10:00:00 10:00:0110:00:02 10:00:03 10:00:04 10:00:05

19. #.19 Copyright 2006 Kyle Hailey Sampling is like taking Pictures

20. #.20 Copyright 2006 Kyle Hailey If happens a lot or for long … we’ll catch it, guaranteed

21. #.21 Copyright 2006 Kyle Hailey Session States IO CPU Idle Wait Work LatencyContention

22. #.22 Copyright 2006 Kyle Hailey IDLE  Example : SQL*Net Message from Client  10g  9i or below – just have to keep track  Statspack STATS$IDLE_EVENT select name from v$event_name where wait_class='Idle'; 58 Rows

23. #.23 Copyright 2006 Kyle Hailey CPU  ASH: SESSION_STATE = “ON CPU”  ASH: wait_time > 0 10g: v$session 9i: v$session_wait w.wait_time != 0 /* on CPU */ and s.status='ACTIVE' /* ACTIVE */ Identifying CPU via v$session_wait is a huge break through

24. #.24 Copyright 2006 Kyle Hailey IO  ASH: SESSION_STATE=‘WAITING’ &  10g: WAIT_CLASS=‘User I/O’  9i:  Look at 10g for ideas  db file sequential read  db file scattered read  db file parallel read  direct path read

25. #.25 Copyright 2006 Kyle Hailey WAITING  ASH: SESSION_STATE=‘WAITING’  ASH: WAIT_TIME <= 0  WAIT_CLASS 1. Administrative 2. Application 3. Cluster 4. Commit 5. Concurrency 6. Configuration 7. Network 8. Other 9. Scheduler 10. System I/O  800+ WAIT

26. Session 1 Session 2 Session 3 Session 4 TIME Samples for all users 10:15:00 10:15:01 10:15:02 10:15:03 10:15:04 10:15:05 10:15:06 10:15:07

27. #.27 Copyright 2006 Kyle Hailey ASH Fields SQL> v$active_session_history Name Null? Type ----------------------------------------- -------- ---------------------------- SAMPLE_ID NUMBER SAMPLE_TIME TIMESTAMP(3) SESSION_ID NUMBER SESSION_SERIAL# NUMBER USER_ID NUMBER SQL_ID VARCHAR2(13) SQL_CHILD_NUMBER NUMBER SQL_PLAN_HASH_VALUE NUMBER SQL_OPCODE NUMBER SERVICE_HASH NUMBER SESSION_TYPE VARCHAR2(10) SESSION_STATE VARCHAR2(7) QC_SESSION_ID NUMBER QC_INSTANCE_ID NUMBER EVENT VARCHAR2(64) EVENT_ID NUMBER EVENT# NUMBER SEQ# NUMBER P1 NUMBER P2 NUMBER P3 NUMBER WAIT_TIME NUMBER TIME_WAITED NUMBER CURRENT_OBJ# NUMBER CURRENT_FILE# NUMBER CURRENT_BLOCK# NUMBER PROGRAM VARCHAR2(48) MODULE VARCHAR2(48) ACTION VARCHAR2(32) CLIENT_ID VARCHAR2(64

28. #.28 Copyright 2006 Kyle Hailey v$active_session_history SESSION_ID NUMBER SESSION_SERIAL# NUMBER USER_ID NUMBER SERVICE_HASH NUMBER SESSION_TYPE VARCHAR2(10) PROGRAM VARCHAR2(64) MODULE VARCHAR2(48) ACTION VARCHAR2(32) CLIENT_ID VARCHAR2(64) EVENT VARCHAR2(64) EVENT_ID NUMBER EVENT# NUMBER SEQ# NUMBER P1 NUMBER P2 NUMBER P3 NUMBER WAIT_TIME NUMBER TIME_WAITED NUMBER CURRENT_OBJ# NUMBER CURRENT_FILE# NUMBER CURRENT_BLOCK# NUMBER0 SQL_ID VARCHAR2(13) SQL_CHILD_NUMBER NUMBER SQL_PLAN_HASH_VALUE NUMBER SQL_OPCODE NUMBER QC_SESSION_ID NUMBER QC_INSTANCE_ID NUMBER SAMPLE_ID NUMBER SAMPLE_TIME TIMESTAMP(3) When Session SQL Wait SESSION_STATE VARCHAR2(7) WAIT_TIME NUMBER State TIME_WAITED NUMBER Duration

29. #.29 Copyright 2006 Kyle Hailey Primary Fields of ASH SESSION_ID EVENT SQL_ID SAMPLE_TIME When Session SQL Wait SESSION_STATE State TIME_WAITED Duration

30. #.30 Copyright 2006 Kyle Hailey Amazing things YOU can do with ASH Consumers  Top Session  Top User  Top SQL  Top Object  Top Module.Action  Top Program  Top Service  Top Client  Top Wait No machine  X Resources CPU Waits Event (800*) I/O File Block Time (32 columns in ASH)

31. #.31 Copyright 2006 Kyle Hailey Groupings – Top Consumer SESSION_ID SESSION_SERIAL# (signal SID reuse) SESSION_TYPE (FOREGROUND,BACKGROUND) CURRENT_OBJ# CURRENT_FILE# CURRENT_BLOCK# USER_ID (SYS, SYSTEM, SCOTT etc) SERVICE_HASH (OE,GL,HR) PROGRAM (SQL, JDBC, Forms etc) MODULE.ACTION(PLSQL tagging) CLIENT_ID (identifying users in a session pool) SQL_ID QC_SESSION_ID - Query Coordinator QC_INSTANCE_ID – RAC EVENT + P1, P2, P3

32. #.32 Copyright 2006 Kyle Hailey Select session_id, count(*) from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Select session_id, count(*) from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Top CPU Session Who is the rogue session ?

33. #.33 Copyright 2006 Kyle Hailey Results Top CPU Session SESSION_ID COUNT(*) ---------- 265 256 264 15 257 12 271 12 276 1 SESSION_ID COUNT(*) ---------- 265 256 264 15 257 12 271 12 276 1 SESSION_ID COUNT(*) ---------- 265 256 264 15 257 12 271 12 276 1

34. #.34 Copyright 2006 Kyle Hailey CPU with Bars SESSION_ID COUNT(*) % Bar ---------- ---------- ---------- ------------ 257 75 25 |*** | 263 62 21 |** | 256 32 11 |* | 264 9 3 | | 277 3 1 | | 258 1 0 | | 280 1 0 | | SESSION_ID COUNT(*) % Bar ---------- ---------- ---------- ------------ 257 75 25 |*** | 263 62 21 |** | 256 32 11 |* | 264 9 3 | | 277 3 1 | | 258 1 0 | | 280 1 0 | | Bar shows 10% increments

35. #.35 Copyright 2006 Kyle Hailey Select session_id, count(*) from v$active_session_history where session_state=‘WAITING’ and SAMPLE_TIME > SYSDATE - (5/(24*60)) group by session_id order by count(*) desc; Select session_id, count(*) from v$active_session_history where session_state=‘WAITING’ and SAMPLE_TIME > SYSDATE - (5/(24*60)) group by session_id order by count(*) desc; Top Waiting Session  Top Waiting Session in last 5 minutes

36. #.36 Copyright 2006 Kyle Hailey SESSION_ID COUNT(*) ---------- 272 224 254 8 249 5 276 5 277 4 270 1 SESSION_ID COUNT(*) ---------- 272 224 254 8 249 5 276 5 277 4 270 1 Top Waiting Session Results

37. #.37 Copyright 2006 Kyle Hailey Top SQL from ASH select ash.SQL_ID, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "WAIT", sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "IO", sum(decode(ash.session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where SQL_ID is not NULL and en.event#=ash.event# group by sql_id order by sum(decode(session_state,'ON CPU',1,1)) desc select ash.SQL_ID, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "WAIT", sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "IO", sum(decode(ash.session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where SQL_ID is not NULL and en.event#=ash.event# group by sql_id order by sum(decode(session_state,'ON CPU',1,1)) desc

38. #.38 Copyright 2006 Kyle Hailey Top SQL from ASH Results SQL_ID CPU WAITING IO TOTAL ------------- ---------- ---------- ---------- ---------- 4c1xvq9ufwcjc 23386 0 0 23386 6wjw6rz5uvbp3 99 0 23 122 968dm8hr9qd03 97 0 22 119 938jp5gasmrah 90 0 25 115 cv8xnv81kf582 42 0 9 51 6p9bzu19v965k 21 0 0 21 5zu8pxnun66bu 15 0 0 15 db2jr13nup72v 9 0 0 9 7ks5gnj38hghv 8 0 0 8 SQL_ID CPU WAITING IO TOTAL ------------- ---------- ---------- ---------- ---------- 4c1xvq9ufwcjc 23386 0 0 23386 6wjw6rz5uvbp3 99 0 23 122 968dm8hr9qd03 97 0 22 119 938jp5gasmrah 90 0 25 115 cv8xnv81kf582 42 0 9 51 6p9bzu19v965k 21 0 0 21 5zu8pxnun66bu 15 0 0 15 db2jr13nup72v 9 0 0 9 7ks5gnj38hghv 8 0 0 8

39. #.39 Copyright 2006 Kyle Hailey Top Session select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING", sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO", sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1)) select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING", sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO", sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1)) select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING", sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO", sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1))

40. #.40 Copyright 2006 Kyle Hailey Top Session Results SESSION_ID SERIAL# USER_ID PROGRAM CPU WAITING IO ---------- ------- ---------- ------------------------- ------- ---------- ---------- 247 61970 1 sqlplus 11698 0 0 277 1 0 oracle@labsfrh903 (LGWR) 14 21 0 276 1 0 oracle@labsfrh903 (CKPT) 19 10 0 278 1 0 oracle@labsfrh903 (DBW0) 29 0 0 280 1 0 oracle@labsfrh903 (PMON) 19 0 0 254 22617 5 Executor.exe 13 0 3 255 12877 5 Executor.exe 11 0 5 257 33729 5 Executor.exe 15 0 1 255 13417 5 Executor.exe 14 0 2 SESSION_ID SERIAL# USER_ID PROGRAM CPU WAITING IO ---------- ------- ---------- ------------------------- ------- ---------- ---------- 247 61970 1 sqlplus 11698 0 0 277 1 0 oracle@labsfrh903 (LGWR) 14 21 0 276 1 0 oracle@labsfrh903 (CKPT) 19 10 0 278 1 0 oracle@labsfrh903 (DBW0) 29 0 0 280 1 0 oracle@labsfrh903 (PMON) 19 0 0 254 22617 5 Executor.exe 13 0 3 255 12877 5 Executor.exe 11 0 5 257 33729 5 Executor.exe 15 0 1 255 13417 5 Executor.exe 14 0 2

41. #.41 Copyright 2006 Kyle Hailey Top Session w/ Username select decode(nvl(to_char(s.sid),-1),-1,'DISCONNECTED','CONNECTED') "STATUS", topsession.session_id "SESSION_ID", u.name "NAME", topsession.program "PROGRAM", max(topsession.CPU) "CPU", max(topsession.WAITING) "WAITING", max(topsession.IO) "IO", max(topsession.TOTAL) "TOTAL" from ( {previous query} ) topsession, v$session s, user$ u where u.user# =topsession.user_id and /* outer join to v$session because the session might be disconnected */ topsession.session_id = s.sid (+) and topsession.session_serial# = s.serial# (+) group by topsession.session_id, topsession.session_serial#, topsession.user_id, topsession.program, s.username,s.sid,s.paddr,u.name order by max(topsession.TOTAL) desc select decode(nvl(to_char(s.sid),-1),-1,'DISCONNECTED','CONNECTED') "STATUS", topsession.session_id "SESSION_ID", u.name "NAME", topsession.program "PROGRAM", max(topsession.CPU) "CPU", max(topsession.WAITING) "WAITING", max(topsession.IO) "IO", max(topsession.TOTAL) "TOTAL" from ( {previous query} ) topsession, v$session s, user$ u where u.user# =topsession.user_id and /* outer join to v$session because the session might be disconnected */ topsession.session_id = s.sid (+) and topsession.session_serial# = s.serial# (+) group by topsession.session_id, topsession.session_serial#, topsession.user_id, topsession.program, s.username,s.sid,s.paddr,u.name order by max(topsession.TOTAL) desc

42. #.42 Copyright 2006 Kyle Hailey Top Session Finding a Rogue User STATUS SESSION_ID NAME PROGRAM CPU WAITING IO --------------- ---------- ---------- ------------------------- ----- ---------- ---- CONNECTED 247 CPU_Monger ChMgr304.exe 11704 0 0 CONNECTED 277 SYS oracle@labsfrh903 (LGWR) 14 19 0 CONNECTED 278 SYS oracle@labsfrh903 (DBW0) 29 0 0 CONNECTED 276 SYS oracle@labsfrh903 (CKPT) 18 9 0 CONNECTED 280 SYS oracle@labsfrh903 (PMON) 20 0 0 DISCONNECTED 255 SYSTEM Executor.exe 11 4 5 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 DISCONNECTED 255 SYSTEM Executor.exe 14 0 2 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 STATUS SESSION_ID NAME PROGRAM CPU WAITING IO --------------- ---------- ---------- ------------------------- ----- ---------- ---- CONNECTED 247 CPU_Monger ChMgr304.exe 11704 0 0 CONNECTED 277 SYS oracle@labsfrh903 (LGWR) 14 19 0 CONNECTED 278 SYS oracle@labsfrh903 (DBW0) 29 0 0 CONNECTED 276 SYS oracle@labsfrh903 (CKPT) 18 9 0 CONNECTED 280 SYS oracle@labsfrh903 (PMON) 20 0 0 DISCONNECTED 255 SYSTEM Executor.exe 11 4 5 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 DISCONNECTED 255 SYSTEM Executor.exe 14 0 2 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3

43. #.43 Copyright 2006 Kyle Hailey SASH – DB Detailed ASL @act Analysis Begin Time : 2007-07-24 11:04:48 Analysis End Time : 2007-07-24 11:19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess -------------------------------------- ---------- ------------ latch free 10.3.02 log buffer space 13.39.02 buffer busy waits 14.42.03 db file scattered read 15.45.03 library cache pin 78 2.34.15 log file sync 213 6.4.4 ON CPU 726 21.82 1.38 enqueue 855 25.7 1.62 db file sequential read 1399 42.05 2.65 ------------ sum 6.3 @act Analysis Begin Time : 2007-07-24 11:04:48 Analysis End Time : 2007-07-24 11:19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess -------------------------------------- ---------- ------------ latch free 10.3.02 log buffer space 13.39.02 buffer busy waits 14.42.03 db file scattered read 15.45.03 library cache pin 78 2.34.15 log file sync 213 6.4.4 ON CPU 726 21.82 1.38 enqueue 855 25.7 1.62 db file sequential read 1399 42.05 2.65 ------------ sum 6.3

44. #.44 Copyright 2006 Kyle Hailey SASH – ASL over time @aveact TM NPTS AVEACT GRAPH CPU WAITS ---------------- ------ ------- ---------------------- ---- ----- 06-AUG 13:00:00 270.33 +- 2 29 59 06-AUG 14:00:00 1040 2.24 ++--------2--- 341 1984 06-AUG 15:00:00 623 6.67 ++++------2---------- 438 3718 06-AUG 16:00:00 1088 2.59 ++--------2---- 335 2486 06-AUG 17:00:00 1104 1.26 ++----- 2 349 1043 06-AUG 18:00:00 1093 1.38 +++---- 2 663 842 06-AUG 19:00:00 1012 1.74 ++------- 2 373 1388 06-AUG 20:00:00 1131.99 +---- 2 304 820 06-AUG 21:00:00 1111 1.22 ++----- 2 344 1012 06-AUG 22:00:00 1010 1.66 ++------ 2 414 1259 06-AUG 23:00:00 1120 1.08 +---- 2 298 913 07-AUG 00:00:00 1024.83 +--- 2 273 576 07-AUG 01:00:00 1006 1.74 ++------- 2 319 1428 07-AUG 02:00:00 1090 2.47 ++--------2---- 347 2345 07-AUG 03:00:00 687 6.59 +++-------2---------- 382 4142 07-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 659 07-AUG 05:00:00 1104 3.08 +++++-----2------ 1170 2226 07-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 558 07-AUG 07:00:00 1115 1.06 +++--- 2 559 618 07-AUG 08:00:00 1140.81 ++-- 2 403 519 07-AUG 09:00:00 1128.88 ++--- 2 386 601 @aveact TM NPTS AVEACT GRAPH CPU WAITS ---------------- ------ ------- ---------------------- ---- ----- 06-AUG 13:00:00 270.33 +- 2 29 59 06-AUG 14:00:00 1040 2.24 ++--------2--- 341 1984 06-AUG 15:00:00 623 6.67 ++++------2---------- 438 3718 06-AUG 16:00:00 1088 2.59 ++--------2---- 335 2486 06-AUG 17:00:00 1104 1.26 ++----- 2 349 1043 06-AUG 18:00:00 1093 1.38 +++---- 2 663 842 06-AUG 19:00:00 1012 1.74 ++------- 2 373 1388 06-AUG 20:00:00 1131.99 +---- 2 304 820 06-AUG 21:00:00 1111 1.22 ++----- 2 344 1012 06-AUG 22:00:00 1010 1.66 ++------ 2 414 1259 06-AUG 23:00:00 1120 1.08 +---- 2 298 913 07-AUG 00:00:00 1024.83 +--- 2 273 576 07-AUG 01:00:00 1006 1.74 ++------- 2 319 1428 07-AUG 02:00:00 1090 2.47 ++--------2---- 347 2345 07-AUG 03:00:00 687 6.59 +++-------2---------- 382 4142 07-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 659 07-AUG 05:00:00 1104 3.08 +++++-----2------ 1170 2226 07-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 558 07-AUG 07:00:00 1115 1.06 +++--- 2 559 618 07-AUG 08:00:00 1140.81 ++-- 2 403 519 07-AUG 09:00:00 1128.88 ++--- 2 386 601

45. #.45 Copyright 2006 Kyle Hailey SASH – ASL over time with waits @aveactn TO_CHAR(STA AAS AAS1 FIRST AAS2 SECOND GRAPH ----------- ------- ----- --------------- ----- --------------- ------------------------ 06 13:00:00.30.11 db file sequent.10 CPU - 2 06 14:00:00 2.24.87 direct path wri.49 log file sync ++--------2--- 06 15:00:00 6.67 2.07 direct path wri 1.93 log file sync ++++------2---------- 06 16:00:00 2.59 1.01 direct path wri.57 log file sync ++--------2---- 06 17:00:00 1.26.63 log file sync.32 CPU ++----- 2 06 18:00:00 1.38.61 CPU.45 log file sync +++---- 2 06 19:00:00 1.86.97 log file sync.39 CPU ++------- 2 06 20:00:00.99.38 log file sync.27 CPU +---- 2 06 21:00:00 1.22.57 log file sync.31 CPU ++----- 2 06 22:00:00 1.66.81 log file sync.41 CPU ++------ 2 06 23:00:00 1.08.49 log file sync.27 CPU +---- 2 07 00:00:00.83.37 log file sync.27 CPU +--- 2 07 01:00:00 1.74.96 log file sync.31 CPU ++------- 2 07 02:00:00 2.48 1.21 direct path wri.52 log file sync ++--------2---- 07 03:00:00 6.59 2.63 direct path wri 1.58 log file sync +++-------2---------- 07 04:00:00 1.95 1.29 CPU.27 log file sync ++++++--- 2 07 05:00:00 3.08 1.05 CPU.83 enqueue +++++-----2------ @aveactn TO_CHAR(STA AAS AAS1 FIRST AAS2 SECOND GRAPH ----------- ------- ----- --------------- ----- --------------- ------------------------ 06 13:00:00.30.11 db file sequent.10 CPU - 2 06 14:00:00 2.24.87 direct path wri.49 log file sync ++--------2--- 06 15:00:00 6.67 2.07 direct path wri 1.93 log file sync ++++------2---------- 06 16:00:00 2.59 1.01 direct path wri.57 log file sync ++--------2---- 06 17:00:00 1.26.63 log file sync.32 CPU ++----- 2 06 18:00:00 1.38.61 CPU.45 log file sync +++---- 2 06 19:00:00 1.86.97 log file sync.39 CPU ++------- 2 06 20:00:00.99.38 log file sync.27 CPU +---- 2 06 21:00:00 1.22.57 log file sync.31 CPU ++----- 2 06 22:00:00 1.66.81 log file sync.41 CPU ++------ 2 06 23:00:00 1.08.49 log file sync.27 CPU +---- 2 07 00:00:00.83.37 log file sync.27 CPU +--- 2 07 01:00:00 1.74.96 log file sync.31 CPU ++------- 2 07 02:00:00 2.48 1.21 direct path wri.52 log file sync ++--------2---- 07 03:00:00 6.59 2.63 direct path wri 1.58 log file sync +++-------2---------- 07 04:00:00 1.95 1.29 CPU.27 log file sync ++++++--- 2 07 05:00:00 3.08 1.05 CPU.83 enqueue +++++-----2------

46. #.46 Copyright 2006 Kyle Hailey Many Ways to Attack Problems Confusing  How to Attack the problem?  Top SQL?  Top wait for that SQL?  Top Waiting Session ?  Top Waits for that Session  Top Waits for Database?  Top Session waiting for that wait  Top SQL for that wait Solution - Graphics

47. #.47 Copyright 2006 Kyle Hailey Mining Data is Non Trivial  Many Dimensions to consider  Constantly Varying time frames  Luckily 10g automates all of this functionality.  collects all of the data  analyses all the wait events  reports on bottlenecks  supplies solutions  Graphics add tremendous ease and power

48. Session 1 Session 2 Session 3 Session 4 TIME Graphical ASH

49. #.49 Copyright 2006 Kyle Hailey Graph of User States

50. #.50 Copyright 2006 Kyle Hailey One Second Graph

51. #.51 Copyright 2006 Kyle Hailey 15 Second Averages

52. #.52 Copyright 2006 Kyle Hailey Maximum CPU Line

53. #.53 Copyright 2006 Kyle Hailey Idle Users

54. #.54 Copyright 2006 Kyle Hailey OEM Perf Page

55. #.55 Copyright 2006 Kyle Hailey OEM Perf Page

56. #.56 Copyright 2006 Kyle Hailey ASH vs Statistics  Statistics  are more expensive  have lag time  lack clear identification of culprits

57. #.57 Copyright 2006 Kyle Hailey Statistic Lag Time Counters Samples Slight Lags

58. #.58 Copyright 2006 Kyle Hailey CPU Lag Problem  ASH is the only way to see CPU usage realtime  V$sysstat reports CPU but  is only updated at the end of the call.  Long calls look deceiving like no CPU is being used

59. #.59 Copyright 2006 Kyle Hailey CPU in ASH vs Stats

60. #.60 Copyright 2006 Kyle Hailey ASH Sampling Cheap Three Methods 1.With PL/SQL it’s less that 1 % CPU with 10 average active sessions 2.DMA - direct memory attach to SGA Using C code 200x Cheaper 3.ASH in 10g should be even cheaper than DMA because it’s done in the kernel

61. #.61 Copyright 2006 Kyle Hailey ASH – In Memory  Collects active session data only  History v$session_wait + v$session + extras  Circular Buffer - 1M to 128M (~2% of SGA)  Flushed every hour to disk or when buffer 2/3 full (it protects itself so you can relax)

62. #.62 Copyright 2006 Kyle Hailey ASH Sizing …  Avg row around 150bytes  3600 secs in an hour  ~ ½ Meg per Active Session per hour  That’s generally over an hour of ASH

63. #.63 Copyright 2006 Kyle Hailey How ASH works V$session_wait + v$session + extras MMON/ MMNL v$active_session_history wrh$_active_session_history Only writes out 1 in 10 rows via direct path inserts

64. #.64 Copyright 2006 Kyle Hailey Select reads backwards - -No latching - -No read consistency - -Index on time Insert point ASH buffer Insert one direction - Touch up wait times

65. #.65 Copyright 2006 Kyle Hailey Family of ASH Tables v$session_wait v$active_session_history wrh$active_session_history V$session_wait_history DBA_HIST_ACTIVE_SESS_HISTORY

66. #.66 Copyright 2006 Kyle Hailey ASH Tables v$session_wait SID SEQ# EVENT P1TEXT P1 P1RAW P2TEXT P2 P2RAW P3TEXT P3 P3RAW WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS WAIT_TIME SECONDS_IN_WAIT STATE v$session_wait_history SID SEQ# EVENT# EVENT P1TEXT P1 P2TEXT P2 P3TEXT P3 WAIT_TIME v$event_name EVENT_ID EVENT# EVENT_ID NAME PARAMETER1 PARAMETER2 PARAMETER3 WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS v$active_session_history SAMPLE_ID SAMPLE_TIME SESSION_ID SESSION_SERIAL# USER_ID SQL_ID SQL_CHILD_NUMBER SQL_PLAN_HASH_VALUE SQL_OPCODE SERVICE_HASH SESSION_TYPE SESSION_STATE QC_SESSION_ID QC_INSTANCE_ID SEQ# EVENT# P1 P2 P3 WAIT_TIME TIME_WAITED CURRENT_OBJ# CURRENT_FILE# CURRENT_BLOCK# PROGRAM MODULE ACTION CLIENT_ID current 10 samples hour DBA_HIST_ACTIVE_SESS_HISTORY wrh$active_session_history SNAP_D DBID INSTANCE_NUMBER SAMPLE_ID SAMPLE_TIME SESSION_ID SESSION_SERIAL# USER_ID SQL_ID SQL_CHILD_NUMBER SQL_PLAN_HASH_VALUE SQL_OPCODE SERVICE_HASH SESSION_TYPE SESSION_STATE QC_SESSION_ID QC_INSTANCE_ID SEQ# EVENT_ID P1 P2 P3 WAIT_TIME TIME_WAITED CURRENT_OBJ# CURRENT_FILE# CURRENT_BLOCK# PROGRAM MODULE ACTION CLIENT_ID 7 days (disk) 1 in 10

67. #.67 Copyright 2006 Kyle Hailey Wait Time vs Time Waited  SESSION_STATE  Waiting, on CPU  Based on WAIT_TIME  WAIT_TIME (v$session, v$session_wait, v$ash)  0 => waiting  >0 => CPU (value is time of last wait)  TIME_WAITED  Actual time waited for event  0 until wait finishes  Fix up values (no one else can do this)

68. #.68 Copyright 2006 Kyle Hailey Oradebug  Dump to trace file SQL> oradebug dump ash 5 SQL> Alter session set events ‘immediate tracename ashdump level 5’; level 5 = # of minutes loader file rdbms/demo/ashldr.ctl SQL> oradebug ashdump 5 SQL> Alter session set events ‘immediate tracename ashdump level 5’;

69. #.69 Copyright 2006 Kyle Hailey INIT.ORA statistics_level = Typical (default) PARAMETER SESSION_VALUE INSTANCE_VAL ----------------------- ---------- ------------ _ash_sampling_interval 1000 1000 milliseconds _ash_size 1048618 1048618 ASH buffer size _ash_enable TRUE TRUE Turn on/off ASH sampling, flushing and the V$ views on ASH _ash_disk_write_enable TRUE TRUE Flush to disk _ash_disk_filter_ratio 10 10 write 1 in 10 points _ash_sample_all FALSE FALSE Sample including idle waits

70. #.70 10.2 ASH Extras  BLOCKING_SESSION  BLOCKING_SESSION_STATUS  BLOCKING_SESSION_SERIAL#  P1TEXT  P2TEXT  P3TEXT  WAIT_CLASS  WAIT_CLASS_ID  XID 70 11/20/2015

71. #.71 Copyright 2006 Kyle Hailey 10.2 Extras  10gR2 add fields to ASH  Blocking Session Id, serial# and state ***  XID, transaction ID  RAC event Fixup  Plan Hash Fix up  10gR1 - during parsing no plan, good way to find parsing problems  10gR2 – get plan hash fixup – good but lose some ability to find parsing problems  Wait Class, needed this for grouping  Force_matching_signature – sql profile id

72. #.72 10.2.0.3 ASH Extras  FORCE_MATCHING_SIGNATURE  PLSQL_ENTRY_OBJECT_ID  PLSQL_ENTRY_SUBPROGRAM_ID  PLSQL_OBJECT_ID  PLSQL_SUBPROGRAM_ID 72 11/20/2015 ALL_PROCEDURES where object_id = plsql_object_id where object_id = plsql_object_id and subprogram_id = plsql_subprogram_id

73. #.73 73 11/20/2015 PLSQL Tracking  Package/Procedure/SQL tracking 10.2.0.3 calling_code SQL_ID COUNT(*) ----------------------------------------------- ---------- ORDERENTRY.NEWORDER 0uuqgjq7k12nf 258 ORDERENTRY.NEWCUSTOMER 0bzhqhhj9mpaa 262 ORDERENTRY.BROWSEANDUPDATEORDERS 41zu158rqf4kf 301 ORDERENTRY.NEWORDER 0yas01u2p9ch4 569 PARSE_SAME 3vjxpmhhzngu4 874 ORDERENTRY.BROWSEANDUPDATEORDERS 05s4vdwsf5802 1669 select object_name package||.|| procedure_name, sql_id, count(*) from v$active_session_history ash, all_procedures procs where ash.PLSQL_ENTRY_OBJECT_ID = procs.object_id and ash.PLSQL_ENTRY_SUBPROGRAM_ID = procs.SUBPROGRAM_ID

74. #.74 Copyright 2006 Kyle Hailey 11g ASH extras SQL Elapsed SQL_EXEC_ID SQL_EXEC_START IN_CONNECTION_MGMT IN_PARSE IN_HARD_PARSE IN_SQL_EXECUTION IN_PLSQL_EXECUTION IN_PLSQL_RPC IN_PLSQL_COMPILATION IN_JAVA_EXECUTION IN_BIND IN_CURSOR_CLOSE SQL_PLAN_LINE_ID SQL_PLAN_OPERATION SQL_PLAN_OPTIONS CURRENT_ROW# EVENT# QC_SESSION_SERIAL# CONSUMER_GROUP_ID FLAGS ETC CPU Analysis – non Timed SQL Row Source REMOTE_INSTANCE# RAC – remote transfers TOP_LEVEL_SQL_ID TOP_LEVEL_SQL_OPCODE Recursive SQL

75. #.75 Copyright 2006 Kyle Hailey 11g ASH extras  Run-time SQL row source information  Identifies current row source within plan  SQL execution ID  Is this same execution as last sample?  can see how long sql has been running  Operation bit vector  Capture non-timed operations  Examples: fetch, binding, close  Remote instance id for Cache transfers  Which instance sourced requested block?

76. #.76 Copyright 2006 Kyle Hailey How Many Active Sessions?  How much data does ASH Collect ?  1 CPU means max 1 Avg Active Session unless there is a bottleneck Big site examples:  Oracle 4 way RAC internal apps  10,000 connected, 200 active  One Site  3000 connected, 30 Active  Site  12,000 connected, 100 active

77. #.77 Copyright 2006 Kyle Hailey Simulating ASH   ASH is new in 10g   ASH data exist since V7   Need Diagnostic Pack License    Collect it yourself   SASH – Simulated ASH http://perfvision.com/sash.php   Graphical Monitor http://perfvision.com/ashmon.php Consumes < 1% CPU for 10 active sessions (a lot)

78. #.78 Copyright 2006 Kyle Hailey Sampling  Sampling is the future  Took clean vision to create  Bit of a leap of faith  Less Accurate but more powerful than past  Inaccuracies unimportant for the most part  Feasible since version 7  You can implement it yourself  Graphics are the only way to full harness the power

79. #.79 Copyright 2006 Kyle Hailey Summary  To solve hard problems you need Sampling  If you have 10g and a license you can use  V$active_session_history  OEM 10g (can access some 9 sampling)  ?/rdbms/admim/ashrpt.sql  If you are on 7,8,9 or 10g without the license  Need custom sampling  SASH – Simulated ASH http://perfvision.com/ash.php  ASHMON – graphical http://perfvision.com/ashmon.php

80. #.80 Copyright 2006 Kyle Hailey ASH Summary Faster, Cheaper, instant (no lag) … but most importantly Rich & Multidimensional  Overview of system load  Drilldowns into problems  Sql  Session  Wait event  Details information on problem resolution  OEM simplifies mining of ASH data