1 3/21/2016 Average Active Sessions (AAS) The Golden Metric ? Kyle Hailey perfvision.com

Copyright 2006 Kyle Hailey Who is Kyle Hailey 1990 Oracle 90 support 92 Ported v6 93 France 95 Benchmarking 98 ST 2000 Dot.Com 2001 Quest 2002 Oracle OEM 10g 2005 Embarcadero 2006 Independent When not being being a Geek writing for the Oaktable etc - Just Married and new home owner in Portland - photographer - and wonder how I missed the dot.com millions

3 3/21/2016 Copyright 2006 Kyle Hailey Goal of this Presentation Cut out unnecessary Hone in on essential Simplify the data and empower the DBA

Graphics “The human nervous system … is exceptionally good at parsing visual information, especially when that information is coded by color and/or motion.” Knowledge representation in cognitive science. Westbury, C. & Wilensky, U. (1998)

5 3/21/2016 Database Performance When it comes to Database Performance How quick can you find Bottleneck in DB If DB is idle Current DB Load what is DB Load ? What do you use? Statspack/AWR V$active_session_history Alerts what do you alert on ? Or as Tanel said “ what the *!####!*!*? is the database doing ?!

6 3/21/2016 Copyright 2006 Kyle Hailey In this Session 1. AAS Single Metric Shows DB Performance 2. Yardstick Max CPU CPU Count To measure AAS against 3. Subcomponents CPU Waits Time series +Geek stuff on ASH & SASH for Hotsos

7 3/21/2016 Statspack / AWR 1000 lines of data / 30 pages What do you look at ? Top 5 Timed Events? Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time CPU time 11, log file sync 163,725 1, enqueue 4, latch free 28, db file sequential read 1,420, Jonathan Lewis minutes CPU Elapsed 15 minutes 48 Processors or 74% idle

8 3/21/2016 Copyright 2006 Kyle Hailey What’s the DB Doing?! It’s 2am … your manager calls Whip out the stethoscope: AAS As Tanel said “ what the *!####!*!*? is the database doing ?!

9 3/21/2016 Copyright 2006 Kyle Hailey Welcome to … The Cult of AAS Once you’ve been initiated … there is no going back For those of you who are already members, welcome back

10 3/21/2016 Copyright 2006 Kyle Hailey Average Active Sessions Centi-seconds per second In the dark ages waits were often as Centi-secs per sec Sometimes called secs/sec Usually didn’t include CPU time, which is essential Average Active Sessions (AAS) OEM 10g Graphs ASH Report Session Load I often refer to it this way

11 3/21/2016 Copyright 2006 Kyle Hailey AAS Average load of sessions on the database Average over 15 secs in OEM 10g Period varies in other tools measured in number of sessions Active Sessions Only Active sessions put load the database Inactive Sessions don’t put load ( Except for memory usage ) The Golden Metric Powerful Multidimensional Indispensable

12 3/21/2016 Copyright 2006 Kyle Hailey AAS Calculation DB TIME AAS = DB TIME / Elapsed Time DB Time = sum over all sessions of time spent in a call ex:Session executing SQL Background process working

13 3/21/2016 Copyright 2006 Kyle Hailey AAS Sources 1. Manually from v$sysstat (9i ) v$system_event 2. Statspack Need several calculations 3. AWR One calculation 4. OEM 10g Directly displayed

14 3/21/2016 Copyright 2006 Kyle Hailey 1. Manually DB Time = sum active session time DB TIME (10g) DB TIME = (9i) Select sum(time_waited) from v$system_event where event not in (... idle events …); + Select value from v$sysstat where name = ‘CPU used by this session’; Select sum(time_waited) from v$system_event where event not in (... idle events …); + Select value from v$sysstat where name = ‘CPU used by this session’; select value from v$sysstat where name = ‘DB time’; select value from v$sysstat where name = ‘DB time’; ‘DB time’

15 3/21/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS Look for Elapsed Time Top 5 Timed Events Start at line 52 of about 1300!Start at line 52 of about 1300!

16 3/21/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS Elapsed Time Look at Top 5 Timed Events Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host LABSF labsf NO labsfr Snap Id Snap Time Sessions Curs/Sess Begin Snap: 1 03-Apr-06 12:34: End Snap: 2 03-Apr-06 12:34: Elapsed: 1.00 (mins) STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host LABSF labsf NO labsfr Snap Id Snap Time Sessions Curs/Sess Begin Snap: 1 03-Apr-06 12:34: End Snap: 2 03-Apr-06 12:34: Elapsed: 1.00 (mins)

17 3/21/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS DBTIME= CPU + WAITS CPU = 32 WAITS = = 278 secs DBTIME=320 Elapsed Time = 60 secs 320 secs / 60 secs AAS = 5.1 Top 5 Timed Events Event Time (s) buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5 Top 5 Timed Events Event Time (s) buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5

18 3/21/ AWR Report AAS = DB Time/Elapsed Time 23.56/59.66 = 0.39 AAS= 0.39

19 3/21/ OEM 10g AAS = ~0.75 OEM 10g

20 3/21/2016 Copyright 2006 Kyle Hailey Got AAS, Now What ? We Need one more item: CPU Count # of CPUs available on System Shared with other applications Need to track CPU used on the system as well On dual & quad cores, lower the CPU count Represents max active sessions that can do work

21 3/21/2016 Copyright 2006 Kyle Hailey CPU Count # of CPUs available in Statspack 10g AWR report OEM 10g Statspack 9i # of CPUs missing # of CPUs SQLPLUS> show parameters cpu_count NAME VALUE cpu_count 2 SQLPLUS> show parameters cpu_count NAME VALUE cpu_count 2

22 3/21/2016 Copyright 2006 Kyle Hailey AAS Formulas Use CPU count as yardstick: < 1 Database is not blocked AAS ~= 0 Database basically idle Problems are in the APP not DB AAS < # of CPUs CPU available Database is probably not blocked Are any single sessions 100% active? AAS > # of CPUs Could have performance problems  AAS >> # of CPUS There is a bottleneck Ideal world – one database solution track CPU at OS AAS > 1 still want to know if a single user is 100% active

23 3/21/2016 Copyright 2006 Kyle Hailey Available CPU vs AAS AAS far above available CPU => problem AAS = 5.1 # of CPU = 2 AAS = 0.39 # of CPU = 2 AAS < 1, database is fine

24 3/21/2016 Copyright 2006 Kyle Hailey Going Farther with AAS AAS can tell you a lot But it’s components tell you much more To go farther need the components of AAS 1.CPU 2.Wait 3.Value over time Only OEM 10g has all 3 (Statspack and AWR are aggregated over the snapshot period)

25 3/21/2016 AAS Components : OEM 10g OEM 10g Performance Page Available CPU AAS: CPU + WAIT Real CPU available: Max CPU - non instance CPU

26 3/21/2016 OEM 10g Relax Get to Work! Looks OK But …

27 3/21/2016

28 3/21/2016 Limited Analysis What if you find a problem ? Of the 800 waits most need SQL and/or P1,P2,P3 to solve What if there is a wait bottleneck ? What SQL Which sessions Values of P1, P2 and P3 Statspack and AWR fail Example “easy” waits free buffer waits log buffer space log file switch (archiving needed) log file switch (checkpoint incomplete) log file switch completion Example “hard” waits Buffer busy wait Row cache lock Latch free row lock contention Latch: cache buffers chains

29 3/21/2016 Copyright 2006 Kyle Hailey Alternative AAS Calc AAS = DB TIME / Elapsed Time But there is another way …

30 3/21/2016 Copyright 2006 Kyle Hailey AAS based on ASH ASH - Active Session History v$active_session_history AAS = count(*) / elapsed_seconds A statistical approximation, but surprisingly close ASH data source empowers drilldowns Top Sql Top Waits Details p1,p2,p3 and more

31 3/21/2016 Copyright 2006 Kyle Hailey AAS from ASH 1. ASHRPT Based entirely on Exec ASH_REPORT_TEXT/HTML 2. OEM 10g Top Activity page Displays it directly Enables Drilldowns 3. Custom queries on v$active_session_history select * from table (dbms_workload_repository.ash_report_text( (select dbid from v$database), (select dbid from v$database), 1, 1, sysdate – 1/24, sysdate – 1/24, sysdate )) ; sysdate )) ;

32 3/21/2016 Copyright 2006 Kyle Hailey 1. ASHRPT ASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host TESTDB testdb NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: Average Active Sessions: Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class % Activity Sessions CPU + Wait for CPU CPU enq: TX - row lock contention Application buffer busy waits Concurrency latch: cache buffers chains Concurrency ASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host TESTDB testdb NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: Average Active Sessions: Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class % Activity Sessions CPU + Wait for CPU CPU enq: TX - row lock contention Application buffer busy waits Concurrency latch: cache buffers chains Concurrency

33 3/21/2016 Copyright 2006 Kyle Hailey 1. ASH RPT 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***

34 3/21/ ASHRPT over Time Waits over Time Not in AAS Difficult but better than nothing Compare to …

35 3/21/ OEM 10g : Top Activity Top Activity Based on ASH Enables Drilldowns Top SQL Top Session Drill into a session Stats Raw waits Open cursors General info Drill into a SQL Stats and text Users executing Explain plan Tuning options

36 3/21/ Custom Scripts Hate Graphics ? Query v$active_session_history directly Join to dba_hist_active_sess_history for week of data act.sql Like top 5 timed events Aveact.sql Charts with text AAS by hour (15 minute, minute, etc) Aveactn.sql Ditto, with top 2 wait events per bucket Following Scripts Available on

37 3/21/2016 Copyright 2006 Kyle Hailey 3. Custom Analysis Begin Time : :04:48 Analysis End Time : :19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess latch free log buffer space buffer busy waits db file scattered read library cache pin log file sync ON CPU enqueue db file sequential read sum Analysis Begin Time : :04:48 Analysis End Time : :19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess latch free log buffer space buffer busy waits db file scattered read library cache pin log file sync ON CPU enqueue db file sequential read sum 6.30

38 3/21/2016 Copyright 2006 Kyle Hailey 3. Custom TM NPTS AVEACT GRAPH CPU WAITS AUG 13:00: AUG 14:00: AUG 15:00: AUG 16:00: AUG 17:00: AUG 18:00: AUG 19:00: AUG 20:00: AUG 21:00: AUG 22:00: AUG 23:00: AUG 00:00: AUG 01:00: AUG 02:00: AUG 03:00: AUG 04:00: AUG 05:00: AUG 06:00: TM NPTS AVEACT GRAPH CPU WAITS AUG 13:00: AUG 14:00: AUG 15:00: AUG 16:00: AUG 17:00: AUG 18:00: AUG 19:00: AUG 20:00: AUG 21:00: AUG 22:00: AUG 23:00: AUG 00:00: AUG 01:00: AUG 02:00: AUG 03:00: AUG 04:00: AUG 05:00: AUG 06:00:

39 3/21/2016 Copyright 2006 Kyle Hailey 3. Custom TO_CHAR(STA AAS AAS1 FIRST AAS2 SECOND GRAPH :00: db file sequent.10 CPU :00: direct path wri.49 log file sync :00: direct path wri 1.93 log file sync :00: direct path wri.57 log file sync :00: log file sync.32 CPU :00: CPU.45 log file sync :00: log file sync.39 CPU :00: log file sync.27 CPU :00: log file sync.31 CPU :00: log file sync.41 CPU :00: log file sync.27 CPU :00: log file sync.27 CPU :00: log file sync.31 CPU :00: direct path wri.52 log file sync :00: direct path wri 1.58 log file sync :00: CPU.27 log file sync :00: CPU.83 enqueue TO_CHAR(STA AAS AAS1 FIRST AAS2 SECOND GRAPH :00: db file sequent.10 CPU :00: direct path wri.49 log file sync :00: direct path wri 1.93 log file sync :00: direct path wri.57 log file sync :00: log file sync.32 CPU :00: CPU.45 log file sync :00: log file sync.39 CPU :00: log file sync.27 CPU :00: log file sync.31 CPU :00: log file sync.41 CPU :00: log file sync.27 CPU :00: log file sync.27 CPU :00: log file sync.31 CPU :00: direct path wri.52 log file sync :00: direct path wri 1.58 log file sync :00: CPU.27 log file sync :00: CPU.83 enqueue

40 3/21/2016 Copyright 2006 Kyle Hailey In Review: Two Sources 1. v$system_event & v$sysstat Indirect Accurate Lags (especially CPU) Limits analysis 2. v$active_session_history Direct Real time Approximation ***Allows drilldowns***

41 3/21/2016 Copyright 2006 Kyle Hailey Samples VS CountersCounters Samples Slight Lags v$system_event v$active_session_history

42 3/21/2016 Copyright 2006 Kyle Hailey CPU in ASH vs Stats

43 3/21/2016 Solutions ? AAS is good at finding problems. How do you find solutions? ADDM ? Includes machine info - not that complicated Paging & CPU Need license Some solutions missing – where do you look? ASH data (source for ASH ) Simulatable for free Time Slicing – no need for snap shot ids, down to the second Multi-dimensional SQL Session, Package, Procedure, Module, Action, Service, Client Object, File, Block Wait Details – P1, P2, P3

Copyright 2006 Kyle Hailey v$active_session_history 10.1 SESSION_ID NUMBER SESSION_SERIAL# NUMBER USER_ID NUMBER SERVICE_HASH NUMBER SESSION_TYPE VARCHAR2(10) PROGRAM VARCHAR2(64) MODULE VARCHAR2(48) ACTION VARCHAR2(32) CLIENT_ID VARCHAR2(64) EVENT VARCHAR2(64) EVENT_ID NUMBER EVENT# NUMBER SEQ# NUMBER P1 NUMBER P2 NUMBER P3 NUMBER WAIT_TIME NUMBER TIME_WAITED NUMBER CURRENT_OBJ# NUMBER CURRENT_FILE# NUMBER CURRENT_BLOCK# NUMBER0 SQL_ID VARCHAR2(13) SQL_CHILD_NUMBER NUMBER SQL_PLAN_HASH_VALUE NUMBER SQL_OPCODE NUMBER QC_SESSION_ID NUMBER QC_INSTANCE_ID NUMBER SAMPLE_ID NUMBER SAMPLE_TIME TIMESTAMP(3) When Session SQL Wait SESSION_STATE VARCHAR2(7) WAIT_TIME NUMBER State TIME_WAITED NUMBER Duration

45 3/21/2016 Simulating ASH Performance Package License? $3000/pack/CPU ? Not on 10g ? Do it yourself V$SESSION 10g most of ASH Before 10g, join v$session and v$session_wait Sample every second Store in a table Free PL/SQL package available on 9i Centralize repository (need partitioning) v$active_session_history v$session SESSION_ID SID SESSION_SERIAL# SERIAL# USER_ID USER# SQL_OPCODE COMMAND PROGRAM SESSION_TYPE TYPE SQL_ID SQL_CHILD_NUMBER MODULE ACTION CURRENT_OBJ# ROW_WAIT_OBJ# CURRENT_FILE# ROW_WAIT_FILE# CURRENT_BLOCK# ROW_WAIT_BLOCK# CLIENT_ID CLIENT_IDENTIFIER BLOCKING_SESSION BLOCKING_SESSION_STATUS SEQ# EVENT# EVENT P1TEXT P1 P2TEXT P2 P3TEXT P3 WAIT_CLASS_ID WAIT_CLASS WAIT_TIME

46 3/21/2016 Simulating – Key Ideas session_state = decode(WAIT_TIME, 0,'WAITING', 'ON CPU') v$session where ON CPU : status='ACTIVE' and wait_time > 0 or Waiting : wait_class != 'Idle‘ select sysdate sample_time, decode(s.WAIT_TIME, 0,'WAITING','ON CPU') "SESSION_STATE“ /* plus other fields */ from v$session where sid != ( select distinct sid from v$mystat where rownum < 2 ) and ( ( s.status='ACTIVE' and wait_time > 0 ) or s.wait_class!='Idle' ) ; 9i – join v$session and v$session_wait

Copyright 2006 Kyle Hailey S-ASH – All HOST ASL GRAPH # of cpus bsn control cont bsn tsukuba limerock.13 2 bsn devnode.02 HOST ASL GRAPH # of cpus bsn control cont bsn tsukuba limerock.13 2 bsn devnode.02 1

48 3/21/2016 ASH extra fields v$session missing V$session extras (added in 11g ASH) FORCE_MATCHING_SIGNATURE v$sql.sql_id = sql_id QC_SESSION_ID QCSID v$px_session.sid=sid, serial# QC_INSTANCE_ID QCINST_ID v$px_session.sid=sid, serial# SQL_PLAN_HASH_VALUE PLAN_HASH_VALUE v$sqlstats.sql_id v$sqlstats.sql_id = sql_id (PLAN_HASH_VALUE = x$ksuse.KSUSESPH) XID v$transaction.ADDR = taddr FIXED_TABLE_SEQUENCE - find sql executions ROW_WAIT_ROW# Not in v$sqlstats

10.2 ASH Extras BLOCKING_SESSION BLOCKING_SESSION_STATUS BLOCKING_SESSION_SERIAL# P1TEXT P2TEXT P3TEXT WAIT_CLASS WAIT_CLASS_ID XID 49 3/21/2016

ASH Extras FORCE_MATCHING_SIGNATURE PLSQL_ENTRY_OBJECT_ID PLSQL_ENTRY_SUBPROGRAM_ID PLSQL_OBJECT_ID PLSQL_SUBPROGRAM_ID 50 3/21/2016 ALL_PROCEDURES where object_id = plsql_object_id where object_id = plsql_object_id and subprogram_id = plsql_subprogram_id

Copyright 2006 Kyle Hailey 11g ASH extras SQL Elapsed SQL_EXEC_ID SQL_EXEC_START IN_CONNECTION_MGMT IN_PARSE IN_HARD_PARSE IN_SQL_EXECUTION IN_PLSQL_EXECUTION IN_PLSQL_RPC IN_PLSQL_COMPILATION IN_JAVA_EXECUTION IN_BIND IN_CURSOR_CLOSE SQL_PLAN_LINE_ID SQL_PLAN_OPERATION SQL_PLAN_OPTIONS CURRENT_ROW# EVENT# QC_SESSION_SERIAL# CONSUMER_GROUP_ID FLAGS ETC CPU Analysis – non Timed TOP_LEVEL_SQL_ID TOP_LEVEL_SQL_OPCODE Track Recursive SQL SQL Row Source REMOTE_INSTANCE# RAC – remote transfers

52 3/21/2016 Wait Drill Downs latch: Cache Buffers Chains Find Top SQL Buffer Busy Wait Find Block Type, Object, Object Type and SQL IO Find SQL, file, block, object enq: HW – contention lock mode, sql enq: TX - row lock contention Get lock mode from P1 Get waiter and SQL, block, file, object and blocker Get transaction from log miner with XID from ASH IO db file sequential read db file scattered read db file parallel read read by other session direct path read direct path write direct path write temp Latches 9i need to translate p2 into latch name (still need to in 10g for some ) Locks 9i need to translate p1 into lock name

Cache Buffers Chains : Diag Hash Buckets s5 s4 s3 s2 s1 Sessions Contention if too many accesses on a bucket latches Block Headers Cache Buffer Chain

54 3/21/2016 Latch: Cache Buffers Chains Select ash.sql_id, count(*), sql_text from v$active_session_history ash, v$sqlstats sql where event='latch: cache buffers chains' and sql.sql_id(+)=ash.sql_id group by ash.sql_id, sql_text; SQL_ID COUNT(*) SQL_TEXT vjxpmhhzngu4 52 SELECT sysdate FROM dual 3uz0s4srzcywr 21 SELECT ID FROM EMP_HASH WHERE EMPNO=6987

Buffer Busy Waits S1 S2 S3 S4 Index

Solutions depend on block type might get from statspack Object Not in statspack In AWR SQL Not in statspack Not in AWR 56 3/21/2016 Buffer Busy Waits Block Types:  Undo Header  use AUM (or add more RBS)  Undo Block  hot spot in UNDO, application issue  Data  index – hot spot, partition  table – free lists, ASSM, partition  Segment header  table data block -> freelists  Freelist blocks –  free lists groups  File Header Block  extent allocation too small / too many

57 3/21/2016 Buffer Busy Waits P1 P2 OBJN OTYPE SQL_ID BLOCK_TYPE BBW_INDEX_VAL_I INDEX 635xhydd6fzgg data block BBW_INDEX_VAL_I INDEX 6avm49ys4k7t6 data block BBW_INDEX_VAL_I INDEX 1hsb81ypyrfs5 data block select ash.p1 p1, ash.p2 p2, nvl(o.object_name,ash.current_obj#) objn, o.object_type otype, ash.SQL_ID, nvl(w.class,to_char(ash.p3)) block_type from v$active_session_history ash, ( select rownum class#, class from v$waitstat ) w, all_objects o where event='buffer busy waits' and w.class#(+)=ash.p3 and o.object_id (+)= ash.CURRENT_OBJ#;

58 3/21/2016 IO EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID db file sequential read 1 ORDER_ITEMS_UK INDEX PARTITION yas01u2p9ch4 db file scattered read 12 ORDERS TABLE PARTITION s4vdwsf5802 db file scattered read 13 ORDERS TABLE PARTITION s4vdwsf5802 db file sequential read 1 ITEM_PRODUCT_IX INDEX yas01u2p9ch4 db file sequential read 1 CUSTOMERS TABLE PARTITION g9y3xmvd db file sequential read 1 CUST_ _IX INDEX bzhqhhj9mpaa db file scattered read 13 ORDERS TABLE PARTITION s4vdwsf5802 select event, ash.p3, o.object_name objn, o.object_type otype, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, ash.SQL_ID from v$active_session_history ash, all_objects o where event like 'db file s%' and o.object_id (+)= ash.CURRENT_OBJ# order by sample_time ; CNT AAS OBJN OTYPE CUSTOMERS TABLE PARTITION ORDER_PK INDEX ORDER_ITEMS_PK INDEX

Copyright 2006 Kyle Hailey HWData Empty High Water Mark Header Table Session 1 Session 2 Session 3

HW Locks 60 3/21/2016 EVENT SQL_ID OBJ FILE# BLOCK# enq: HW - contention 49ch3jfkncnsp T1_I select event, event, sql_id, sql_id, CURRENT_OBJ# || ' ' || name obj CURRENT_OBJ# || ' ' || name obj,CURRENT_FILE# file#,CURRENT_FILE# file#,CURRENT_BLOCK# block#,CURRENT_BLOCK# block# from v$active_session_history ash, from v$active_session_history ash, obj$ o obj$ o where where event like 'enq: HW%' event like 'enq: HW%' and o.obj# (+)= ash.current_obj# and o.obj# (+)= ash.current_obj# order by sample_time ; order by sample_time ;

61 3/21/2016 Enq: TX – row lock contention SID LM P2 P3 OBJN OTYPE FN BLOCKN SQL_ID BSID XID TITI TABLE z5h2z2qqw I1 INDEX 0 0 azav296xxqcjx F00F ahm7c9rupbz9r bjvx94vnxtxgv 140 0A002300ED P2=usn<<16 | slot P3=sequence XID logmnr row lock fk pk Bitmap index

62 3/21/2016 PLSQL Tracking Package/Procedure/SQL tracking calling_code SQL_ID COUNT(*) ORDERENTRY.NEWORDER 0uuqgjq7k12nf 258 ORDERENTRY.NEWCUSTOMER 0bzhqhhj9mpaa 262 ORDERENTRY.BROWSEANDUPDATEORDERS 41zu158rqf4kf 301 ORDERENTRY.NEWORDER 0yas01u2p9ch4 569 PARSE_SAME 3vjxpmhhzngu4 874 ORDERENTRY.BROWSEANDUPDATEORDERS 05s4vdwsf select object_name package||.|| procedure_name, sql_id, count(*) from v$active_session_history ash, all_procedures procs where ash.PLSQL_ENTRY_OBJECT_ID = procs.object_id and ash.PLSQL_ENTRY_SUBPROGRAM_ID = procs.SUBPROGRAM_ID

63 3/21/2016 Top SQL Aggregate by SQL id Plan hash Force_matching_signature SQL_ID PLAN_HASH_ TYPE CPU WAIT IO TOTAL vjxpmhhzngu SELECT uz0s4srzcywr SELECT nxsran3qmq 0 PL/SQL EXE u3hm6xnq4k1n 0 PL/SQL EXE wqps1quuxqr UPDATE xhydd6fzgg UPDATE hsb81ypyrfs UPDATE avm49ys4k7t UPDATE nhzyn560xyhs SELECT

Time Slicing Previous examples were unbounded Need Time slice bounds Where sample_time > &begin and sample_time < &end Detail down to the seconds 64 3/21/2016 TM NPTS AAS GRAPH CPU WAITS MAR 12:28: MAR 12:28: MAR 12:29: MAR 12:29: MAR 12:29: MAR 12:29: MAR 12:30: MAR 12:30: MAR 12:30: MAR 12:30: MAR 12:32: MAR 12:33: MAR 12:34:

Explain Plans DBMS_XPLAN.DISPLAY_CURSOR( sql_id ); 65 3/21/2016 set linesize 150 set pagesize 100 col PLAN_TABLE_OUTPUT for a100 select * from TABLE(DBMS_XPLAN.DISPLAY_CURSOR('&1')); SQL_ID 777z5h2z2qqw6, child number delete from titi where dummy='X' Plan hash value: | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | | 0 | DELETE STATEMENT | | | | 2 (100)| | | 1 | DELETE | TITI | | | | | |* 2 | TABLE ACCESS FULL| TITI | 1 | 2 | 2 (0)| 00:00:01 | Predicate Information (identified by operation id): filter("DUMMY"='X') Note dynamic sampling used for this statement 9i: SELECT * FROM TABLE( dbms_xplan.display('SQLPLANS','&sql_id'));

Graphics How do you keep track of Scripts for Types of wait events Aggregations by different dimensions Changing Time Slices 66 3/21/2016 Graphics

OEM / ASHMON Aggregates Drilldowns Time Slicing Guidance 67 3/21/2016

Copyright 2006 Kyle Hailey Test 3 Elapsed 1:23 14:44 14:59 Elapsed Time 1:23 10x Improvement ! Still more room for Improvement – a lot Of CPU left

Copyright 2006 Kyle Hailey ASHMON “OEM-lite” Written in TCL/TK Only Overview SQL Text

DEMO DEMO ASHMON? 70 3/21/2016

Parsability of Graphics Infocus – (overhead projectors) sited a study that humans can parse graphical information 400,000 times faster than textual data

What are the Trends?

Trends: With Graphics I II III IV

Trends: Waits Over Time date17:1317:1317:1417:1417:1517:1517:1617:1617:1717:17 17:1817:1817:1917:1917:2017:2017:2117:2117:22 17:2217:2317:2317:2417:2417:2517:2517:2617:26 17:2717:2717:2817:2817:2917:2917:3017:3017:31 17:3117:3217:3217:3317:3317:3417:3417:3517:35 17:3617:3617:3717:3717:3817:3817:3917:3917:40 17:4017:4117:4117:42 log_file_switch_completion_ log_buffer_space_ QL*Net_break/reset_to_client_ latch_free_ rdbms_ipc_reply_ log_file_sync_ enqueue_ db_file_sequential_read_ itch_(checkpoint_incomplete)_

Trends: Waits over Time

76 3/21/2016 Counties in US 3101 Counties in US 50 pages

Copyright 2006 Kyle Hailey Launch: Pressure Midnight before January 28, 1986 Lives are on the line Thanks to Edward Tufte

Copyright 2006 Kyle Hailey 13 Pages Faxed

Copyright 2006 Kyle Hailey Correlated with Temperatures

Copyright 2006 Kyle Hailey Trying to Show data an Analysis

Copyright 2006 Kyle Hailey Congressional Hearings Evidence

Copyright 2006 Kyle Hailey Only Damage Engineering Document only Showed Damage

Copyright 2006 Kyle Hailey Clearer 1.Include successes 2.Remove Irrelevant 3.Normalize same temp 4.Scale known vs unknown ? ? ?

Copyright 2006 Kyle Hailey Do You Want? Engineering Data?

Copyright 2006 Kyle Hailey Do You Want? Pretty Pictures

Copyright 2006 Kyle Hailey Do You Want? Clean and Clear

Copyright 2006 Kyle Hailey Difficult NASA Engineers Fail Congressional Investigators Fail Data is Difficult But … Lack of Clarity can be devastating

Copyright 2006 Kyle Hailey Imagine Trying to Drive your Car And is updated once and hour Or would you like it to look … if you Dashboard looked like:

Copyright 2006 Kyle Hailey Or This And is updated once and hour

Conclusios AAS Identify Monitor, Alert ASH Drilldown and Solutions Top SQL/Session Module/Action Client/Service Package/procedure SQL, Wait, p1/2/3, Object, file, block, XID, Graphics Ease of Use Data Compression Parsability Guidance 90 3/21/2016 References Download at ASH Simulation Graphical “OEM-lite” ASH Scripts