Average Active Sessions (AAS) The Golden Metric ? Kyle Hailey

2 11/29/2015 Database Performance  How quick can you find  Bottleneck in DB  If DB is idle  Current DB Load  what is DB Load ?  What do you use?  Statspack/AWR  V$active_session_history  Alerts  what do you alert on ? what the *!####!*!*? is the database doing ?!

# /29/2015 Statspack / AWR  1000 lines of data / 30 pages  What do you look at ?  Top 5 Timed Events? Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time CPU time 11, log file sync 163,725 1, enqueue 4, latch free 28, db file sequential read 1,420, Jonathan Lewis minutes CPU Elapsed 15 minutes 48 Processors or 74% idle

# /29/2015 Copyright 2006 Kyle Hailey In this Session 1.AAS  Single Metric  Shows DB Performance 2.Yardstick  Max CPU  CPU Count  To measure AAS against 3.Subcomponents  CPU  Waits  Time series

5 11/29/2015 Copyright 2006 Kyle Hailey Goal of this Presentation Cut out unnecessary Hone in on essential Simplify the data and empower the DBA

6 11/29/2015 Copyright 2006 Kyle Hailey What’s the DB Doing?! It’s 2am … your manager calls Whip out the stethoscope: AAS what the *!####!*!*? is the database doing ?!

#.7 Copyright 2006 Kyle Hailey Welcome to … The Cult of AAS Once you’ve been initiated … there is no going back For those of you who are already members, welcome back

#.8 Copyright 2006 Kyle Hailey AAS Calculation AAS = DB TIME / Elapsed Time DB Time (DBT) = sum over all sessions of time spent in a call. A call could be executing SQL background work DBWR writing blocks LGWR writing redo

# /29/2015 Copyright 2006 Kyle Hailey Average Active Sessions  Centi-seconds per second  In the dark ages waits were often as Centi-secs per sec  Sometimes called secs/sec  Usually didn’t include CPU time, which is essential  Average Active Sessions (AAS)  OEM 10g Graphs  ASH Report  Session Load  I often refer to it this way

# /29/2015 Copyright 2006 Kyle Hailey AAS  Average load of sessions on the database  Average over 15 secs in OEM 10g  Period varies in other tools  measured in number of sessions  Active Sessions Only  Active sessions put load the database  Inactive Sessions don’t put load ( Except for memory usage )  The Golden Metric  Powerful  Multidimensional  Indispensable

# /29/2015 Copyright 2006 Kyle Hailey AAS Sources 1.Manually from  v$sysstat  (9i ) v$system_event 2.Statspack  Need several calculations 3. AWR  One calculation 4.OEM 10g  Directly displayed

#.12 Copyright 2006 Kyle Hailey 1. Manually DB Time (DBT) = sum active session time DB TIME (10g) = DB TIME (9i) = Select sum(time_waited) from v$system_event where event not in (... idle events …); + Select value from v$sysstat where name = ‘CPU used by this session’; Select sum(time_waited) from v$system_event where event not in (... idle events …); + Select value from v$sysstat where name = ‘CPU used by this session’; select value from v$sysstat where name = ‘DB time’; select value from v$sysstat where name = ‘DB time’; ‘DB time’

# /29/2015 Copyright 2006 Kyle Hailey 2. Statspack AAS  Look for  Elapsed Time  Top 5 Timed Events  Start at line 52 of about 1300!

#.14 Copyright 2006 Kyle Hailey AWR/Statspack: Cheat Sheet  Install  Connect as SYSDBA  Run  Exec statspack.snap;  Generate Reports

# /29/2015 Copyright 2006 Kyle Hailey 2. Statspack AAS  Elapsed Time  Look at Top 5 Timed Events Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host LABSF labsf NO labsfr Snap Id Snap Time Sessions Curs/Sess Begin Snap: 1 03-Apr-06 12:34: End Snap: 2 03-Apr-06 12:34: Elapsed: 1.00 (mins) STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host LABSF labsf NO labsfr Snap Id Snap Time Sessions Curs/Sess Begin Snap: 1 03-Apr-06 12:34: End Snap: 2 03-Apr-06 12:34: Elapsed: 1.00 (mins)

# /29/2015 Copyright 2006 Kyle Hailey 2. Statspack AAS  DBTIME= CPU + WAITS  CPU = 32  WAITS = = 278 secs DBTIME=320  Elapsed Time = 60 secs  320 secs / 60 secs AAS = 5.1 Top 5 Timed Events Event Time (s) buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5 Top 5 Timed Events Event Time (s) buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5

# /29/ AWR Report  AAS = DB Time/Elapsed Time  23.56/59.66 = 0.39 AAS= 0.39

# /29/ OEM 10g AAS = ~0.75 OEM 10g

# /29/2015 Copyright 2006 Kyle Hailey Got AAS, Now What ? We Need one more item: CPU Count  # of CPUs available on System  Shared with other applications  Need to track CPU used on the system as well  On dual & quad cores, lower the CPU count  Represents max active sessions that can do work

# /29/2015 Copyright 2006 Kyle Hailey CPU Count  # of CPUs available in  Statspack 10g  AWR report  OEM 10g  Statspack 9i # of CPUs missing # of CPUs SQLPLUS> show parameters cpu_count NAME VALUE cpu_count 2 SQLPLUS> show parameters cpu_count NAME VALUE cpu_count 2

# /29/2015 Copyright 2006 Kyle Hailey AAS Formulas Use CPU count as yardstick: < 1 Database is not blocked AAS ~= 0 Database basically idle Problems are in the APP not DB AAS < # of CPUs CPU available Database is probably not blocked Are any single sessions 100% active?  AAS > # of CPUs Could have performance problems  AAS >> # of CPUS There is a bottleneck Ideal world – one database solution track CPU at OS AAS > 1 still want to know if a single user is 100% active

# /29/2015 Copyright 2006 Kyle Hailey Available CPU vs AAS AAS far above available CPU => problem AAS = 5.1 # of CPU = 2 AAS = 0.39 # of CPU = 2 AAS < 1, database is fine

# /29/2015 Copyright 2006 Kyle Hailey Going Farther with AAS  AAS can tell you a lot  But it’s components tell you much more  To go farther need the components of AAS 1. CPU 2. Wait 3. Value over time Only OEM 10g has all 3 (Statspack and AWR are aggregated over the snapshot period)

#.24 Copyright 2006 Kyle Hailey EM DB Home Page

#.25 Copyright 2006 Kyle Hailey OEM 10g Perf Pages DB Home Performance AAS Point in Time AAS over Time

# /29/2015 AAS Components : OEM 10g  OEM 10g Performance Page Available CPU AAS: CPU + WAIT Real CPU available: Max CPU - non instance CPU

# /29/2015 OEM 10gRelax Get to Work! Looks OK But …

# /29/2015 Limited Analysis  What if you find a problem ?  Of the 800 waits most need SQL and/or P1,P2,P3 to solve  What if there is a wait bottleneck ?  What SQL  Which sessions  Values of P1, P2 and P3  Statspack and AWR fail Example “easy” waits free buffer waits log buffer space log file switch (archiving needed) log file switch (checkpoint incomplete) log file switch completion Example “hard” waits Buffer busy wait Row cache lock Latch free row lock contention Latch: cache buffers chains

#.29 Copyright 2006 Kyle Hailey Alternative AAS Calculations AAS = DB TIME / Elapsed Time But there is another way …

# /29/2015 Copyright 2006 Kyle Hailey AAS based on ASH  ASH - Active Session History  v$active_session_history  AAS = count(*) / elapsed_seconds  A statistical approximation, but surprisingly close  ASH data source empowers drilldowns  Top Sql  Top Waits  Details p1,p2,p3 and more

# /29/2015 Copyright 2006 Kyle Hailey AAS from ASH 1.ASHRPT  Based entirely on v$active_session_history  Exec ASH_REPORT_TEXT/HTML 2.OEM 10g  Top Activity page  Displays it directly  Enables Drilldowns 3.Custom queries on v$active_session_history select * from table (dbms_workload_repository.ash_report_text( (select dbid from v$database), (select dbid from v$database), 1, 1, sysdate – 1/24, sysdate – 1/24, sysdate )) ; sysdate )) ;

# /29/2015 Copyright 2006 Kyle Hailey 1. ASHRPT ASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host TESTDB testdb NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: Average Active Sessions: Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class % Activity Sessions CPU + Wait for CPU CPU enq: TX - row lock contention Application buffer busy waits Concurrency latch: cache buffers chains Concurrency ASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host TESTDB testdb NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: Average Active Sessions: Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class % Activity Sessions CPU + Wait for CPU CPU enq: TX - row lock contention Application buffer busy waits Concurrency latch: cache buffers chains Concurrency

# /29/2015 Copyright 2006 Kyle Hailey 1. ASH RPT 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***

# /29/ ASHRPT over Time  Waits over Time  Not in AAS  Difficult but better than nothing Compare to …

# /29/ OEM 10g : Top Activity Top Activity Based on ASH Enables Drilldowns Top SQL Top Session Drill into a session Stats Raw waits Open cursors General info Drill into a SQL Stats and text Users executing Explain plan Tuning options

# /29/ Custom Scripts  Hate Graphics ?  Query v$active_session_history directly  Join to dba_hist_active_sess_history for week of data  act.sql  Like top 5 timed events  Aveact.sql  Charts with text AAS by hour (15 minute, minute, etc)  Aveactn.sql  Ditto, with top 2 wait events per bucket  Following Scripts Available on 

# /29/2015 Copyright 2006 Kyle Hailey 3. Custom Analysis Begin Time : :04:48 Analysis End Time : :19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess latch free log buffer space buffer busy waits db file scattered read library cache pin log file sync ON CPU enqueue db file sequential read sum Analysis Begin Time : :04:48 Analysis End Time : :19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess latch free log buffer space buffer busy waits db file scattered read library cache pin log file sync ON CPU enqueue db file sequential read sum 6.30

# /29/2015 Copyright 2006 Kyle Hailey 3. Custom TM NPTS AVEACT GRAPH CPU WAITS AUG 13:00: AUG 14:00: AUG 15:00: AUG 16:00: AUG 17:00: AUG 18:00: AUG 19:00: AUG 20:00: AUG 21:00: AUG 22:00: AUG 23:00: AUG 00:00: AUG 01:00: AUG 02:00: AUG 03:00: AUG 04:00: AUG 05:00: AUG 06:00: TM NPTS AVEACT GRAPH CPU WAITS AUG 13:00: AUG 14:00: AUG 15:00: AUG 16:00: AUG 17:00: AUG 18:00: AUG 19:00: AUG 20:00: AUG 21:00: AUG 22:00: AUG 23:00: AUG 00:00: AUG 01:00: AUG 02:00: AUG 03:00: AUG 04:00: AUG 05:00: AUG 06:00:

# /29/2015 Copyright 2006 Kyle Hailey 3. Custom TO_CHAR(STA AAS AAS1 FIRST AAS2 SECOND GRAPH :00: db file sequent.10 CPU :00: direct path wri.49 log file sync :00: direct path wri 1.93 log file sync :00: direct path wri.57 log file sync :00: log file sync.32 CPU :00: CPU.45 log file sync :00: log file sync.39 CPU :00: log file sync.27 CPU :00: log file sync.31 CPU :00: log file sync.41 CPU :00: log file sync.27 CPU :00: log file sync.27 CPU :00: log file sync.31 CPU :00: direct path wri.52 log file sync :00: direct path wri 1.58 log file sync :00: CPU.27 log file sync :00: CPU.83 enqueue TO_CHAR(STA AAS AAS1 FIRST AAS2 SECOND GRAPH :00: db file sequent.10 CPU :00: direct path wri.49 log file sync :00: direct path wri 1.93 log file sync :00: direct path wri.57 log file sync :00: log file sync.32 CPU :00: CPU.45 log file sync :00: log file sync.39 CPU :00: log file sync.27 CPU :00: log file sync.31 CPU :00: log file sync.41 CPU :00: log file sync.27 CPU :00: log file sync.27 CPU :00: log file sync.31 CPU :00: direct path wri.52 log file sync :00: direct path wri 1.58 log file sync :00: CPU.27 log file sync :00: CPU.83 enqueue

# /29/2015 Copyright 2006 Kyle Hailey Samples VS CountersCounters Samples Slight Lags v$system_event v$active_session_history

# /29/2015 Copyright 2006 Kyle Hailey CPU in ASH vs Stats

# /29/2015 Copyright 2006 Kyle Hailey In Review: Two Sources 1.v$system_event & v$sysstat  Indirect  Accurate  Lags (especially CPU)  Limits analysis 2.v$active_session_history  Direct  Real time  Approximation  ***Allows drilldowns***

#.43 Copyright 2006 Kyle Hailey The Power ASH gives AAS DB Home Performance Top Activity ASH eventsstatistics

#.44 Copyright 2006 Kyle Hailey ASH in OEM Top Activity gives more information

#.45 Copyright 2006 Kyle Hailey Top Activity : Based on ASH missing ThanksToASH

#.46 Copyright 2006 Kyle Hailey Top Activity : ASH Dimensions

#.47 Copyright 2006 Kyle Hailey AAS – %Session Time Issue Shown in % DB Time Missing % Session Time

#.48 Copyright 2006 Kyle Hailey Top Activity: ASH Sessions Many Users Active On Performance Page, no way to tell how many users But Top Activity Page fixes that

#.49 Copyright 2006 Kyle Hailey Top Activity: ASH Sessions Two Users Active

#.50 Copyright 2006 Kyle Hailey OEM 10g Perf Pages DB Home Performance Top Activity SQL Session

#.51 Copyright 2006 Kyle Hailey OEM 10g Perf Pages Top Activity SQL Session

#.52 Copyright 2006 Kyle Hailey Session : ASH Activity

#.53 Copyright 2006 Kyle Hailey SQL : ASH Activity

#.54 Copyright 2006 Kyle Hailey Getting the Most out of AAS  Need to know the System’s Profile  What your application is like?  Data Warehouse  OLTP  Typical load  Once you get to know it you can see anomalies  Is AAS near 0 when it should be higher  Is that Data Warehouse query running normal  Do you know what it looks like?  Is there an unusual bottleneck

#.55 Copyright 2006 Kyle Hailey Knowing your DB Profile

#.56 Copyright 2006 Kyle Hailey When to tune? General rules of Thumb  Waits >> CPU  CPU > Max CPU

#.57 Copyright 2006 Kyle Hailey Waits > CPU

#.58 Copyright 2006 Kyle Hailey CPU > Max CPU

#.59 Copyright 2006 Kyle Hailey Idle Database  Value of proving the database is Idle  It’s the Databases Fault  How many times do you hear that?  Database Idle  No load on database  Database “performance” is fine  Under utilized  Problem lies elsewhere  Saved me time and stress many times

#.60 Copyright 2006 Kyle Hailey Idle Database – Perf Page

#.61 Copyright 2006 Kyle Hailey Idle Database – Top Activity

#.62 Copyright 2006 Kyle Hailey Harnessing AAS  Statspack/AWR report  Lacking  Statspack free and any version  AWR automatically installed in10g  Both  Needs numerical massaging  Lacking detailed SQL and Session info  OEM 10g  Clearest, easiest, most powerful  Overview  Load chart - AAS  Breakdown –  Top SQL and Session  Or Service, Module, Action, Objects, Files  Drill downs  SQL  Session  Plus  ADDM – Automatic Database Diagnostics Monitor  SQL Advisor – SQL tuning advice

#.63 Copyright 2006 Kyle Hailey OEM 10g Perf Pages DB Home Performance Top Activity SQL Session

#.64 Copyright 2006 Kyle Hailey In summary  AAS is simple and Powerful  AAS’s components are even more powerful  CPU  WAIT  Value over Time  Use # of CPUs as a yardstick  Know your application load profile to see anomalies  Monitor AAS with OEM 10g