#.1 Average Active Sessions (AAS) The Golden Metric ? Kyle Hailey

#.2 2 2/8/2016 Copyright 2006 Kyle Hailey In this Session 1.AAS  Single Metric  Shows DB Performance  Two methods to calculate  Sampling  Time statistics 2.Yardstick  Max CPU  CPU Count  To measure AAS against 3.Subcomponents  CPU  Waits  Time series

#.3 3 2/8/2016 Database Performance  How quick can you find  Bottleneck in DB  If DB is idle  Current DB Load  what is DB Load ?  What do you use?  Statspack/AWR  V$active_session_history  Alerts  what do you alert on ? what the *!####!*!*? is the database doing ?!

#.4 Active Session Copyright 2006 Kyle Hailey Database Machine SQL* Plus Shadow Process SQL* Plus Shadow Process Application Server Machine Shadow Process Shadow Process Shadow Process Application SGA

#.5 idle Active Session Copyright 2006 Kyle Hailey SQL*Plus (ie application) idle Have a coffee waiting Typing waiting typing Query 1 Query 2 Query 3 Database (shadow process) Time Active Shadow Process

#.6 Active Session Database User 1 User 2 User 3 User 4 Active Sessions = Graph represents # of sessions active, but also represents amount of time active in the database

#.7 Active Session Copyright 2006 Kyle Hailey Active Sessions Users Waiting For Every Active Session there is a user (or application) waiting

#.8 Measuring Active Sessions Copyright 2006 Kyle Hailey Fast query run often Fast query run rarely Slow query If happens a lot or for long … we’ll catch it, guaranteed User 1 User 2 User 3 Sampling Every Second...

#.9 Copyright 2006 Kyle Hailey Sampling is like taking Pictures

#.10 idle Active Session Copyright 2006 Kyle Hailey idleidle idle Query 1 Query 2 Query 3 Database To execute a query, some CPU will be used but we also might spend time waiting for IO or on waiting for concurrency resources like latches CPU IO Wait idleidleidle idle Database Query 1 Query 2 Query 3 Activity can thus further be broken down into the type of activity: CPU, IO or WAIT Work LatencyContention

#.11 Session 1 Session 2 Session 3 Session 4 TIME Graphical ASH

#.12 Copyright 2006 Kyle Hailey Graph of User States

#.13 Copyright 2006 Kyle Hailey One Second Graph

#.14 Copyright 2006 Kyle Hailey 15 Second Averages

#.15 Copyright 2006 Kyle Hailey Maximum CPU Line

#.16 Copyright 2006 Kyle Hailey Idle Users

#.17 Copyright 2006 Kyle Hailey OEM Perf Page

#.18 AAS in OEM LOAD AAS

#.19 Copyright 2006 Kyle Hailey The Power ASH gives AAS DB Home Performance Top Activity Based on ASH Based on TIME (events Statistics) AAS=db time/elapsed time AAS=count active users /samples

#.20 Copyright 2006 Kyle Hailey DB TIME = area under the curve DB Time = Δt DB Time = active sessions(ti) * Δt n Σ 0 Height = # of Sessions Width = seconds DB Time Area under curve = DB Time DB Time = sum of active time in database

# /8/2016 Copyright 2006 Kyle Hailey AAS Sources 1.Manually from  v$sysstat (9i : v$system_event ) 2.Statspack  Need several calculations 3. AWR  One calculation 4.OEM 10g  Directly displayed AAS = DB Time/Elapsed Time

#.22 Copyright 2006 Kyle Hailey 1. Manually DB Time (DBT) = Time Spent in Database DB TIME (10g) = DB TIME (9i) = Select sum(time_waited) from v$system_event where event not in (... idle events …); + Select value from v$sysstat where name = ‘CPU used by this session’; Select sum(time_waited) from v$system_event where event not in (... idle events …); + Select value from v$sysstat where name = ‘CPU used by this session’; select value from v$sysstat where name = ‘DB time’; select value from v$sysstat where name = ‘DB time’; ‘DB time’ still need to take delta values Note: stats$idle_event : 70 v$event_name.wait_class=‘Idle’ :62 AAS = DB TIME / Elapsed Time

# /8/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS  Look for  Elapsed Time  Top 5 Timed Events  Start at line 52 of about 1300

# /8/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS  Elapsed Time  Look at Top 5 Timed Events Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host LABSF labsf NO labsfr Snap Id Snap Time Sessions Curs/Sess Begin Snap: 1 03-Apr-06 12:34: End Snap: 2 03-Apr-06 12:34: Elapsed: 1.00 (mins) STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host LABSF labsf NO labsfr Snap Id Snap Time Sessions Curs/Sess Begin Snap: 1 03-Apr-06 12:34: End Snap: 2 03-Apr-06 12:34: Elapsed: 1.00 (mins)

# /8/2016 Copyright 2006 Kyle Hailey 2. Statspack AAS  DBTIME= CPU + WAITS  CPU = 32  WAITS = = 278 secs DBTIME=320  Elapsed Time = 60 secs  320 secs / 60 secs AAS = 5.1 Top 5 Timed Events Event Time (s) buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5 Top 5 Timed Events Event Time (s) buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5

# /8/ AWR Report  AAS = DB Time/Elapsed Time  23.56/59.66 = 0.39 AAS= 0.39

# /8/ OEM 10g AAS = ~0.75 OEM 10g

# /8/2016 Copyright 2006 Kyle Hailey Got AAS, Now What ? We Need one more item: CPU Count  # of CPUs available on System  Shared with other applications  Need to track CPU used on the system as well  On dual & quad cores, lower the CPU count  Represents max active sessions that can do work

# /8/2016 Copyright 2006 Kyle Hailey CPU Count  # of CPUs available in  Statspack 10g  AWR report  OEM 10g  Statspack 9i # of CPUs missing # of CPUs SQLPLUS> show parameters cpu_count NAME VALUE cpu_count 2 SQLPLUS> show parameters cpu_count NAME VALUE cpu_count 2

# /8/2016 Copyright 2006 Kyle Hailey What’s the DB Doing?! It’s 2am … your manager calls Whip out the stethoscope: AAS what the *!####!*!*? is the database doing ?!

# /8/2016 Copyright 2006 Kyle Hailey AAS Formulas Use CPU count as yardstick: AAS < 1 Database is not blocked AAS ~= 0 Database basically idle Problems are in the APP not DB AAS < # of CPUs CPU available Are any single sessions 100% active?  AAS > # of CPUs Could have performance problems  AAS >> # of CPUS There is a bottleneck Ideal world – one database solution track CPU at OS AAS > 1 still want to know if a single user is 100% active

# /8/2016 Copyright 2006 Kyle Hailey Available CPU vs AAS AAS far above available CPU => problem AAS = 5.1 # of CPU = 2 AAS = 0.39 # of CPU = 2 AAS < 1, database is fine Statspack  DBTIME= CPU + WAITS  CPU = 32  WAITS = = 278 secs DBTIME=320  Elapsed Time = 60 secs  320 secs / 60 secs AAS = 5.1 AWR Report AAS = 0.75 # of CPU = 2 AAS < 1, database is fine

# /8/2016 Copyright 2006 Kyle Hailey Going Farther with AAS  AAS can tell you a lot  But it’s components tell you much more  To go farther need the components of AAS 1. CPU 2. Wait 3. Value over time  Only OEM 10g shows the value over time, Statspack and AWR are aggregated over the snapshot period

#.34 Copyright 2006 Kyle Hailey EM DB Home Page

#.35 Copyright 2006 Kyle Hailey OEM 10g Perf Pages DB Home Performance AAS Point in Time AAS over Time

# /8/2016 AAS Components : OEM 10g  OEM 10g Performance Page Available CPU AAS: CPU + WAIT Real CPU available: Max CPU - non instance CPU

# /8/2016 OEM 10g Relax Get to Work! Looks OK But …

#.38 Copyright 2006 Kyle Hailey Idle Database – Perf Page  Value of proving the database is Idle  It’s the Databases Fault  How many times do you hear that?  Database Idle  No load on database  Database “performance” is fine  Under utilized  Problem lies elsewhere  Saved me time and stress many times  And weeks of debate about where the problem is coming from

#.39 Copyright 2006 Kyle Hailey More than AAS and #CPU Knowing your DB Profile

#.40 Copyright 2006 Kyle Hailey When to Tune 1.Machine a) CPU  Response times skewed  100% CPU might be fine  Users wait in queue (run queue) => machine underpowered b) Memory  Paging  Wait times skewed (ex : latch free)  Erratic response times ( ex : ls ) 2.Oracle 1) Waits > CPU ?  tune waits 2) CPU > 100% ?  tune top CPU SQL 3) Else  It’s the application Oracle Load (AAS) Top Session Top Wait Top SQL SQL Detail Session Detail File Detail Object Detail Wait Detail Host AAS > #CPU AAS > 1 Waits > CPU CPU > Waits CPU Memory

# /8/2016 Limited Analysis  What if you find a problem ?  Of the 800 waits which in order to solve most need to know  What SQL  Which sessions  Values of P1, P2 and P3  Statspack and AWR fail AAS = DB TIME / Elapsed Time But there is another way …

# /8/2016 Copyright 2006 Kyle Hailey AAS based on ASH  ASH - Active Session History  v$active_session_history  Samples sessions once a second  AAS = count(*) / elapsed_seconds  A statistical approximation, but surprisingly close  ASH data source empowers drilldowns  Top Sql  Top Waits  Details p1,p2,p3 and more

#.43 Copyright 2006 Kyle Hailey ASH vs Statistics  Statistics  are more expensive  have lag time  lack clear identification of culprits

#.44 Copyright 2006 Kyle Hailey Statistic Lag Time Statistics Samples (ASH) Slight Lags

#.45 Copyright 2006 Kyle Hailey CPU Lag Problem  ASH is the only way to see CPU usage realtime  V$sysstat reports CPU but  is only updated at the end of the call.  Long calls look deceiving like no CPU is being used  Time Model also reports CPU  Updated quicker

#.46 Copyright 2006 Kyle Hailey CPU in ASH vs Stats

# /8/ OEM 10g : Top Activity Top Activity Based on ASH Enables Drilldowns Top SQL Top Session Drill into a session Stats Raw waits Open cursors General info Drill into a SQL Stats and text Users executing Explain plan Tuning options

#.48 Copyright 2006 Kyle Hailey Top Activity : Based on ASH missing ThanksToASH

#.49 Copyright 2006 Kyle Hailey AAS – %Session Time Issue Whenever AAS > 1 I want to know if any one session is 100% active Unfortunately OEM isn’t set up for this (ASHMON, my free tool, is )

#.50 Copyright 2006 Kyle Hailey Top Activity: ASH Sessions Many Users Active On Performance Page, no way to tell how many users But Top Activity Page fixes that

#.51 Copyright 2006 Kyle Hailey Top Activity: ASH Sessions Two Users Active Shown in % DB Time Missing % Session Time

#.52 Copyright 2006 Kyle Hailey OEM 10g Perf Pages DB Home Performance Top Activity SQL Session Copyright 2006 Kyle Hailey Top Activity SQL Session

#.53 Copyright 2006 Kyle Hailey Session : ASH Activity

#.54 Copyright 2006 Kyle Hailey SQL : ASH Activity

# /8/2016 Copyright 2006 Kyle Hailey AAS from ASH 1.ASHRPT  Based entirely on v$active_session_history  Exec ASH_REPORT_TEXT/HTML select * from table (dbms_workload_repository.ash_report_text( (select dbid from v$database), (select dbid from v$database), 1, 1, sysdate – 1/24, sysdate – 1/24, sysdate )) ; sysdate )) ;

# /8/2016 Copyright 2006 Kyle Hailey 1. ASHRPT ASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host TESTDB testdb NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: Average Active Sessions: Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class % Activity Sessions CPU + Wait for CPU CPU enq: TX - row lock contention Application buffer busy waits Concurrency latch: cache buffers chains Concurrency ASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host TESTDB testdb NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: Average Active Sessions: Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class % Activity Sessions CPU + Wait for CPU CPU enq: TX - row lock contention Application buffer busy waits Concurrency latch: cache buffers chains Concurrency

# /8/2016 Copyright 2006 Kyle Hailey 1. ASH RPT 1. ASH RPT 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***

# /8/ ASHRPT over Time  Waits over Time  Not in AAS  Difficult but better than nothing Compare to …

# /8/ Custom Scripts  Hate Graphics ?  Query v$active_session_history directly  Join to dba_hist_active_sess_history for week of data  act.sql  Like top 5 timed events  Aveact.sql  Charts with text AAS by hour (15 minute, minute, etc)  Aveactn.sql  Ditto, with top 2 wait events per bucket  Following Scripts Available on 

# /8/2016 Copyright 2006 Kyle Hailey 3. Custom Analysis Begin Time : :04:48 Analysis End Time : :19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess latch free log buffer space buffer busy waits db file scattered read library cache pin log file sync ON CPU enqueue db file sequential read sum Analysis Begin Time : :04:48 Analysis End Time : :19:45 Start time, mins ago: 15 Request Duration : 15 Collections : 528 Data Values : 3327 Elapsed Time: 15 mins WAIT_EVENT CNT % Active Ave_Act_Sess latch free log buffer space buffer busy waits db file scattered read library cache pin log file sync ON CPU enqueue db file sequential read sum 6.30

# /8/2016 Copyright 2006 Kyle Hailey 3. Custom TM NPTS AVEACT GRAPH CPU WAITS AUG 13:00: AUG 14:00: AUG 15:00: AUG 16:00: AUG 17:00: AUG 18:00: AUG 19:00: AUG 20:00: AUG 21:00: AUG 22:00: AUG 23:00: AUG 00:00: AUG 01:00: AUG 02:00: AUG 03:00: AUG 04:00: AUG 05:00: AUG 06:00: TM NPTS AVEACT GRAPH CPU WAITS AUG 13:00: AUG 14:00: AUG 15:00: AUG 16:00: AUG 17:00: AUG 18:00: AUG 19:00: AUG 20:00: AUG 21:00: AUG 22:00: AUG 23:00: AUG 00:00: AUG 01:00: AUG 02:00: AUG 03:00: AUG 04:00: AUG 05:00: AUG 06:00:

#.62 Copyright 2006 Kyle HaileyAveact.sql

#.63Aveact.sql Copyright 2006 Kyle Hailey “-” = WAIT “+” = CPU which waits ? -> aveactn.sql

#.64 Aveactn.sql Copyright 2006 Kyle Hailey

# /8/2016 Copyright 2006 Kyle HaileySummary  AAS: Two Sources 1. v$system_event & v$sysstat  Performance page in OEM  Statspack, awr report  Indirect – based on time converted to AAS  Accurate  Lags (especially CPU)  Limits analysis 2. v$active_session_history  Top activity in OEM  ashrpt  Direct measure of AAS  Real time  Approximation  ***Allows drilldowns***  Tuning:  Machine first – CPU, Memory  Oracle  AAS ~= 0 Idle  AAS > 1 : possible sessions blocked  AAS > # CPU : bottleneck Oracle Load (AAS) Top Session Top Wait Top SQL SQL Detail Session Detail File Detail Object Detail Wait Detail Host AAS > #CPU AAS > 1 Waits > CPU CPU > Waits CPU Memory ADDM SQL Tuning Advisor

#.66Q1 Copyright 2006 Kyle Hailey What is the easiest way to find the load on the database a. top 5 timed events b. Average Active Sessions and CPU count c. The transaction rate d. The commit rate b. Average Active Sessions takes into account all the major criteria in Oracle performance - cpu usage, wait time, elapsed time others: a. lacks the elapsed time c & d - the amount of load created by commits and transactions varies drastically between database to database and even between different loads on a database during the day

#.67Q2 Copyright 2006 Kyle Hailey Average active sessions, the measurement in the performance chart in OEM 10g can be calculated by which formula(s) a. db time / elapsed time b. wait time / elapsed time c. count of rows in v$active_session_history over an interval/ number of samples in interval d. count of rows in v$active_session_history / seconds of elapsed time answer a,c,d c, d are equivalent because ASH samples once a second others b doesn't work because it is missing CPU time

#.68Q3 Copyright 2006 Kyle Hailey If Average Active Session is less than one, we can say a. the database is using less than 100% CPU on machine b. the database is using 100% CPU c. No session is completely blocked d. Database performance should be acceptable answer a, c and d

#.69Q4 Copyright 2006 Kyle Hailey An Average Active Session value of zero means a. the database is down b. the database is hung c. the database is idle d. the database is overloaded answer c

#.70 Q5 Copyright 2006 Kyle Hailey CPU is updated in v$sysstat a. when call finishes b. every second c. every five seconds d. in real time a - which can be a problem in monitoring tools if the call last a long time showing no CPU usage until the call finishes, then reporting an unrealistic spike of CPU when the call finishes