1. IO Waits Kyle Hailey Kyle_hailey@hotmail.com http://perfvision.com/ftp/sb

2. #.2 Copyright 2006 Kyle Hailey Waits Covered in this Section 1.db file sequential read 2.db file scattered read 3.db file parallel read 4.read by other session 5.direct path read 6.direct path write 7.direct path read temp 8.direct path write temp 9.direct path write (lob) 10.file open 11.file identify 12.Local write wait 13.data file init write 14.sort segment request Standard I/O Direct I/O Special Cases

3. #.3 Copyright 2006 Kyle Hailey Buffer Cache REDO Lib Cache Buffer Cache Locks Network I/O

4. #.4 Copyright 2006 Kyle Hailey IO Waits Data Files DBWR Buffer Cache User1 SGA Library Cache Log Buffer

5. #.5 Copyright 2006 Kyle Hailey Standard Table/Index IO 1.db file sequential read  Single block read 2.db file scattered read  Multi block read 3.db file parallel read  Non-contiguous multi block read 4.read by other session  Wait for another session to do the io

6. #.6 Copyright 2006 Kyle Hailey db file sequential read ShadowProcess 1) 1)Search Buffer Cache for the block by rowid 2) 2)Fails 3) 3)Reads of disk File Block Buffer Cache   The top wait   Single Block Read   Data via Index and/or Rowid   Rollback Select * from emp where empno = 9999; Note: “sequential” means A sequence like a rowid Index on emp.empno

7. #.7 Copyright 2006 Kyle Hailey db file scattered readShadowProcess 1) 1)Search Buffer Cache for the blocks 2) 2)Fails 3) 3)Reads off disk File Block Multi Block Read Count Buffer Cache   Multi Block Read   Full Table Scan   Index Fast Full Scans SQL> select * from all_objects; Note: Scattered Means Blocks are read and scattered throughout buffer cache

8. #.8 Copyright 2006 Kyle Hailey db file parallel read Shadow Process 1) 1)Search Buffer Cache for the blocks 2) 2)Fails 3) 3)Reads that block off Disk Buffer Cache   Process issues multiple single block reads in parallel   Documentation says only for recovery   But seems to happen for normal read ops   Async Call – wait for all reads to complete   Examples   Not contiguous multi block read   Index full or range scan   Where values in Select * from emp where Empno in (1,2,3,4,5,6); Select * from emp where Empno in (1,2,3,4,5,6);

9. #.9 Copyright 2006 Kyle Hailey Read by other Session 1.Block not found in cache 2.Read block from disk 3.Found other session already reading block from disk 4.Wait for the other session to finish IO S1 Buffer Cache S2 Multiple sessions reading the same data that requires IO Ex) to users doing a full table scan at the same time

10. #.10 Copyright 2006 Kyle Hailey Db file Sequential Read: Statspack Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time ------------------------ ------------ ----------- -------- db file scattered read 7,549 5 52.26 CPU time 2 23.29 db file sequential read 884 2 19.11 control file sequential read 154 0 2.68 control file parallel write 255 0 1.87 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time ------------------------ ------------ ----------- -------- db file scattered read 7,549 5 52.26 CPU time 2 23.29 db file sequential read 884 2 19.11 control file sequential read 154 0 2.68 control file parallel write 255 0 1.87 What SQL ? What Object ?

11. #.11 Copyright 2006 Kyle Hailey Db file sequential read

12. #.12 Copyright 2006 Kyle Hailey db file scattered Read

13. #.13 Copyright 2006 Kyle Hailey Read by other session

14. #.14 Copyright 2006 Kyle Hailey IO – Further Investigation select parameter1, parameter2, parameter3 from v$event_name NAME P1 P2 P3 ----------------------- ----------- --------- -------- read by other session file# block# class# db file sequential read file# block# blocks db file scattered read file# block# blocks direct path read file number first dba block cnt direct path read temp file number first dba block cnt direct path write file number first dba block cnt direct path write temp file number first dba block cnt db file parallel read files blocks requests P1 and P2 are the same, file#, block# We can use P1, P2, P3 from ASH to get more info Exception: db file parallel read – p1,p2,p3 not useful

15. #.15 Copyright 2006 Kyle Hailey ASH – sequential reads select event, event, ash.p3, ash.p3, o.object_name objn, o.object_name objn, o.object_type otype, o.object_type otype, CURRENT_FILE# filen, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, CURRENT_BLOCK# blockn, ash.SQL_ID ash.SQL_ID from v$active_session_history ash, all_objects o all_objects o where event like 'db file sequential read' and o.object_id (+)= ash.CURRENT_OBJ# and o.object_id (+)= ash.CURRENT_OBJ# order by sample_time; EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID ----------------------- -- ------------------------- ----- ----- ------ ------------- db file sequential read 1 49890 MGMT_METRICS_1HOUR_ INDEX 3 41737 db file sequential read 1 50908 MGMT_DB_SGA_ECM TABLE 3 28489 35at43xqj7bs0 db file sequential read 1 55303 TOTO TABLE 1 60218 7xftj55rvjw9s

16. #.16 Copyright 2006 Kyle Hailey ASH – scattered reads select event, event, ash.p3, ash.p3, o.object_name objn, o.object_name objn, o.object_type otype, o.object_type otype, CURRENT_FILE# filen, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, CURRENT_BLOCK# blockn, ash.SQL_ID ash.SQL_ID from v$active_session_history ash, all_objects o all_objects o where event like 'db file scattered read' and o.object_id (+)= ash.CURRENT_OBJ# and o.object_id (+)= ash.CURRENT_OBJ# order by sample_time; EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID ---------------------- -- ------------------------- ----- ----- ------ ------------- db file scattered read 8 9078 WRH$_SYSMETRIC_HISTO TABLE 3 52363 db file scattered read 5 8781 WRI$_ALERT_HISTORY TABLE 3 2676 db file scattered read 8 57 OBJAUTH$ TABLE 1 33993 3t2mk1prj24hu

17. #.17 IO by TBS & File select io.cnt, round(io.cnt/(&v_minutes*60),2) aas, round(io.cnt/(&v_minutes*60),2) aas, io.event, io.event, io.p1 p1, io.p1 p1, f.tablespace_name f.tablespace_name from ( select count(*) cnt, count(*) cnt, substr(event,0,25) event, substr(event,0,25) event, ash.p1 p1 ash.p1 p1 from v$active_session_history ash from v$active_session_history ash where ( event like 'db file s%' or event like 'direct%' ) where ( event like 'db file s%' or event like 'direct%' ) and sample_time > sysdate - &v_minutes/(60*24) and sample_time > sysdate - &v_minutes/(60*24) group by group by event, event, ash.p1 ash.p1 ) io, ) io, dba_data_files f dba_data_files fwhere f.file_id = io.p1 f.file_id = io.p1 Order by io.cnt ; CNT AAS EVENT P1 TABLESPACE CNT AAS EVENT P1 TABLESPACE ---- --- ----------------------- --- ---------- 1.00 db file sequential read 1 SYSTEM 1.00 db file sequential read 1 SYSTEM 2.00 db file sequential read 3 SYSAUX 2.00 db file sequential read 3 SYSAUX 38.06 db file sequential read 6 SOE 38.06 db file sequential read 6 SOE 179.30 db file sequential read 7 SOEINDEX 179.30 db file sequential read 7 SOEINDEX

18. #.18 IO by Objectselect count(*) cnt, count(*) cnt, CURRENT_OBJ#||' '||o.object_name objn, CURRENT_OBJ#||' '||o.object_name objn, o.object_type otype o.object_type otype from v$active_session_history ash, all_objects o all_objects o where ( event like 'db file s%' or event like 'direct%' ) and o.object_id (+)= ash.CURRENT_OBJ# and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - &1/(60*24) and sample_time > sysdate - &1/(60*24) and session_state='WAITING' and session_state='WAITING' group by CURRENT_OBJ#, o.object_name, CURRENT_OBJ#, o.object_name, o.object_type o.object_type Order by count(*) CNT AAS OBJN OTYPE CNT AAS OBJN OTYPE ---- ------- ------------------------- --------------- 79.00 52949 ORDER_ITEMS TABLE PARTITION 79.00 52949 ORDER_ITEMS TABLE PARTITION 97.00 -1 97.00 -1 130.00 53117 ORD_STATUS_IX INDEX 130.00 53117 ORD_STATUS_IX INDEX 498.01 53120 CUST_EMAIL_IX INDEX 498.01 53120 CUST_EMAIL_IX INDEX 512.01 0 512.01 0 1632.03 53112 ITEM_ORDER_IX INDEX

19. #.19 IO by Object/tbs/wait Select io.cnt cnt, io.aas aas, io.aas aas, io.event event, io.event event, substr(io.obj,1,20) obj, substr(io.obj,1,20) obj, io.p1 p1, io.p1 p1, f.tablespace_name tablespace_name f.tablespace_name tablespace_name from ( select count(*) cnt, count(*) cnt, round(count(*)/(&v_minutes*60),2) aas, round(count(*)/(&v_minutes*60),2) aas, substr(event,0,15) event, substr(event,0,15) event, nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj, nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj, ash.p1, ash.p1, o.object_type otype o.object_type otype from v$active_session_history ash, from v$active_session_history ash, all_objects o all_objects o where ( event like 'db file s%' or event like 'direct%' ) where ( event like 'db file s%' or event like 'direct%' ) and o.object_id (+)= ash.CURRENT_OBJ# and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - &v_minutes/(60*24) and sample_time > sysdate - &v_minutes/(60*24) group by group by substr(event,0,15), substr(event,0,15), CURRENT_OBJ#, o.object_name, CURRENT_OBJ#, o.object_name, o.object_type, o.object_type, ash.p1 ) io, ash.p1 ) io, dba_data_files f dba_data_files f where f.file_id = io.p1 Order by io.cnt; CNT AAS EVENT OBJ P1 TABLESPACE_NAME --- --- --------------- ----------------- --- --------------- 1.02 db file sequent ORDER_ITEMS_PK 7 SOEINDEX 1.02 db file sequent ORDER_ITEMS_PK 7 SOEINDEX 1.02 db file sequent 0 3 SYSAUX 1.02 db file sequent 0 3 SYSAUX 2.03 db file scatter WAIT_OBJECTS 1 SYSTEM 2.03 db file scatter WAIT_OBJECTS 1 SYSTEM 2.03 db file sequent ORDER_ITEMS 6 SOE 2.03 db file sequent ORDER_ITEMS 6 SOE 3.05 db file sequent CUST_EMAIL_IX 7 SOEINDEX 3.05 db file sequent CUST_EMAIL_IX 7 SOEINDEX 4.07 db file sequent CUST_LNAME_IX 7 SOEINDEX 4.07 db file sequent CUST_LNAME_IX 7 SOEINDEX 6.10 db file sequent ORD_ORDER_DATE_IX 7 SOEINDEX 6.10 db file sequent ORD_ORDER_DATE_IX 7 SOEINDEX 13.22 db file sequent ITEM_ORDER_IX 7 SOEINDEX 13.22 db file sequent ITEM_ORDER_IX 7 SOEINDEX 25.42 db file sequent 0 2 UNDOTBS1 25.42 db file sequent 0 2 UNDOTBS1

20. #.20 IO by SQLselect sum(cnt) over ( partition by io.sql_id order by sql_id ) tcnt, sum(cnt) over ( partition by io.sql_id order by sql_id ) tcnt, io.sql_id, io.sql_id, io.cnt cnt, io.cnt cnt, io.aas aas, io.aas aas, io.objn objn, io.objn objn, io.obj obj, io.obj obj, io.p1 p1, io.p1 p1, f.tablespace_name tablespace_name f.tablespace_name tablespace_namefrom( select select sql_id, sql_id, count(*) cnt, count(*) cnt, round(count(*)/(&v_minutes*60),2) aas, round(count(*)/(&v_minutes*60),2) aas, CURRENT_OBJ# objn, CURRENT_OBJ# objn, nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj, nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj, o.object_type otype, o.object_type otype, ash.p1 ash.p1 from v$active_session_history ash from v$active_session_history ash,all_objects o,all_objects o where ( event like 'db file s%' or event like 'direct%' ) where ( event like 'db file s%' or event like 'direct%' ) and o.object_id (+)= ash.CURRENT_OBJ# and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - &v_minutes/(60*24) and sample_time > sysdate - &v_minutes/(60*24) group by group by CURRENT_OBJ#, CURRENT_OBJ#, o.object_name, o.object_name, o.object_type, o.object_type, ash.p1, ash.p1, sql_id sql_id ) io, dba_data_files f dba_data_files fwhere f.file_id = io.p1 f.file_id = io.p1 Order by tcnt, io.sql_id, io.cnt TCNT SQL_ID CNT AAS OBJN OBJ P1 TABLESPAC ---- ------------- --- --- ----- --------------- -- --------- - 30 0yas01u2p9ch4 1.01 53113 ITEM_PRODUCT_IX 7 SOEINDEX 1.01 53079 ORDER_ITEMS_UK 7 SOEINDEX 28.16 53112 ITEM_ORDER_IX 7 SOEINDEX 58 6v6gm0fd1rgrz 4.02 54533 WAIT_OBJECTS 1 SYSTEM 54.30 0 0 2 UNDOTBS1

21. #.21 Copyright 2006 Kyle Hailey Missing Object IDs  dba_extents – get object name  Dba_extents is notoriously slow  Options  Create a table as select – use table  recreate as object extents change  Catch blocks in x$bh when waits occur select segment_name, segment_type from dba_extents where file_id = P1 and P2 between block_id and block_id + blocks – 1; select segment_name, segment_type from dba_extents where file_id = P1 and P2 between block_id and block_id + blocks – 1;

22. #.22 Copy of dba_extents Enter value for file: 3 Enter value for block: 6619 OWNER SEGMENT_NAME SEGMENT_TYPE --------------- -------------------- ------------------ WMSYS LOG_TAB_PK INDEX Elapsed: 00:00:41.25 create table myextents as select * from dba_extents Table created. Elapsed: 00:01:25.73 CNT OWN SEGMENT_NAME SEGMENT_TYPE --- ---- ---------------- ------------ 11 SYS SMON_SCN_TO_TIME CLUSTER 11 SYS SMON_SCN_TO_TIME CLUSTER 993 SYS _SYSSMU7$ TYPE2 UNDO 993 SYS _SYSSMU7$ TYPE2 UNDO 150 rows selected. Elapsed: 00:00:01.03

23. #.23 Copyright 2006 Kyle Hailey IO Solutions 1.Check average read times per file  Should be between 5-20 ms  Data in Statspack under “File IO Stats” 2.Check Cache buffer Hit ratio  Check db_cache_advice 9i and higher  Data in Statspack under “Buffer Pool Advisory”  Want to optimize data caching 3.Tune High IO SQL

24. #.24 Copyright 2006 Kyle Hailey Step 1a: Ave Read Time File IO Stats DB/Inst:labsf03 Snaps: 1-2 Tablespace Filename ------------------------ ---------------------------------------------------- Av Mx Av Av Rd Rd Av Av Buffer BufWt Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms) ---------- ---- ------- ----- --- ------- ------------ -------- ---------- -- SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf 445 15 0.4 16 1.0 1,157 39 2,744 93.3 USERS /u01/app/oracle/oradata/labsf03/users01.dbf 223 7 0.5 ### 1.0 9,725 324 4 100.0 File IO Stats DB/Inst:labsf03 Snaps: 1-2 Tablespace Filename ------------------------ ---------------------------------------------------- Av Mx Av Av Rd Rd Av Av Buffer BufWt Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms) ---------- ---- ------- ----- --- ------- ------------ -------- ---------- -- SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf 445 15 0.4 16 1.0 1,157 39 2,744 93.3 USERS /u01/app/oracle/oradata/labsf03/users01.dbf 223 7 0.5 ### 1.0 9,725 324 4 100.0 IO should be under 20ms Fastest possible is around 7ms Faster speeds are due to read caching

25. #.25 Copyright 2006 Kyle Hailey Step 1c: Ave Read Time IO should be under 20ms Fastest possible is around 7ms Faster speeds are due to read caching select to_char(begin_interval_time,'yyyy-mm-dd hh24:mi') snap_time, file#, readtim/nullif(phyrds,0) avg_read_ms, phyrds from DBA_HIST_FILESTATXS f, dba_hist_snapshot s where f.snap_id=s.snap_id ; select to_char(begin_interval_time,'yyyy-mm-dd hh24:mi') snap_time, file#, readtim/nullif(phyrds,0) avg_read_ms, phyrds from DBA_HIST_FILESTATXS f, dba_hist_snapshot s where f.snap_id=s.snap_id ; SNAP_TIME FILE# AVG_READ_MS PHYRDS ---------------- ---------- ----------- ---- 2008-01-05 12:00 9 36.67 39 2008-01-05 12:00 10 32.31 39 2008-01-05 13:00 1 11.63 178224 2008-01-05 13:00 2 56.37 2014 2008-01-05 13:00 3 17.38 50668 2008-01-05 13:00 4 9.39 565321 2008-01-05 13:00 5 38.78 41 2008-01-05 13:00 6 28.29 41 2008-01-05 13:00 7 27.44 39 2008-01-05 13:00 8 42.56 39 2008-01-05 13:00 9 36.67 39 2008-01-05 13:00 10 32.31 39 SNAP_TIME FILE# AVG_READ_MS PHYRDS ---------------- ---------- ----------- ---- 2008-01-05 12:00 9 36.67 39 2008-01-05 12:00 10 32.31 39 2008-01-05 13:00 1 11.63 178224 2008-01-05 13:00 2 56.37 2014 2008-01-05 13:00 3 17.38 50668 2008-01-05 13:00 4 9.39 565321 2008-01-05 13:00 5 38.78 41 2008-01-05 13:00 6 28.29 41 2008-01-05 13:00 7 27.44 39 2008-01-05 13:00 8 42.56 39 2008-01-05 13:00 9 36.67 39 2008-01-05 13:00 10 32.31 39

26. #.26 Copyright 2006 Kyle Hailey Step 1c: Ave Read Time IO should be under 20ms Fastest possible is around 7ms Faster speeds are due to read caching select to_char(begin_time,'yyyy-mm-dd hh24:mi') begin_time, file_id fid, average_read_time *10 avgrd_ms, average_write_time *10 avgwr_ms, physical_reads pr, physical_writes pw from V$FILEMETRIC_HISTORY f order by begin_time; select to_char(begin_time,'yyyy-mm-dd hh24:mi') begin_time, file_id fid, average_read_time *10 avgrd_ms, average_write_time *10 avgwr_ms, physical_reads pr, physical_writes pw from V$FILEMETRIC_HISTORY f order by begin_time; BEGIN_TIME FID AVGRD_MS AVGWR_MS PR PW ---------------- ---- -------- -------- ---------- ---------- 2008-01-30 12:25 1 5.2 2.5 100 24 2008-01-30 12:25 5 80.0 20.0 2 2 2008-01-30 12:25 2 36.7 2.0 3 218 2008-01-30 12:25 4 22.8 1.8 89 2754 2008-01-30 12:25 3 22.9 2.6 14 47 BEGIN_TIME FID AVGRD_MS AVGWR_MS PR PW ---------------- ---- -------- -------- ---------- ---------- 2008-01-30 12:25 1 5.2 2.5 100 24 2008-01-30 12:25 5 80.0 20.0 2 2 2008-01-30 12:25 2 36.7 2.0 3 218 2008-01-30 12:25 4 22.8 1.8 89 2754 2008-01-30 12:25 3 22.9 2.6 14 47

27. #.27 Copyright 2006 Kyle Hailey Step 2 : Buffer Pool Advisory Buffer Pool Advisory Size for Size Buffers for Read Estimated P Est (M) Factor Estimate Factor Physical Reads --- -------- ------ ------------ ------ -------------- D 56.1 6,986 2.3 58,928 D 112.2 13,972 1.6 42,043 D 224.4 27,944 1.0 25,772 D 336.6 41,916 1.0 25,715 D 448.8 55,888 1.0 25,715 D 596 1.0 74,351 1.0 25,715 D 728 1.2 90,818 1.0 25,715 D 840 1.4 104,790 1.0 25,715 D 952 1.6 118,762 1.0 25,715 D 1,064 1.8 132,734 1.0 25,715 Buffer Pool Advisory Size for Size Buffers for Read Estimated P Est (M) Factor Estimate Factor Physical Reads --- -------- ------ ------------ ------ -------------- D 56.1 6,986 2.3 58,928 D 112.2 13,972 1.6 42,043 D 224.4 27,944 1.0 25,772 D 336.6 41,916 1.0 25,715 D 448.8 55,888 1.0 25,715 D 596 1.0 74,351 1.0 25,715 D 728 1.2 90,818 1.0 25,715 D 840 1.4 104,790 1.0 25,715 D 952 1.6 118,762 1.0 25,715 D 1,064 1.8 132,734 1.0 25,715

28. #.28 Copyright 2006 Kyle Hailey Step 3: Find Top I/O SQL select count(*),sql_id, event from v$active_session_history where event in ('db file sequential read', 'db file scattered read', 'db file parallel read') group by sql_id, event order by count(*); select count(*),sql_id, event from v$active_session_history where event in ('db file sequential read', 'db file scattered read', 'db file parallel read') group by sql_id, event order by count(*); COUNT(*) SQL_ID EVENT --------- ------------- ------------------------ 10 8hk7xvhua40va db file sequential read 335 3hatpjzrqvfn7 db file sequential read 343 0uuqgjq7k12nf db file sequential read 481 db file scattered read 549 75621g9y3xmvd db file sequential read 1158 db file sequential read 1311 0bzhqhhj9mpaa db file sequential read 1523 41zu158rqf4kf db file sequential read 1757 0yas01u2p9ch4 db file sequential read COUNT(*) SQL_ID EVENT --------- ------------- ------------------------ 10 8hk7xvhua40va db file sequential read 335 3hatpjzrqvfn7 db file sequential read 343 0uuqgjq7k12nf db file sequential read 481 db file scattered read 549 75621g9y3xmvd db file sequential read 1158 db file sequential read 1311 0bzhqhhj9mpaa db file sequential read 1523 41zu158rqf4kf db file sequential read 1757 0yas01u2p9ch4 db file sequential read

29. #.29 Copyright 2006 Kyle Hailey Step 3: OEM find top SQL

30. #.30 Copyright 2006 Kyle Hailey IO Solutions  Check  File IO response times  Buffer Cache Hit Ratio  Always Check SQL to Tune

31. #.31 Copyright 2006 Kyle Hailey IO Configuration Issues  Unix Buffer Cache  Disk Spindles  Raid 5

32. #.32 Copyright 2006 Kyle Hailey Machine Memory Unix File Buffer CacheSGA Unix File Cache If average IO read times are under 7ms them probably coming from Unix File Cache

33. #.33 Copyright 2006 Kyle Hailey IO Histograms IO at > 1ms comes From Unix File Cache

34. #.34 Copyright 2006 Kyle Hailey IO Throughput  IO Operations/Sec  150 IOPS/sec optimistic Large Disks – Few IOPS Small Disks – Many IOPS

35. #.35 Copyright 2006 Kyle Hailey Disk 2Gb vs 70GB Seagate Barracuda 4LP Seagate Cheetah 73LP Capacity2.16GB73.4GB Rotation Speed7200rpm10000rpm Rotational Delay(avg)4.1ms3ms Time to read 32Kb6ms3ms Seek Time (avg)9.4ms4.9 Total time for Single I/O19.5ms10.9ms I/O per second (conservative) 5192 IOPs/sec per 100GB2550126

36. #.36 Copyright 2006 Kyle Hailey Raid 5 http://www.miracleas.com/BAARF/BAARF2.html B – Battle A – Against A – Any R – Raid F - Five

37. #.37 Copyright 2006 Kyle Hailey Direct I/O WAITS Direct I/O : This mechanism lets the client bypass the buffer cache for I/O intensive operations. The disk blocks are written into and read from process private memory.  direct path read :  Parallel Query  direct path write  sqlldr  loading/reading LOBs  parallel DMLs  create table as select  create index  direct path read temp, direct path write temp  Sorting

38. #.38 Copyright 2006 Kyle Hailey Direct IO ShadowProcess Buffer Cache PGA Sort Area Direct Path PQO Reads X

39. #.39 Copyright 2006 Kyle Hailey direct path read  Parallel Query  See v$px_session _serial_direct_read = true select parameter1, parameter2, parameter3 from v$event_name where name='direct path read'; PARAMETER1 PARAMETER2 PARAMETER3 ---------- ---------- ---------- file number first dba block cnt

40. #.40 Copyright 2006 Kyle Hailey Further Investigation ASH ----- ----- ---- ----------- ------ -- ------ ------------ - SID QSID P3 OBJN OTYPE FN BLOCKN SQL_ID --- ---- -- ----- ------ -- ------ ------------- 149 152 8 TOTO TABLE 1 194072 4gp8tg0b2s722 144 152 8 TOTO TABLE 1 192304 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 190592 4gp8tg0b2s722 54 152 6 TOTO TABLE 1 201274 4gp8tg0b2s722 149 152 8 TOTO TABLE 1 198552 4gp8tg0b2s722 144 152 8 TOTO TABLE 1 200464 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 196784 4gp8tg0b2s722 ----- ----- ---- ----------- ------ -- ------ ------------ - SID QSID P3 OBJN OTYPE FN BLOCKN SQL_ID --- ---- -- ----- ------ -- ------ ------------- 149 152 8 TOTO TABLE 1 194072 4gp8tg0b2s722 144 152 8 TOTO TABLE 1 192304 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 190592 4gp8tg0b2s722 54 152 6 TOTO TABLE 1 201274 4gp8tg0b2s722 149 152 8 TOTO TABLE 1 198552 4gp8tg0b2s722 144 152 8 TOTO TABLE 1 200464 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 196784 4gp8tg0b2s722 select session_id sid, QC_SESSION_ID qsid, ash.p3, CURRENT_OBJ#||' '||o.object_name objn, o.object_type otype, CURRENT_FILE# fn, CURRENT_BLOCK# blockn, ash.SQL_ID from v$active_session_history ash, all_objects o where event like 'direct path read' and o.object_id (+)= ash.CURRENT_OBJ# Order by sample_time; select session_id sid, QC_SESSION_ID qsid, ash.p3, CURRENT_OBJ#||' '||o.object_name objn, o.object_type otype, CURRENT_FILE# fn, CURRENT_BLOCK# blockn, ash.SQL_ID from v$active_session_history ash, all_objects o where event like 'direct path read' and o.object_id (+)= ash.CURRENT_OBJ# Order by sample_time;

41. #.41 select ash.SQL_ID, QC_SESSION_ID qsid, count(*) cnt, count (distinct session_id) deg, nvl(o.object_name,to_char(CURRENT_OBJ#)) obj, o.object_type otype, decode(session_state, 'WAITING',event,'CPU') event from v$active_session_history ash, all_objects o where o.object_id (+)= ash.CURRENT_OBJ# and qc_session_id is not null group by qc_session_id, sql_id, o.object_name, o.object_type, CURRENT_OBJ#, event, session_state Order by qc_session_id, sql_id PQO - ASH SQL_ID QSID CNT DEG OBJ OTYPE EVENT ------------- ---- ---- --- ------------ ----- ------------------------- - 6rvy3wj2u0h6u 123 42 4 WAIT_OBJECTS TABLE direct path read 123 3 3 WAIT_OBJECTS TABLE CPU 123 3 3 WAIT_OBJECTS TABLE CPU 123 96 4 0 db file sequential read 123 96 4 0 db file sequential read 123 9 4 0 read by other session 123 9 4 0 read by other session 123 2 2 0 CPU 123 2 2 0 CPU 7p3jt75phub2d 144 386 4 WAIT_OBJECTS TABLE PX Deq Credit: send blkd 144 4 3 WAIT_OBJECTS TABLE PX qref latch 144 4 3 WAIT_OBJECTS TABLE PX qref latch 144 37 1 WAIT_OBJECTS TABLE db file sequential read 144 37 1 WAIT_OBJECTS TABLE db file sequential read 144 3 2 WAIT_OBJECTS TABLE direct path read 144 3 2 WAIT_OBJECTS TABLE direct path read 144 70 1 WAIT_OBJECTS TABLE CPU 144 70 1 WAIT_OBJECTS TABLE CPU 144 21 4 0 PX Deq Credit: send blkd 144 21 4 0 PX Deq Credit: send blkd 144 12 4 0 db file sequential read 144 12 4 0 db file sequential read

42. #.42 Copyright 2006 Kyle Hailey direct path write   Occurs when:   insert /*+ APPEND */   sql*loader direct=y   Create table as select select parameter1, parameter2, parameter3 from v$event_name where name='direct path write'; PARAMETER1 PARAMETER2 PARAMETER3 ---------- ---------- ---------- file number first dba block cnt

43. #.43 Copyright 2006 Kyle Hailey Direct Path Read Temp Direct Path Write Temp  Sorting  Write to Temp  Read from Temp select parameter1, parameter2, parameter3 from v$event_name where name in ('direct path write temp‘, ‘direct path read temp’); ‘direct path read temp’); PARAMETER1 PARAMETER2 PARAMETER3 ---------- ---------- ---------- file number first dba block cnt

44. #.44 Copyright 2006 Kyle Hailey Further investigation “real time“ can use Select su.username, su.segtype, s.sql_id, su.blocks from v$sort_usage su, v$session s Where s.saddr = su.session_addr and s.serial# = su.session_num; Select su.username, su.segtype, s.sql_id, su.blocks from v$sort_usage su, v$session s Where s.saddr = su.session_addr and s.serial# = su.session_num; USERNAME SEGTYPE SQL_ID BLOCKS --------- ------- ------------- ------ SYS SORT 3xp88dstadjs4 109440 Note: SQL_ID doesn’t seem filled in for v$sort_usage.sql_id

45. #.45 Copyright 2006 Kyle Hailey sort segment request  SMON allocates sort space in Temp Tablespace  If SMON is busy this can cause waits for sort space  SMON may be stuck rolling back

46. #.46 Copyright 2006 Kyle Hailey file open  Args  P1 fib  P2 iov  P3 0  Wait on an open file  The time it takes to open the file  Times should be order of a block read  To tune, try pre-opening files  Leave sessoins connected  High reconnection rates can exacerbate  Can try holding log files open  (see script from Steve Adams)

47. #.47 Copyright 2006 Kyle Hailey file identify  The time it takes to identify a file so that it can be opened later.  Datafile should be identified just once  Logfiles are identified every checkpoint  Reduce checkpoints  Arguments  p1 fib  p2 filename  p3 opcode (1 test for create, 2 get details)

48. #.48 Copyright 2006 Kyle Hailey Data file init write  When autoextend is set and many extensions are happening

49. #.49 Copyright 2006 Kyle Hailey Local Write Wait  Truncating a table  Wait for Data Cache to be cleared of all blocks of truncated table  Wait by shadow for DBWR  If a problem try  GTT  Reduce fast_start_mttr_target

50. #.50 Copyright 2006 Kyle Hailey Further Investigation NAME P1 P2 P3 ----------------- ----- ------ ----- local write wait file# block# NAME P1 P2 P3 ----------------- ----- ------ ----- local write wait file# block# select * from v$event_name where name = ‘local write wait' select ash.p1,ash.p2, ash.SQL_ID, count(*) from v$ash ash where event='local write wait' group by ash.p1,ash.p2, ash.SQL_ID; P1 P2 SQL_ID COUNT --- -- ------------- ----- 201 2 0q8k9xfmz0k2k 194 201 5 0q8k9xfmz0k2k 34 201 6 0q8k9xfmz0k2k 9 201 4 0q8k9xfmz0k2k 10

51. #.51 Copyright 2006 Kyle Hailey Temporary File #’s SQL> select file# from v$datafile; FILE# ---------- 1 2 3 4 5 6 7 8 SQL> select file# from v$tempfile; FILE# ---------- 2 1 SQL> show parameters db_files NAME VALUE ---------- ------------- db_files 200 Wait Temporary File#’s = Db_files + file# File# 201 = v$tempfile 1

52. #.52 Copyright 2006 Kyle Hailey Local Write Wait:extent allocation CREATE TEMPORARY TABLESPACE "NEWTEMP" TEMPFILE '/d3/temp01.dbf' SIZE 100M AUTOEXTEND ON EXTENT MANAGEMENT LOCAL UNIFORM SIZE 64K; select a.*, b.* from dba_objects a, dba_objects b order by a.owner ALTER DATABASE DEFAULT TEMPORARY TABLESPACE newtemp;

53. #.53 Copyright 2006 Kyle Hailey Local Write Waits

54. #.54 Copyright 2006 Kyle Hailey Local Write Wait: Solutions  Truncating a table  Reduce cache size  Use GTT  Temporary Table Space Allocations  Increase extent size

55. #.55 Copyright 2006 Kyle Hailey Summary I/O  Buffer Cache IO  db file sequential read  db file scattered read  db file parallel read  Tune  I/O’s should be 5-15ms  Check Buffer Cache Advisory  Tune SQL

56. #.56 Copyright 2006 Kyle Hailey Summary Direct I/O  direct path read : PQO  direct path write : direct path operations  direct path read temp: sorts  direct path write temp: sorts  direct path write (lob) : stored NOCACHE

57. #.57 Copyright 2006 Kyle Hailey IO Summary Other  sort segment request : SMON occupied  file open : log files or session reconnects  file identify : log files or session reconnects  local write wait –  Temp extensions  Truncates  data file init write – data file extension