Monitoring and Testing I/O Using Perfmon, Oracle tools and IOMeter for monitoring and load testing the I/O Subsystem

Objectives At the end of this module the student will understand the following tasks and concepts. Monitoring the I/O Subsystem using Perfmon Monitoring the I/O Subsystem using Oracle Tools Load Testing the I/O Subsystem using TTCP

Oracle and I/O I/O performance is crucial to Oracle Performance Data buffers are sensitive to read I/O latency Log buffers are sensitive to write I/O latency Any slowdown in I/O performance can cause a significant performance loss for Oracle. Although Oracle uses Buffer Cache very effectively, all data must at some point be read into memory, andall transactions must be written out to the redo log files..

Oracle I/O Problems Performance Degradation Index lookup 10 I/Os 10 x 6 ms = 60 ms (not bad) 10 x 100 ms = 1 sec (noticeable) Table Scan 30,000 I/Os 30,000 x 6 ms = 90 sec (not bad for a Table Scan) 30,000 x 100 ms = 3,000 sec = 50 minutes You are getting phone calls and emails

Monitoring I/O Performance Perfmon General Tips Set the sample rate up 1 sec is too fast Not accurate on all counters Not enough time to see all the counters Way to fast for saving the data Heizenburg Uncertainty Principle Use both graph and report as appropriate Save data if possible

Monitoring I/O Performance Perfmon Logical Disk Counters Lists the Physical Disks by Drive Letter Good for monitoring file I/O Physical Disk Counters Lists the Physical Disks by Physical Disk Good for monitoring the LUNs or Volumes

Monitoring I/O Performance Perfmon Latencies IOPS (I/Os Per Second) Throughput I/O sizes

Monitoring I/O Performance Perfmon Latencies Avg. Disk sec/Read Provides Read Latency Statistics Avg. Disk sec/Write Provides Write Latency Statistics

Monitoring I/O Performance Latency

Monitoring I/O Performance Perfmon IOPS Disk Reads/sec Disk Transfers/sec Useful for total volume IOPS (remember RAID overhead) Disk Writes/sec

Monitoring I/O Performance IOPS

Monitoring I/O Performance Perfmon Throughput Disk Bytes/sec Total Throughput Disk Read Bytes/sec Disk Write Bytes/sec

Monitoring I/O Performance Throughput

Monitoring I/O Performance Perfmon I/O sizes Avg. Disk Bytes/Read Avg. Disk Bytes/Write These counters provide information that allows you to get a little bit of an idea of what Oracle is doing. When Oracle is doing index lookups or random reads and writes the standard block size of 8 KB will be used and you will see an 8 KB Avg. Disk Bytes/Read and an 8 KB Avg. Disk Bytes/Write. When Oracle is doing table scans the block size will be much larger. The maximum size is determined by the maximum I/O size allowable by the OS. In addition, the Oracle log writes are variable based on how much data is in the log buffer when the log writer thread is ready to write it out.

Monitoring I/O Performance I/O Sizes

Monitoring I/O Performance Perfmon Miscellaneous Current Disk Queue Length The current disk queue length could give you an indication that there are performance problems with the I/O subsystem. With SAN storage the disk queue length should not be over 1 or 2. If this number is very high you should investigate further. This is not as good of an indication of a slowdown as the latency counters.

Monitoring I/O Performance Oracle Tools OEM Performance Page Instance Disk I/O Instance Throughput

Monitoring I/O Performance Oracle Tools Oracle Counters for Windows Performance Monitor Add Oracle Data Files counter physreads/sec Physwrites/se Choose data files to monitor Shows I/O for database files

Monitoring I/O Performance Oracle Tools STATSPACK Reports Tablespace IO Stats Data File IO Stats ------------------------------------------------------------- Tablespace IO Stats for DB: DSSA Instance: dssa4 Snaps: 1 -2 ->ordered by IOs (Reads + Writes) desc Tablespace ------------------------------ Av Av Av Av Buffer Av Buf Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms) -------------- ------- ------ ------- ------------ -------- ---------- ------ UNDOTBS4 0 0 0.0 3,587 2 1 0.0 USERS 96 0 0.6 3.4 1,961 1 0 0.0 TEMP 952 0 1.7 15.0 952 0 0 0.0 TOOLS 0 0 0.0 1,080 1 0 0.0 SYSTEM 30 0 6.0 1.0 453 0 0 0.0 File IO Stats for DB: DSSA Instance: dssa4 Snaps: 1 -2 ->ordered by Tablespace, File Tablespace Filename ------------------------ ---------------------------------------------------- SYSTEM /u11/dssa/system01.dbf 30 0 6.0 1.0 453 0 0 TEMP /u11/dssa/temp01.dbf 952 0 1.7 15.0 952 0 0 TOOLS /u11/dssa/tools01.dbf 0 0 1,080 1 0 UNDOTBS4 /u11/dssa/undotbs04.dbf 0 0 3,587 2 1 0.0 USERS /u11/dssa/users01.dbf 96 0 0.6 3.4 1,961 1 0

Load Testing I/O Used to determine I/O problems in a controlled and scientific manner Used to identify the upper limits of the storage subsystem Used to prove or disprove hypothesis or theories

Load Testing I/O IOMeter IOMeter is a public domain utility Available at http://www.sourceforge.net Generates I/O based on your specifications Provides accurate results Is repeatable

Load Testing I/O IOMeter Made up of two major components GUI Used for configuration and results Dynamo Used to generate the load

Load Testing I/O IOMeter Configuring IOMeter Disk Targets Network Targets Access Specifications Results Display Test Setup

Load Testing I/O Disk Targets Maximum Disk Size For random I/Os this must be sufficiently large # of Outstanding I/Os Defines concurrency of I/Os If set to one there will only be one I/O issued at a time and IOPS = 1 / latency The maximum disk size should be sufficiently large to cause seeking on the disk drives. In some cases, this must be very large. A maximum disk size that is too small will skew your results. The number of outstanding I/Os must be sufficiently high to simulate Oracle asynchronous I/O. If you leave it at the default of 1, only one I/O will be outstanding at time, thus resulting in no queueing.

Load Testing I/O Network Targets Used for network I/O testing

Load Testing I/O Access Specifications Specifies the type of I/O Block Size Read/Write ratio Sequential/Random Ratio Create new spec for Oracle

Load Testing I/O Test Setup Used to set up automated tests Run time Ramp up Number of workers Etc.

Load Testing I/O Results Display View the performance during the test IOPS Throughput Latency CPU Utilization

Review What type of I/O problem causes the biggest problem with Oracle Data Files ? Name some Perfmon counters to monitor for I/O? What counter does the Oracle Counters for Windows Performance Monitor add to Perfmon that measures Oracle I/O? What are the two components of IOMeter?

Summary Monitoring the I/O Subsystem using Perfmon Monitoring the I/O Subsystem using Oracle Tools Load Testing the I/O Subsystem using TTCP