Evaluating and testing storage performance for Oracle DBs

Evaluating and testing storage performance for Oracle DBs
11/25/08 Evaluating and testing storage performance for Oracle DBs Luca Canali, CERN Dawid Wojcik, CERN UKOUG, Birmingham, December 1st, 2009 The title has to be worked on. 1

CERN and LHC experiments
Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik
Outline Storage for Physics DB CERN Architecture: 10g RAC + ASM on commodity HW DBAs and storage – our working model Performance testing and (new) HW evaluation Tools: ORION, SQL-based test harness Discussion of some test results What is interesting to measure FC commodity HW, tests of iSCSI 1 and 10Gig Measuring IO performance from prod DBs Our experience of production workload measurement Metrics for capacity planning Testing for robustness Pre-prod stress testing Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

CERN Set-up Dual-CPU quad-core 2950 DELL servers, 16GB memory, Intel series “Harpertown”; 2.33GHz clock Dual power supplies, mirrored local disks, 4 NIC (2 private/ public), dual HBAs, “RAID 1+0 like” with ASM Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik 4 Tell there are other production RACs – ATLAS, CMS, LHCb, WLCG… redundant hardware – all machines belonging to one RAC are exactly the same no single point of failure – every component is doubled dual power supply and critical power line – connected to UPS and diesel standardized and homogeneous configuration – certified for usage by Oracle Completely scalable within needs of the experiment

Oracle Cluster Storage, ASM
ASM for mirroring and striping across storage arrays Allows the use of commodity HW (mirror across arrays) Disks can be added and removed online for scheduled and unscheduled changes Example: disk groups: data and flash recovery DATADG RECODG Mirroring Striping Compressing Very Large Data Sets in Oracle, Luca Canali

Why ASM and commodity HW
Storage market is very conservative Enterprise storage typically priced much higher than consumer storage Opportunities Commodity HW/grid-like solutions provide order of magnitude gain in cost/performance Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Service Management and DBAs
DBAs at CERN work across the full stack of activities necessary to run the DB service: User-facing: match users requirements with DB deployments help developers to optimize their DB usage Technology-facing: Involved also in configuration and tuning of ‘lower level stack’ Guarantee SLA, tune SQL execution, backup, security, replication Database software installation, monitoring, patching Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

DBAs and Storage Testing
Opportunity for testing new HW Test for performance Test for stability Measure what we know are the critical metrics Often this means small-read random IOPS Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

HD – the basic element Hard disk technology Basic block of storage since 40 years Main intrinsic limitation: latency Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

HD specs HDs are limited In particular seek time is unavoidable (7.2k to 15k rpm, ~2-10 ms) IOPS Throughput ~100MB/s, typically limited by interface Capacity range 300GB -2TB Failures: mechanical, electric, magnetic, firmware issues. In our experience with ~2000 disks in prod we have about 1 disk failure per week Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Enterprise disks? Performance Enterprise disks offer more ‘performance’ They spin faster and have better interconnect protocols (e.g. SAS vs SATA) Typically of low capacity Our experience: often not competitive in cost/perf vs. SATA Reliability Evidence that low-end and high end disks don’t differ significantly Still an open question Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Scaling out the disk The challenge for storage systems Scale out the disk performance to meet demands Throughput IOPS Latency Capacity Sizing storage systems Sizing must focus on critical metric(s) Depend on the workload of the DB Avoid ‘capacity trap’ Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

RAID and redundancy Storage arrays with RAID are the traditional approach implement RAID to protect data. Parity based: RAID5, RAID6 Stripe and mirror: RAID10 Scalability problem of RAID For very large configurations the time between two failures can become close to RAID volume rebuild time (!) That’s also why RAID6 is becoming more popular than RAID5 Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Oracle ASM way Physics DB storage uses Oracle ASM Volume manager and cluster file system integrated with Oracle Oracle files are divided in chunks Chunks are distributed evenly across storage Chunks are written in multiple copies (2 or 3 it depends on file type and configuration) Allows the use of low-cost storage arrays: does not need RAID support (JBOD is enough) Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

VLDB and storage interconnect
Throughput challenge It takes 1 day to copy/backup 10 TB over 1 GBPS network Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Techology Several technologies available with different characteristics SAN NAS iSCSI Direct attach Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Fiber channel SAN FC SAN is currently most used architecture for enterprise level storage Fast and low overhead on server CPU Used for physics DBs at CERN and Tier1s SAN networks with 64 ports at low cost Measured: 8 Gbps transfer rate (4+4 dual ported HBAs for redundancy and load balancing) Proof of concept FC backup (LAN free) reached full utilization of tape heads Scalable: proof of concept ‘Oracle supercluster’ of 410 SATA disks, and 14 dual quadcore servers Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

How to Test the Whole stack?
Oracle workload based tests Testing using production copies and application-like behaviour Best option but very difficult Benchmarking Not the real thing but can save a lot of time Oracle’s ORION Tests only the access to storage Uses async IO as ASM would do Results from our tests prove it reliable to compare storage solutions Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Testing storage with ORION
Oracle utility that has proven to give similar results as more complex SQL-based tests In the following some examples of results IOPS measured for various disk types FC results iSCSI 1 Gbps and 10 GigE results Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Metrics of interest Basic IO metrics measured by ORION IOPS for random I/O (8KB) MBPS for sequential I/O (in chunks of 1 MB) Latency associated with the IO operations Simple to use Getting started: ./orion_linux_em64t -run simple -testname mytest -num_disks 2 More info: Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

ORION output, an example

ORION results, small random read IOPS
Disks Used Array IOPS IOPS / DISK Mirrored Capacity 128x SATA Infortrend 16-bay 12000 100 24 TB 120x Raptor 2.5’’ 12-bay 17600 150 18 TB 144xWD ‘Green disks’ 10300 12600 70 90 72 TB 22 TB 96x Raptor 3.5’’cmsonline 16000 160 6.5 TB 80x SAS Pics Netapps RAID-DP 17000 210 7.5 TB For the WD green disks 2 numbers are given. The ones ‘on top’ are measured when the entire disks are used for testing. The numbers on the row below correspond to a test where only part of the disk is used (destroking). Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Example of HW testing 14 servers Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Test hardware 8 FC switches: 4Gbps (10Gbps uplink) Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Test hardware 26 storage arrays (16 SATA disks each) Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Results Measured, sequential I/O Read: 6 GB/sec Read-Write: 3+3 GB/sec Measured, small random IO Read: 40K IOPS (8 KB read ops) Note: 410 SATA disks, 26 HBAS on the storage arrays Servers: 14 x 4+4Gbps HBAs, 112 cores, 224 GB of RAM Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

SQL-Based Custom Testing
11/25/08 A custom SQL-based DB workload: IOPS: Probe randomly a large table (several TBs) via several parallel queries slaves (each reads a single block at a time) MBPS: Read a large (several TBs) table with parallel query The test table was 5 TB in size To reduce storage caching effects Test table was probed by a small table via nested loop join V$systat and v$system_event data used to ensure each row of the ‘probe table’ when joined caused a random read on the large test table define NUMBLOCKS= define PARALLDEG=40 Create table testtable1 pctused 10 pctfree 80 tablespace testiops2 nologging as select /*+ parallel (a &PARALLDEG) parallel (b &PARALLDEG) */ rownum num, rpad('name',2000,'P') name from sys.col$ a, sys.col$ b where rownum<=&NUMBLOCKS; Elapsed: 01:40:12.34 set timing on define NUMBLOCKS= define PARALLDEG=100 set autotrace on alter session force parallel ddl parallel &PARALLDEG; -- seems to get stuck for the original testtable with a join, while it is OK for simpler queries Create table testtable_5T pctused 10 pctfree 80 select * from testiops3.testtable1 union all ; exec dbms_stats.set_table_stats('TESTIOPS2','TESTTABLE_5T',numrows=>&NUMBLOCKS,numblks= >&NUMBLOCKS) exec dbms_stats.set_table_stats('TESTIOPS3','TESTTABLE_5T',numrows=>&NUMBLOCKS,numblks= >&NUMBLOCKS) alter session force parallel ddl parallel &PARALLDEG; create table testiops3.probetest2_norandom pctfree 0 tablespace testiops3 nologging select /*+ parallel (a &PARALLDEG ) */ rowid id from testiops3.testtable_5t a ; exec dbms_stats.set_table_stats('TESTIOPS3','PROBETEST2_NORANDOM',numrows=>&NUMBLOCK S) create table testiops3.probetest2_2pct pctfree 0 select a.id id from (select rownum n, id from testiops3.probetest2_norandom) a where mod(a.n,50)=1 order by dbms_random.value(1,&NUMBLOCKS); exec dbms_stats.set_table_stats('TESTIOPS3','PROBETEST2_2PCT',numrows=>&NUMBLOCKS/50) create or replace procedure testrandomio (paralldeg_in in number) v_time1 timestamp; v_DeltaT INTERVAL DAY TO SECOND; v_DeltaSec number; v_dummy1 number; v_query varchar2(300); v_tab varchar(1) default chr(9); v_cpu1 number; v_cpu2 number; v_dbtime1 number; v_dbtime2 number; v_sortdisk1 number; v_sortdisk2 number; v_phywrite1 number; v_phywrite2 number; v_phyread1 number; v_phyread2 number; v_phyreaddir1 number; v_phyreaddir2 number; v_phyreadbytes1 number; v_phyreadbytes2 number; v_wait_userio1 number; v_wait_userio2 number; v_wait_app1 number; v_wait_app2 number; v_waitconcurr1 number; v_waitconcurr2 number; v_wait_clust1 number; v_wait_clust2 number; v_db_file_sequ1 number; v_db_file_sequ2 number; v_db_file_scatt1 number; v_db_file_scatt2 number; v_parseela1 number; v_parseela2 number; v_waits_sequ1 number; v_waits_sequ2 number; v_wait_time_sequ1 number; v_wait_time_sequ2 number; v_waits_scat1 number; v_waits_scat2 number; v_wait_time_scat1 number; v_wait_time_scat2 number; v_waits_direct1 number; v_waits_direct2 number; v_wait_time_direct1 number; v_wait_time_direct2 number; begin dbms_output.put_line('parall#, DeltaTS, DeltaSec, Dummy, CPU, DBTime, SortDisk, PhyWrite, PhyRead, PhyReadDirect, PhyReadBytes, ParseElap, WaitUserIO, WaitApp, WaitConcurr, WaitClust, Wait_time_seq, wait_seq, wait_time_scatt, wait_scatt, wait_direct, wait_time_direct'); for i in 1..2 loop select sum(case event when 'db file sequential read' then TOTAL_WAITS end) waits_sequ, sum(case event when 'db file sequential read' then TIME_WAITED end) wait_time_sequ, sum(case event when 'db file scattered read' then TOTAL_WAITS end) waits_scat, sum(case event when 'db file scattered read' then TIME_WAITED end) wait_time_scat, sum(case event when 'direct path read' then TOTAL_WAITS end) waits_direct, sum(case event when 'direct path read' then TIME_WAITED end) wait_time_direct into v_waits_sequ1, v_wait_time_sequ1, v_waits_scat1, v_wait_time_scat1, v_waits_direct1, v_wait_time_direct1 from gv$system_event; sum(case name when 'CPU used by this session' then value end) CPU, sum(case name when 'DB time' then value end) DBTime, sum(case name when 'sorts (disk)' then value end) SortDisk, sum(case name when 'physical writes' then value end) Phywrite, sum(case name when 'physical reads' then value end) PhyRead, sum(case name when 'physical reads direct' then value end) PhyReadDir, sum(case name when 'physical read bytes' then value end) PhyReadBytes, sum(case name when 'parse time elapsed' then value end) parseela, sum(case name when 'user I/O wait time' then value end) Wait_UserIO, sum(case name when 'application wait time' then value end) Wait_App, sum(case name when 'concurrency wait time' then value end) Wait_concurr, sum(case name when 'cluster wait time' then value end) Wait_Clust into v_cpu1, v_dbtime1, v_sortdisk1, v_phywrite1, v_phyread1, v_phyreaddir1, v_phyreadbytes1, v_parseela1, v_wait_userio1, v_wait_app1, v_waitconcurr1, v_wait_clust1 gv$sysstat; -- PLACE YOUR QUERY HERE! v_time1:=systimestamp; v_query:='select /*+USE_NL(t p) parallel(p '||paralldeg_in ||')*/ sum(1) from testtable_5t t, probetest2_2pct p ' ||'where t.rowid=p.id'; execute immediate v_query into v_dummy1; v_DeltaT:=systimestamp-v_time1; v_DeltaSec:=extract(hour from v_DeltaT)*3600+extract(minute from v_DeltaT)*60+extract(second from v_DeltaT); -- END QUERY v_cpu2, v_dbtime2, v_sortdisk2, v_phywrite2, v_phyread2, v_phyreaddir2, v_phyreadbytes2, v_parseela2, v_wait_userio2, v_wait_app2, v_waitconcurr2, v_wait_clust2 into v_waits_sequ2, v_wait_time_sequ2, v_waits_scat2, v_wait_time_scat2, v_waits_direct2, v_wait_time_direct2 dbms_output.put(paralldeg_in||v_tab); dbms_output.put(v_DeltaT||v_tab); dbms_output.put(v_DeltaSec||v_tab); dbms_output.put(v_dummy1||v_tab); dbms_output.put(v_cpu2-v_cpu1||v_tab); dbms_output.put(v_dbtime2-v_dbtime1||v_tab); dbms_output.put(v_sortdisk2-v_sortdisk1||v_tab); dbms_output.put(v_phywrite2-v_phywrite1||v_tab); dbms_output.put(v_phyread2-v_phyread1||v_tab); dbms_output.put(v_phyreaddir2-v_phyreaddir1||v_tab); dbms_output.put(v_phyreadbytes2-v_phyreadbytes1||v_tab); dbms_output.put(v_parseela2-v_parseela1||v_tab); dbms_output.put(v_wait_userio2-v_wait_userio1||v_tab); dbms_output.put(v_wait_app2-v_wait_app1||v_tab); dbms_output.put(v_waitconcurr2-v_waitconcurr1||v_tab); dbms_output.put(v_wait_clust2-v_wait_clust1||v_tab); dbms_output.put(v_wait_time_sequ2-v_wait_time_sequ1||v_tab); dbms_output.put(v_waits_sequ2-v_waits_sequ1||v_tab); dbms_output.put(v_wait_time_scat2-v_wait_time_scat1||v_tab); dbms_output.put(v_waits_scat2-v_waits_scat1||v_tab); dbms_output.put(v_waits_direct2-v_waits_direct1||v_tab); dbms_output.put(v_wait_time_direct2-v_wait_time_direct1||v_tab); dbms_output.put_line(''); dbms_output.put_line('Parallelism= '||paralldeg_in||' ,IOPS (small IO, 8KB)= '||round((v_phyread2-v_phyread1)/v_DeltaSec,1)); dbms_lock.sleep(1); end loop; end; / set serveroutput on exec testrandomio (1000) -> fast disks

SQL-Based vs. ORION SQL-based vs. ORION Recent experience
11/25/08 SQL-based vs. ORION SQL-based uses more CPU Closer to a ‘real workload’ Each physical IO in Oracle needs a logical IO too, that is CPU + latch, etc ORION just stresses async IO of the OS Recent experience Tested a config of 371 raptor disks of 150 random IOPS per disk SQL-based test maxed out CPU (60% usr, 40% sys) at 42K random IOPS Should have gone up to 55K IOPS define NUMBLOCKS= define PARALLDEG=40 Create table testtable1 pctused 10 pctfree 80 tablespace testiops2 nologging as select /*+ parallel (a &PARALLDEG) parallel (b &PARALLDEG) */ rownum num, rpad('name',2000,'P') name from sys.col$ a, sys.col$ b where rownum<=&NUMBLOCKS; Elapsed: 01:40:12.34 set timing on define NUMBLOCKS= define PARALLDEG=100 set autotrace on alter session force parallel ddl parallel &PARALLDEG; -- seems to get stuck for the original testtable with a join, while it is OK for simpler queries Create table testtable_5T pctused 10 pctfree 80 select * from testiops3.testtable1 union all ; exec dbms_stats.set_table_stats('TESTIOPS2','TESTTABLE_5T',numrows=>&NUMBLOCKS,numblks= >&NUMBLOCKS) exec dbms_stats.set_table_stats('TESTIOPS3','TESTTABLE_5T',numrows=>&NUMBLOCKS,numblks= >&NUMBLOCKS) alter session force parallel ddl parallel &PARALLDEG; create table testiops3.probetest2_norandom pctfree 0 tablespace testiops3 nologging select /*+ parallel (a &PARALLDEG ) */ rowid id from testiops3.testtable_5t a ; exec dbms_stats.set_table_stats('TESTIOPS3','PROBETEST2_NORANDOM',numrows=>&NUMBLOCK S) create table testiops3.probetest2_2pct pctfree 0 select a.id id from (select rownum n, id from testiops3.probetest2_norandom) a where mod(a.n,50)=1 order by dbms_random.value(1,&NUMBLOCKS); exec dbms_stats.set_table_stats('TESTIOPS3','PROBETEST2_2PCT',numrows=>&NUMBLOCKS/50) create or replace procedure testrandomio (paralldeg_in in number) v_time1 timestamp; v_DeltaT INTERVAL DAY TO SECOND; v_DeltaSec number; v_dummy1 number; v_query varchar2(300); v_tab varchar(1) default chr(9); v_cpu1 number; v_cpu2 number; v_dbtime1 number; v_dbtime2 number; v_sortdisk1 number; v_sortdisk2 number; v_phywrite1 number; v_phywrite2 number; v_phyread1 number; v_phyread2 number; v_phyreaddir1 number; v_phyreaddir2 number; v_phyreadbytes1 number; v_phyreadbytes2 number; v_wait_userio1 number; v_wait_userio2 number; v_wait_app1 number; v_wait_app2 number; v_waitconcurr1 number; v_waitconcurr2 number; v_wait_clust1 number; v_wait_clust2 number; v_db_file_sequ1 number; v_db_file_sequ2 number; v_db_file_scatt1 number; v_db_file_scatt2 number; v_parseela1 number; v_parseela2 number; v_waits_sequ1 number; v_waits_sequ2 number; v_wait_time_sequ1 number; v_wait_time_sequ2 number; v_waits_scat1 number; v_waits_scat2 number; v_wait_time_scat1 number; v_wait_time_scat2 number; v_waits_direct1 number; v_waits_direct2 number; v_wait_time_direct1 number; v_wait_time_direct2 number; begin dbms_output.put_line('parall#, DeltaTS, DeltaSec, Dummy, CPU, DBTime, SortDisk, PhyWrite, PhyRead, PhyReadDirect, PhyReadBytes, ParseElap, WaitUserIO, WaitApp, WaitConcurr, WaitClust, Wait_time_seq, wait_seq, wait_time_scatt, wait_scatt, wait_direct, wait_time_direct'); for i in 1..2 loop select sum(case event when 'db file sequential read' then TOTAL_WAITS end) waits_sequ, sum(case event when 'db file sequential read' then TIME_WAITED end) wait_time_sequ, sum(case event when 'db file scattered read' then TOTAL_WAITS end) waits_scat, sum(case event when 'db file scattered read' then TIME_WAITED end) wait_time_scat, sum(case event when 'direct path read' then TOTAL_WAITS end) waits_direct, sum(case event when 'direct path read' then TIME_WAITED end) wait_time_direct into v_waits_sequ1, v_wait_time_sequ1, v_waits_scat1, v_wait_time_scat1, v_waits_direct1, v_wait_time_direct1 from gv$system_event; sum(case name when 'CPU used by this session' then value end) CPU, sum(case name when 'DB time' then value end) DBTime, sum(case name when 'sorts (disk)' then value end) SortDisk, sum(case name when 'physical writes' then value end) Phywrite, sum(case name when 'physical reads' then value end) PhyRead, sum(case name when 'physical reads direct' then value end) PhyReadDir, sum(case name when 'physical read bytes' then value end) PhyReadBytes, sum(case name when 'parse time elapsed' then value end) parseela, sum(case name when 'user I/O wait time' then value end) Wait_UserIO, sum(case name when 'application wait time' then value end) Wait_App, sum(case name when 'concurrency wait time' then value end) Wait_concurr, sum(case name when 'cluster wait time' then value end) Wait_Clust into v_cpu1, v_dbtime1, v_sortdisk1, v_phywrite1, v_phyread1, v_phyreaddir1, v_phyreadbytes1, v_parseela1, v_wait_userio1, v_wait_app1, v_waitconcurr1, v_wait_clust1 gv$sysstat; -- PLACE YOUR QUERY HERE! v_time1:=systimestamp; v_query:='select /*+USE_NL(t p) parallel(p '||paralldeg_in ||')*/ sum(1) from testtable_5t t, probetest2_2pct p ' ||'where t.rowid=p.id'; execute immediate v_query into v_dummy1; v_DeltaT:=systimestamp-v_time1; v_DeltaSec:=extract(hour from v_DeltaT)*3600+extract(minute from v_DeltaT)*60+extract(second from v_DeltaT); -- END QUERY v_cpu2, v_dbtime2, v_sortdisk2, v_phywrite2, v_phyread2, v_phyreaddir2, v_phyreadbytes2, v_parseela2, v_wait_userio2, v_wait_app2, v_waitconcurr2, v_wait_clust2 into v_waits_sequ2, v_wait_time_sequ2, v_waits_scat2, v_wait_time_scat2, v_waits_direct2, v_wait_time_direct2 dbms_output.put(paralldeg_in||v_tab); dbms_output.put(v_DeltaT||v_tab); dbms_output.put(v_DeltaSec||v_tab); dbms_output.put(v_dummy1||v_tab); dbms_output.put(v_cpu2-v_cpu1||v_tab); dbms_output.put(v_dbtime2-v_dbtime1||v_tab); dbms_output.put(v_sortdisk2-v_sortdisk1||v_tab); dbms_output.put(v_phywrite2-v_phywrite1||v_tab); dbms_output.put(v_phyread2-v_phyread1||v_tab); dbms_output.put(v_phyreaddir2-v_phyreaddir1||v_tab); dbms_output.put(v_phyreadbytes2-v_phyreadbytes1||v_tab); dbms_output.put(v_parseela2-v_parseela1||v_tab); dbms_output.put(v_wait_userio2-v_wait_userio1||v_tab); dbms_output.put(v_wait_app2-v_wait_app1||v_tab); dbms_output.put(v_waitconcurr2-v_waitconcurr1||v_tab); dbms_output.put(v_wait_clust2-v_wait_clust1||v_tab); dbms_output.put(v_wait_time_sequ2-v_wait_time_sequ1||v_tab); dbms_output.put(v_waits_sequ2-v_waits_sequ1||v_tab); dbms_output.put(v_wait_time_scat2-v_wait_time_scat1||v_tab); dbms_output.put(v_waits_scat2-v_waits_scat1||v_tab); dbms_output.put(v_waits_direct2-v_waits_direct1||v_tab); dbms_output.put(v_wait_time_direct2-v_wait_time_direct1||v_tab); dbms_output.put_line(''); dbms_output.put_line('Parallelism= '||paralldeg_in||' ,IOPS (small IO, 8KB)= '||round((v_phyread2-v_phyread1)/v_DeltaSec,1)); dbms_lock.sleep(1); end loop; end; / set serveroutput on exec testrandomio (1000) -> fast disks

iSCSI Storage iSCSI is interesting for cost reduction Some trends to get rid of ‘specialized’ FC network Reuse existing infrastructure and know-how Not applicable to all systems IP interconnect throughput CPU usage Adoption seems to be only for low throughput systems at the moment However 10GigE tests are very promising Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

iSCSI 1 Gbps Orion scalability tests, small IO, FC vs. iSCSI ./orion_linux_x run advanced -write 0 -matrix point -duration testname mytest -simulate raid0 -num_disks 24 -cache_size num_small num_large 0 Double the number of actual disks to push them to their limits Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

iSCSI 1 Gbps Orion scalability tests, large IO, FC vs. iSCSI ./orion_linux_x run advanced -write 0 -matrix point -duration testname mytest2 -simulate raid0 -num_disks 24 -cache_size num_small 0 -num_large 1000 Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

iSCSI on 10 GigE We have recently tested custom iSCSI 10GigE storage ‘CERN-made’ disk servers that export storage as iSCSI over 10 GigE 2 Clovertown quad-core processors of 2.00 GHz 8 GB of RAM 16 SATA-II drives of 500 GB, 7'200 rpm RAID controller 3ware 9650SE-16ML Intel 10GigE dual port server adapter PCIexpress (EXPX9502CX4 - Oplin) HP Procurve 10GigE switch Data: H. Meinhard, CERN Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

iSCSI tests, 10 GigE Preliminary results ORION tests with up to 3 disk arrays (14 disks each) Almost linear scalability Up to 42 disks tested -> 4000 IOPS at saturation 85% CPU idle during test IOPS of a single disk: ~110 IOPS Promising interconnect technology, if it can be made rock solid like FC Performance of iSCSI stacks varies a lot from version to version Data: A. Horvath, CERN Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Disclaimer Simple testing with ORION is not the same as production workload testing CPU cycles also important for IOPS You need to combine different IO patterns ... see next slides Testing has many sides We need reasonable numbers quickly Results should allow comparison between platforms and HW types Results needs to be confronted with real life performance Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Storage Capacity Planning
Capacity planning – iterative process, also for ‘stable’ services Size is not the only metric for storage Measure and understand current system utilization Identify bottlenecks (if any) Prepare forecast for future growth Plan for HA Try to plan for the ‘unexpected’ Service expansion Plan stability issues HW failures Consolidate if possible – better resource utilization Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Storage Capacity Planning
Capacity planning requires historical data Diagnostic Pack (AWR – DBA_HIST_SYSMETRIC_SUMMARY) Custom probing (V$SYSMETRIC_SUMMARY) Choose some important metrics: Physical Read Total Bytes Per Sec Physical Read Total IO Requests Per Sec Physical Write Total Bytes Per Sec Physical Write Total IO Requests Per Sec Redo Generated Per Sec Redo Writes Per Sec ... DB size Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Capacity Planning – plots

Reproducing IO load pattern
Historical data allows to set-up simple IO tests that will resemble production workload Reproduce different IO sizes (large vs small IO ratio) Reproduce mixed workload (read/write ratio) Reproduce cluster-wide IO load (nodes IO ratios) Single node testing trap - parallel (multi-node) IO tests required to detect some potential SCSI level or storage issues (both performance and stability) Test for spikes (short term and long term) Test on storage of similar size Cache influence, uneven platter performance Watch for storage cache Disable, purge, warm-up Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Testing for Robustness
Our experience Putting in production new HW and OS versions needs extensive testing Especially true with commodity HW Lessons learned after few years of running different tests Mixed read/write (if possible) Mixed IO size Simulate workload from several nodes Periodic testing: high load, low load, high load ... Proper testing should also detect most of the 'infant mortality’ cases Beware – disk or storage firmware issues can take long time to debug Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Testing for Robustness
Some issues can still appear only on a running production system Be proactive – schedule for tests! ASM normal redundancy – backup validate check logical is not enough! Corruption can affect secondary extents only! We have developed a tool that scans all ASM files and compares primary and secondary extents Set-up media scans on the storage arrays in low activity periods Test HA features (new kernel version may have bugs) Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Conclusions Storage is key for successful DB implementations Testing different technologies DBAs can profit by getting a deeper knowledge of their storage subsystem Typical case: new HW evaluation, pre-prod stress testing There are simple tools for testing, ORION Monitoring and measuring production environment Helps in keeping service level/ production performance Provides data for capacity planning Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Acknowledments Many thanks to Physics DB team at CERN, in particular for this work: Maria Girone, Jacek Wojcieszuk Physics later today: Eva Dafonte Pérez, 13:15, “Worldwide distribution of experimental physics data using Oracle Streams” Luca Canali, 15:55, “Compressing very large data sets in Oracle” Testing Storage Performance for Oracle DBs – L.Canali, D.Wojcik

Evaluating and testing storage performance for Oracle DBs

Similar presentations

Presentation on theme: "Evaluating and testing storage performance for Oracle DBs"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Evaluating and testing storage performance for Oracle DBs

Similar presentations

Presentation on theme: "Evaluating and testing storage performance for Oracle DBs"— Presentation transcript:

Similar presentations

About project

Feedback