PAE SuperCluster Peformance June 12, 2012

SuperCluster/Exadata Dashboard Focus Area Previous Status (05/23) Current Status (06/12) Comments Virtualization (scaling-within-a-box) G OLTP: Consistent performance across zones achieved by manually re-targeting ixgbe interrupt with SR-IOV VF. Currently testing on zones w/ SR-IOV VF IB support verifying that interrupt re-targeting is working as expected. DSS: CPU-intensive queries scaling close to linear from 1 to 8 zones (1 zone/core) on single-socket in split PCI-E LDOM. IO-intensive query doesn’t scale due to IO bottleneck. Exadata Expansion Rack Evaluation Avg IB latency of cells in Expansion Rack same as in SSC rack rds-stress peaks at 1.08M IOPS at 8 cells. Degrades to 0.9M at 21 cells. Orion 8k peak of 850K IOPS, avg 815K IOPS @ 6-8 cells. Throughput degrades slowly to 21 cells at 790K IOPS. Orion 8K IOPS: 24 cells scaling from 1 to 4 T4-4’s non-partitioned: 2.45M IOPS 1M Throughput: 24 cells: 1xT4-4=10.8GB/s, 4xT4-4=24GB/s IOPS issues due to CR 7169352, Throughput issues under investigation 2-socket 11gR2 LDOM vs T4-2 11GR2 LDom has 5% better remote latency than T4-2 (lmbench) iGenRAM & iGenCPU tests result in equivalent performance Storage Cells vs Cached Storage Array (log file sync) Flexcube log file sync issue due to I/O saturation of Storage Cells. Igen oltp log file sync improved from 6-9ms to 2ms w/ disabling nxge driver Storage Cell Flashlog Disk First investigation Niwot Controller Cache is faster than Aura flash for log writes: 97% of log writes satisfied by disk. If controller cache turned off 100% log writes satisfied by flash. With heavy concurrent read workload disk writes redo 100MB/s faster than flash. Exadata Flashcache with Aura2 R On hold: Exadata Storage Server software to support Aura2 not be available. RAC Scalability On Hold: Resources redirected to Storage Cells vs. Cached Storage Array project.

SuperCluster/Exadata – Highlights Performance issues reported by Field/non-DPA Orgs T4-2 vs 2-Socket 11gR2 LDOM CPU/Memory Performance Issue: T4-2 has better CPU and memory performance compared to 2-socket 11gR2 LDOM CPU/Memory Resolution: PAE repeated tests using iGenCPU and iGenRAM resulted in equivalent performance Exadata Storage Cell vs Cached Storage Array (log file sync issues) Issue: iGenOLTP on SSC had 35% better Response Time with Redo Logs on ZFSSA 7320 than on Exadata Storage Cells with FlashLog Resolution: Disabling nxge driver reduced Log File Sync from 6-9ms to 2ms. Response Time with Storage Cells is now 1.6x better than ZFSSA Issue: General Observation that with Exadata FlashLog only small percentage of redo writes are satisfied by flash. Resolution: Purpose of FlashLog is to eliminate outliers when controller cache is congested. Confirmed that Niwot Controller Cache on Exadata Storage is faster than Aura flash for log writes: 97% of log writes satisfied by disk. If controller cache turned off 100% log writes satisfied by flash. FlashLog behavior is correct. SuperCluster/Exadata – Highlights Confidential

SuperCluster/Exadata – Lowlights Performance issues reported by Field/non-DPA Orgs Issue: Flexcube team reported that performance for given batch step was 2.2x faster on T4-2 w/ ST6180 than on SSC 11gR2 Ldom w /3xCells due to redo log performance. Review of data showed I/O saturated on the storage cells. Also test conditions were not exactly the same. Highlights that SSC/Exadata performance work has concentrated on read performance only. Will expand scope to include write-intensive workloads. Evaluate Exadata Expansion Rack Target Goals for Orion I/O tests on Full Rack SSC connect to Full Exadata Expansion Rack (4xT4-4 and 24xExadata Storage Cells) 3M IOPS (random 8K reads) 48 GB/s (random 1M reads) Actual Results 2.45M IOPS (random 8k reads) 24GB/s (random 1M reads) Performance shortfall due mainly to 56 event queues/socket limitation which results in cpus sharing interrupts (CR 7169352) Exadata FlashCache with Aura2 Exadata Storage Server software to support Aura2 is still not available for testing. SuperCluster/Exadata – Lowlights Confidential

SuperCluster/EECS Performance Dashboard Focus Area Previous Status (02/12) Current Status (03/25) Comments MWM-E Met and exceeded December RR goals G Self-Tuning/worker thread optimization now works generally well based on MWM-E Lock-Less Resource Manager (LLRM) does not show overhead. LLRM is recent EECS optimization Improved OTD performance by 20%. Can reach IR 5900 using OTD or IR 6000 without OTD Virtualization (scaling-within-a-box) N/A WIP Characterize & Analyze VM, both Zones and SR-IOV LDOMs, scaling based on current SSC split PCIe LDoms configurations Current low-level SR-IOV IB preliminary performance status : Single Connection latency over VF to external client is 5-10% worse than baremetal to external client Request/response profile performance – IPoIB over VF performance is 30% less than as expected. Under investigation OTD Observe throughput drop with higher connections (IPoIB to WLS). Under Investigation Differentiator exploration – OTD + ILB: Solaris ILB is Layer 4 kernel Load Balancer. When conducting layer 4 LB, ILB consumes ¼ CPU of OTD for delivering the same throughput Found out ILB performance under IB UD mode is much better than under CM mode due to lack of soft-interrupts feature under IB CM mode. Prototyping in progress Infinibus/Coherence Proof of concept status: Unit tests passed, both C and Java portion, on Solaris x86 platform SPARC porting in progress TBD: likely need OFUV library optimization – from gcc to Sun Studio

OTD vs OTD + ILB Performance With Layer 4 LB Rules OTD Only With ILB (+OTD)

OTD Differentiator Next Step OTD + ILB + OpenFlow OS Kernel (ILB) Hardware To-Do:SSL offloading and acceleration Back-end servers Clients