MicroTerabyte Leveraging InfiniBand to Build a Powerful, Scalable Oracle Database and Application Platform Brian Dougherty Chief Architect, CMA

Background Exploding data volumes are presenting new challenges to small and medium sized organizations. These organizations need a new generation of technology that delivers powerful analytical capability with reduced cost and complexity. CMA, in partnership with Dell, QLogic, Oracle, and EMC, has developed a sophisticated solution to address this growing need.

MicroTerabyte Platform What is It ? A pre-configured, integrated, fault tolerant, high performance, commodity hardware and software based runtime environment Leverages the power of the Oracle database running on commodity Linux servers, and a unified InfiniBand fabric over QLogic multi-protocol fabric directors Provides a scalable, reliable, lower cost platform for Business Intelligence, Custom Software, or Commercial Software

MicroTerabyte Platform Features High Performance Scalable Fault Tolerant Commodity Hardware Lower Cost Smaller Footprint Reduced Power Consumption

MicroTerabyte Platform Attributes Red Hat Linux Operating System Commodity Hardware (Dell / EMC CX / QLogic) 42U Rack Footprint with Fault Tolerant 42U rack Clustered Oracle Database Clustered Storage Provisioning Layer Unified Storage and Interconnect Fabric

MicroTerabyte Single Rack Configuration Sample Hardware Components (per rack) (2) Dell 1950 Database Servers 32GB RAM Each 16 Processor Cores (1) Dell 1950 ETL Server 16GB RAM 8 Processor Cores (1) Dell 1950 Business Intelligence Server 16GB RAM 8 Processor Cores (1) QLogic 9020 InfiniBand Fabric Director (2) FVIC Modules (2) EMC CX3-40 Storage Arrays 2TB – 5TB Storage (1) Dell PowerConnect 48 Port GigE Switch (1) Belkin KVM (1) Dell Console Physical Data Guard

MicroTerabyte Mid-Range Configuration Hardware components (two racks) (4) Dell 1950 Database Servers 32GB RAM Each 32 Processor Cores (1) Dell 1950 ETL Server 32GB RAM 8 Processor Cores (1) Dell 1950 Business Intelligence Server 32GB RAM 8 Processor Cores (1) QLogic 9040 InfiniBand Fabric Director (2) FVIC Modules (2) EMC CX3-40 Storage Arrays 8TB – 15TB Storage (2) EMC SATA 1TB Drives (for backup) (1) Dell PowerConnect 48 Port GigE Switch (1) Belkin KVM (1) Dell Console

MicroTerabyte Large Scale Configuration Hardware components (three racks) (8) Dell 1950 Database Servers 32GB RAM Each 64 Processor Cores (2) Dell 1950 ETL Server 32GB RAM Each 8 Processor Cores (2) Dell 1950 Business Intelligence Server 32GB RAM Each 8 Processor Cores (1) QLogic 9040 InfiniBand Fabric Director (4) FVIC Modules (4) EMC CX3-40 Storage Arrays 24TB – 32TB Storage (2) EMC SATA 1TB Drives (for backup) (1) Dell PowerConnect 48 Port GigE Switch (1) Belkin KVM (1) Dell Console

MicroTerabyte Architecture Mid-Level Diagram Storage Nodes Unified Fabric Layer Compute Nodes QLogic Multi-Protocol Fabric Director Dell 1950 PowerEdge Servers / RHEL v5 EMC CX3 / CX4 Flare OS / Navisphere

MicroTerabyte Architecture Detailed Diagram Public NetworkingDell PowerConnect 6248 Server ComputeDell 1950/ QLogic 7104 HCA Unified Fabric QLogic 9020/9040 Multi-Protocol Director Storage InfrastructureEMC CX3/CX4 Red Hat Linux O/S QLogic OFED InfiniBand Drivers Oracle ASMlib Oracle Clusterware 11g Oracle ASM 11g Oracle EE 11g w/ RAC and Partitioning ERP Application Package ISV Third Party Software CMA BI Suite Business Intelligence MicroTerabyte Platform MT Core MT O/S Binding MT Clustered Database / Storage Provisioning Applications Oracle Grid Control 11g

MicroTerabyte Platform Oracle Software

MicroTerabyte Compute Nodes Dell 1950/2950 MicroTerabyte solution consists of 2/4/8 RAC Database nodes 2 ETL and BI Nodes Each node is a Dell 1950/2950 consisting of: –Processor: One/Two quad-core Intel Xeon 2.66GHz –Memory: GB –Hard drives: 218GB-3TB Internal Storage –RAID Controller: PERC 5/i – 1 DDR InfiniBand HCA –Network interface cards: Dual gigabit NICs (100baseTx-FD) –Power supply: 670W, optional hot-plug redundant power (1+1) –Operating system: RedHat Enterprise Linux v5

MicroTerabyte Storage Nodes EMC CX4 Model 480 Front-End Host Connectivity Two storage processors per CX4 Each processor has: Four 4Gb Fibre Channel optical ports FCP SCSI-3 protocol Back-End Disk Connectivity Each processor has 4Gb Fibre Channel arbitrated loops. Multiple RAID groups may be distributed across redundant loops Supports a maximum of 480 disk drives System Memory Each processor has 8GB of Memory Power Consumption (Processor Chassis) 355 VA (290W max) Power Consumption (Disk Expansion Chassis) 440 VA (425W max)

MicroTerabyte Unified Fabric Layer What is it? QLogic 9020 & 9040 Multi-Protocol Fabric Directors –9020 with two (2) FVIC IB-FC Virtual I/O Controllers –9040 with up to four (4) FVIC IB-FC Virtual I/O Controllers –each FVIC provides 10 DDR (20Gb) IB ports & 8 4Gb FC ports supports up to 128 Virtual HBA ports per module automatic sensing 1/2/4 Gb/s load balancing automatic port and module fail-over LUN mapping and masking features QLogic 7104 Host Channel Adapters –Dual Port, DDR –IPoIB, RDS, SRP

MicroTerabyte Unified Fabric Layer General Benefits Managing one fabric Reduced footprint Compact implementation Fewer host components needed to support I/O and interconnect Increased bandwidth and reduced latency Reduced host resources (1 HCA vs. several HBAs) Path failover through SRP protocol Well positioned to take advantage of advances in InfiniBand technology

MicroTerabyte Unified Fabric Layer Oracle RAC Benefits Scalable platform to support Oracle RAC More predictable response times Capability to drive more Oracle I/O through fewer compute nodes Ability to exploit storage capability at a lower cost Reduced Oracle messaging latency via RDS/IB

CONFIGURATION A mid-size MicroTerabyte configuration, including: Servers: Four (4) Dell 1950 Intel quad core servers. Each 1950 includes 8 cores, 16GB memory and 1 dual channel HCA. The server is running Red Hat Linux 5 update 1 ( kernel). Storage: Two (2) EMC CX3-40 storage arrays. Each storage array includes 8 4Gbps front-end fiber channel connections for a total of 16 4Gbps FE adapters -- approx 7.5TB usable storage configured in 4+1 RAID sets. Unified Interconnect and Storage IB fabric: One (1) QLogic 9040 Multi-Protocol Director with 2 FVIC modules HARDWARE COST Approximate total cost (market): $500,000 TEST METHOD AND RESULTS Testing simulator: Oracle ORION -- Oracle I/O Numbers Calibration Tool for Linux >> Random I/O Test Results: Test Type: random I/O I/O size: 8K Rate observation 1: 9,600 sustained 1.87 ms per I/O avg node latency Rate observation 2: 30,566 sustained 2.96 ms per I/O avg node latency Rate observation 3: 42,615 sustained 2.96 ms per I/O avg node latency >> Sequential I/O Test Results: Test Type: sequential I/O I/O size: 1MB Rate observation 1: 677MB/sec sustained I/O seq ms per 1MB I/O avg node latency Rate observation 2: 2.098GB/sec sustained I/O seq ms per 1MB I/O avg node latency MicroTerabyte ORION Benchmarks

MicroTerabyte Oracle Benchmarks Servers: (4) Dell 1950 Dual Socket Quad Core 16 GB RAM per server Single dual port IB HCA Unified I/O Fabric:QLogic 9040 (2) FVIC Modules Storage:(2) EMC CX-3 Model 40 (16) Fibre Channel ports O/S: Red Hat Linux v 5.1 Kernel UDEV Unified I/O Fabric:QLogic IB drivers v IB/RDS IB/SRP IPoIB Database:Oracle RAC 10gR2 Oracle Clusterware 10gR2 Oracle ASM Oracle RMAN Oracle DataGuard Storage:Flare O/S EMC Navisphere Backup:EMC Networker Software Hardware

MicroTerabyte Benchmark Comparisons

Summary As demonstrated in CMAs MicroTerabyte platform, Infiniband can provide an extremely capable transport mechanism for unifying interconnect and i/o traffic Increases bandwidth and reduces latency Reduces Oracle messaging latency via RDS/IB Provides more predictable response times Reduces host resource requirements (i/o processing workload off-loaded to HCA card) Consistent with Oracles strategic technology direction

Brian Dougherty Chief Architect, CMA For More Information