Experiences with VI Communication for Database Storage Yuanyuan Zhou, Angelos Bilas, Suresh Jagannathan, Cezary Dubnicki, Jammes F. Philbin, Kai Li

Presentation Outline Motivation  Use VI-based interconnects to improve I/O path performance between a database server and the storage subsystem. Approach  Design a block-level storage architecture V3 – VI-attached Volume Vault  Implement a software layer between the application and VI DSA – Direct Storage Access  Perform measurement by using Microsoft SQL Server 2000 on a 4-cpu and 32-cpu database server.

V3 Architecture V3 client  Small scale uniprocessor  SMP system  Large scale multiprocessor server V3 server  V3 volume  A volume provides more than 2TB storage  Span multiple nodes by combinations of RAID  More than 250TB storage in V3 back-end VI interconnect Customer software

DSA Implementations Block-level I/O module specification layer between the application and VI. Take advantage of VI-specific features  RDMA  Minimize kernel involvement  minimize multiple copying  Large number of Overlapping I/O requests

DSA Implementation Provide new features  Flow control, retransmission, reconnection  Optimization Memory registration and deregistration Interrupt handling Lock synchronization  One kernel-level implementation  Two user-level implementations

Kernel-level DSA Implementation Implemented on top of a kernel-level version of the VI specification. Provide the standard I/O interface for storage driver. Support user-level / kernel- level application without modification. Leverage the benefits of VI in kernel-level storage APIs

User-level DSA Implementation Replace all I/O calls to V3 storage Support standard windows I/O interface. Issue I/O requests without kernel involvement. Need kernel involvement for I/O completion. Notify wDSA by the interrupt to complete the corresponding I/O requests.

User-level DSA Implementation A new I/O API Minimize kernel involvement, context switches and interrupt Trade off transparency for performance. May need modification of the application. Application-controlled I/O completion mode either polling or interrupt

System Optimization for DSA VI registration and deregistration  Not feasible to pre-registering all I/O buffers at startup in database systems. Dynamically manage registered memory. kDSA – Windows I/O manager cDSA – Address Windowing Extensions  Batched deregistration

System Optimization Interrupts  Notify the database for a completed asynchronous I/O request.  Interrupt batching to reduce the high cost kDSA –When I/O requests > the threshold, use polling. cDSA – Set a completion flag by RDMA. Polling the flag during a time interval; switch to wait for interrupt after then.

System Optimization Lock Synchronization  Count a significant percentage on the I/O path.  Reduce the lock/unlock operations cDSA has more control of the lock synchronization pairs.

Experiment Platform A mid-size, 4-cpu SMP A large-size, 32-CPU SMP Giganet network Maximum end-to-end user-level bandwidth is about 110MB/s One way latency for a 64-bytes message is about 7 us.

Micro-benchmark Results DSA Overhead V3 configuration  A single application client runs the micro- benchmark.  A single storage node presents the virtual disk Raw VI  A locally attached disk, without any V3 software Reading a data block from the storage server.

DSA Overhead A round-trip delay for a read I/O request cDSA has the lowest CPU overhead

V3 Cached-block Performance

V3 vs. Local Case

On-line Transaction Processing BenchMark – TPC-C Commercial databases issue multiple concurrent I/Os and tolerant high I/O response times and low-throughput disks. Large database configuration Mid-size database configuration

Large Database Configuration

Mid-size Database Configuration

Conclusion cDSA provides a 18% transaction rate improvement over FC for large database configuration. Effective use of VI in I/O intensive environments requires substantial enhancements to flow control, reconnection, interrupt handling, memory registration and lock synchronization. New storage APIs that help minimize kernel involvement in the I/O path are needed to fully exploit the benefits of VI-based communication.