PARAID: A Gear-Shifting Power-Aware RAID Charles Weddle, Mathew Oldham, Jin Qian, An-I Andy Wang – Florida St. University Peter Reiher – University of California, Los Angeles Geoff Kuenning – Harvey Mudd College

Motivation Energy costs are rising Energy costs are rising An increasing concern for servers An increasing concern for servers No longer limited to laptops No longer limited to laptops Energy consumption of disk drives Energy consumption of disk drives 24% of the power usage in web servers 24% of the power usage in web servers 27% of electricity cost for data centers 27% of electricity cost for data centers Root to other issues, e.g. server room cooling Root to other issues, e.g. server room cooling Is it possible to reduce energy consumption in RAID devices without degrading performance while maintaining reliability? Is it possible to reduce energy consumption in RAID devices without degrading performance while maintaining reliability? PARAID: A Gear-Shifting Power-Aware RAID

Challenges Energy Energy Not enough opportunities to spin down RAIDs Not enough opportunities to spin down RAIDs Performance Performance Essential for peak loads Essential for peak loads Reliability Reliability Server-class drives are not designed for frequent power switching Server-class drives are not designed for frequent power switching PARAID: A Gear-Shifting Power-Aware RAID

Existing Work Most trade performance for energy savings directly. Most trade performance for energy savings directly. For example, vary the speed of disks For example, vary the speed of disks Most are simulated results Most are simulated results PARAID: A Gear-Shifting Power-Aware RAID

Observations Over provisioning of resources Over provisioning of resources RAID is configured for peak performance RAID is configured for peak performance RAID keeps all drives spinning for light loads RAID keeps all drives spinning for light loads Unused storage capacity Unused storage capacity Over-provision of storage capacity Over-provision of storage capacity Unused storage can be traded for energy savings Unused storage can be traded for energy savings Fluctuating load Fluctuating load Cyclic fluctuation of loads Cyclic fluctuation of loads Infrequent on-off power transitions can be effective Infrequent on-off power transitions can be effective PARAID: A Gear-Shifting Power-Aware RAID

Power-Aware RAID Skewed striping for energy savings Skewed striping for energy savings Preserving peak performance Preserving peak performance Maintaining reliability Maintaining reliability Evaluation Evaluation Conclusion Conclusion Questions Questions PARAID: A Gear-Shifting Power-Aware RAID

Skewed Striping for Energy Saving Use over-provisioned spare storage Use over-provisioned spare storage Can use fewer disks for light loads Can use fewer disks for light loads PARAID: A Gear-Shifting Power-Aware RAID RAID Soft State Gears

Skewed Striping for Energy Saving Operate in gear 1 Operate in gear 1 Disks 4 and 5 are powered off Disks 4 and 5 are powered off PARAID: A Gear-Shifting Power-Aware RAID RAID Soft State Gears

Skewed Striping for Energy Saving Approximate the workload Approximate the workload Gear shift into most appropriate gear Gear shift into most appropriate gear Minimize the opportunity lost to save power Minimize the opportunity lost to save power Energy ( Powered On Disks ) Workload ( Disk Parallelism ) Conventional RAIDPARAID workload PARAID: A Gear-Shifting Power-Aware RAID

Skewed Striping for Energy Saving Adapt to cyclic fluctuating workload Adapt to cyclic fluctuating workload Gear shift when gear utilization threshold is met Gear shift when gear utilization threshold is met time load utilization threshold gear shift PARAID: A Gear-Shifting Power-Aware RAID

Preserving Peak Performance Operate in the highest gear Operate in the highest gear When the system demands peak performance When the system demands peak performance Uses the same disk layout Uses the same disk layout Maximize parallelism within each gear Maximize parallelism within each gear Load is balanced on each gear Load is balanced on each gear Uniform striping pattern within each gear Uniform striping pattern within each gear Delay block replication until gear shifts Delay block replication until gear shifts Track block writes as an optimization Track block writes as an optimization PARAID: A Gear-Shifting Power-Aware RAID

Maintaining Reliability Reuse existing RAID levels (RAID-5) Reuse existing RAID levels (RAID-5) Also used in soft state Also used in soft state Drives have a limited number of power cycles Drives have a limited number of power cycles Ration number of power cycles Ration number of power cycles PARAID: A Gear-Shifting Power-Aware RAID

File System RAID PARAID Block Handler Disk Device Driver User Space Linux kernel Soft RAID Reliability Manager Monitor Disk Manager Admin Tool Logical Component Design PARAID: A Gear-Shifting Power-Aware RAID

Data Layout PARAID: A Gear-Shifting Power-Aware RAID Disk 1Disk 2Disk 3Disk 4Disk 5 Soft State (RAID-5) (1-4)812((1-4),8,12) 1620(16,20,_)_ RAID (1-4) 567(5-8)8 910(9-12) (13-16) (17-20) Parity for 5 disks does not work for 4 disks Parity for 5 disks does not work for 4 disks For example, replicated block 12 on disk 3 For example, replicated block 12 on disk 3

Data Layout PARAID: A Gear-Shifting Power-Aware RAID Disk 1Disk 2Disk 3Disk 4Disk 5 Soft State (RAID-5) (1-4)812((1-4),8,12) 1620(16,20,_)_ RAID (1-4) 567(5-8)8 910(9-12) (13-16) (17-20) Cascading parity updates Cascading parity updates Must also update parity in soft state Must also update parity in soft state

Update Propagation Up-shift propagation Up-shift propagation Full synchronization Full synchronization On-demand synchronization On-demand synchronization For example, shifting from 3 to 5 disks For example, shifting from 3 to 5 disks Downshift propagation Downshift propagation Full synchronization Full synchronization PARAID: A Gear-Shifting Power-Aware RAID

Asymmetric Gear-Shifting Policies Up-shift (aggressive) Up-shift (aggressive) Moving average + moving standard deviation > threshold Moving average + moving standard deviation > threshold Downshift (conservative) Downshift (conservative) Modified moving average + moving standard deviation < threshold Modified moving average + moving standard deviation < threshold Moving average modified to account for extra parity updates Moving average modified to account for extra parity updates PARAID: A Gear-Shifting Power-Aware RAID

Implementation Prototyped in Linux Prototyped in Linux Open source, software RAID Open source, software RAID Implemented block I/O handler, monitor, disk manager Implemented block I/O handler, monitor, disk manager Implemented user admin tool to configure device Implemented user admin tool to configure device Updated Raid Tools to recognize PARAID level Updated Raid Tools to recognize PARAID level PARAID: A Gear-Shifting Power-Aware RAID

Evaluation Challenges Challenges Prototyping PARAID Prototyping PARAID Commercial machines Commercial machines Benchmarks are designed to measure peak performance Benchmarks are designed to measure peak performance Trace replay Trace replay Time consuming Time consuming PARAID: A Gear-Shifting Power-Aware RAID

Evaluation multimeter USB cable client server power supply 12v & 5v power lines power measurement probes SCSI cable crossover cable Xeon 2.8 Ghz, 512 MB RAM 36.7 GB 10k RPM SCSI P4 2.8 Ghz, 1 GB RAM 160 GB 7200 RPM SATA RAID BOOT PARAID: A Gear-Shifting Power-Aware RAID Measurement framework Measurement framework

Evaluation PARAID: A Gear-Shifting Power-Aware RAID ServerClient ProcessorIntel Xeon 2.8 GhzIntel Pentium Ghz Memory512 Mbytes1 Gbytes NetworkGigabit Ethernet Disks 36.7Gbytes 15K RPM SCSI Ultra Gbytes 7200 RPM SATA

Evaluation Three different workloads using two different RAID settings Three different workloads using two different RAID settings Web trace - RAID level 0 – 2 gears (2,5) Web trace - RAID level 0 – 2 gears (2,5) Mostly read activity Mostly read activity Cello99 - RAID level 5 – 2 gears (3,5) Cello99 - RAID level 5 – 2 gears (3,5) I/O intensive workload with writes I/O intensive workload with writes Postmark - RAID level 5 – 2 gears (3,5) Postmark - RAID level 5 – 2 gears (3,5) Measure peak performance and gear shifting overhead Measure peak performance and gear shifting overhead Speed up trace playback Speed up trace playback To match hardware To match hardware Explore range of speed up factors and power savings Explore range of speed up factors and power savings PARAID: A Gear-Shifting Power-Aware RAID

Web Trace UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006) UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006) File system: ~32 GB (~500k files) File system: ~32 GB (~500k files) Trace replay: ~95k requests with ~4 GB data (~260 MB unique) Trace replay: ~95k requests with ~4 GB data (~260 MB unique) PARAID: A Gear-Shifting Power-Aware RAID

Web Trace Energy Savings PARAID: A Gear-Shifting Power-Aware RAID 64x – 60 requests/sec 128x – 120 requests/sec256x – 240 requests/sec 64x - 34% 128x - 28% 256x - 10% Energy Savings

Web Trace Latency PARAID: A Gear-Shifting Power-Aware RAID 256x 128x64x 256x - within 2.7% 64x - 240% 80ms vs. 33ms Overhead

Web Trace Bandwidth PARAID: A Gear-Shifting Power-Aware RAID 256x 128x64x 256x - within 1.3% in high gear Overhead

Cello99 Trace Cello99 Workload Cello99 Workload HP Storage Research Labs HP Storage Research Labs 50 hours beginning on 9/12/ hours beginning on 9/12/1999 I/O intensive with 42% writes I/O intensive with 42% writes PARAID: A Gear-Shifting Power-Aware RAID

Cello99 Energy Savings PARAID: A Gear-Shifting Power-Aware RAID 128x – 1000 requests/sec 32x – 270 requests/sec 64x – 550 requests/sec 32x - 13% 64x - 8.2% 128x - 3.5% Energy Savings

Cello99 Completion Time PARAID: A Gear-Shifting Power-Aware RAID 128x 64x32x 32x - 1.8ms, 26% slower due to time spent in low gear Overhead

Cello99 Bandwidth PARAID: A Gear-Shifting Power-Aware RAID 64x32x 128x Overhead < 1% degra- dation during peak hours

Postmark Benchmark Popular synthetic benchmark Popular synthetic benchmark Generates ISP-style workloads Generates ISP-style workloads Stresses peak read/write performance of storage device Stresses peak read/write performance of storage device PARAID: A Gear-Shifting Power-Aware RAID

Postmark Performance PARAID: A Gear-Shifting Power-Aware RAID

Postmark Power Measurements PARAID: A Gear-Shifting Power-Aware RAID

Ongoing Work Try more workloads Try more workloads Optimize PARAID gear configuration Optimize PARAID gear configuration Explore asynchronous update propagation Explore asynchronous update propagation Speed up recovery Speed up recovery Live testing Live testing PARAID: A Gear-Shifting Power-Aware RAID

Lessons Learned Third version of design, early design not portable Third version of design, early design not portable Data alignment problems Data alignment problems Difficult to measure system under normal load Difficult to measure system under normal load Hardware and operating system optimizations Hardware and operating system optimizations Matching trace environment Matching trace environment PARAID: A Gear-Shifting Power-Aware RAID

Conclusion PARAID reuses standard RAID-levels without special hardware while decreasing their energy use by 34%. PARAID reuses standard RAID-levels without special hardware while decreasing their energy use by 34%. Optimized version can save even more energy Optimized version can save even more energy Empirical evaluation important Empirical evaluation important PARAID: A Gear-Shifting Power-Aware RAID

Questions

PARAID Gear-Shifting 256x128x64x Number of gear switches % time spent in low gear52%88%98% % extra I/Os for update propagations 0.63%0.37%0.21% 128x64x32x Number of gear switches % time spent in low gear47%74%88% % extra I/Os for update propagations 8.0%15%21% Web Trace Gear-Shifting Stats Cello99 Gear-Shifting Stats

Storage consumption S i for the total RAID for the ith gear; PARAID uses around (D – G 1 )/(D – 1) of the total RAID-5 storage to store soft states. For RAID-5, D > 3 disks, M gears with G i disks within the ith gear (1 < i < M, 3 < G i < G i+1 < G M = D) Storage Requirement for PARAID5

Disk 1Disk 2Disk 3Disk 4Disk 5 Gear 1 RAID-5 (1-4)812((1-4),8,12) 1620(16,20,_)_ Gear 2 RAID (1-4) 567(5-8)8 910(9-12) (13-16) (17-20) S1 S2 G1 = 4 G2 = 5 S2(G2 – G1) S1(G1 – 1) Space needed for extra parity blocks S1 + S2 = 1 S1(G1 – 1) = S2(G2 – G1)

PARAID Target percentage energy savings; Energy savings increase with more disks and fewer disks in the lowest gear. A higher active/standby ratio. D = the number of disks in the array G 1 = the disks in gear 1. P = power in standby/active/idle

PARAID Modified moving utilization; Accounts for cascading parity writes in lower gear. G i = the disks in gear i. A read/write = the read or write activity W = the weight to account for additional parity writes (RAID5 = 1.5)

PARAID maxminconfigurationempty storage max energy savingsformula %25.00% %40.00% %20.00% %40.00% %50.00% %33.00% %17.00% %50.00% %50.00% %33.00% %50.00% %57.14% %42.86% %28.57% %14.29%