1. 1 PARAID: A Gear-Shifting Power-Aware RAID Charles Weddle, Mathew Oldham, Jin Qian, An-I Andy Wang – Florida St. University Peter Reiher – University of California, Los Angeles Geoff Kuenning – Harvey Mudd College

2. 2 Research Areas Power-aware RAID [FAST07, TOS07] Power-aware RAID [FAST07, TOS07] Electric-field-based routing [SIGMOBILE04, ICCCN07, MSWiM07] Electric-field-based routing [SIGMOBILE04, ICCCN07, MSWiM07] Conquest disk-persistent-RAM hybrid file system [HotOS98, USENIX02, TOS06] Conquest disk-persistent-RAM hybrid file system [HotOS98, USENIX02, TOS06] Optimistic replication [Sigmetrics05, SPECTS05, Simulation07] Optimistic replication [Sigmetrics05, SPECTS05, Simulation07] Real-time systems [RTAS07, RTAS08] Real-time systems [RTAS07, RTAS08]

3. 3 Motivation Energy costs are rising Energy costs are rising An increasing concern for servers An increasing concern for servers No longer limited to laptops No longer limited to laptops Energy consumption of disk drives Energy consumption of disk drives 24% of the power usage in web servers 24% of the power usage in web servers 27% of electricity cost for data centers 27% of electricity cost for data centers More energy  more heat  more cooling  lower computational density  more space  more costs More energy  more heat  more cooling  lower computational density  more space  more costs Is it possible to reduce energy consumption without degrading performance while maintaining reliability? Is it possible to reduce energy consumption without degrading performance while maintaining reliability? PARAID: A Gear-Shifting Power-Aware RAID

4. 4 Challenges Energy Energy Not enough opportunities to spin down RAIDs Not enough opportunities to spin down RAIDs Performance Performance Essential for peak loads Essential for peak loads Reliability Reliability Server-class drives are not designed for frequent power switching Server-class drives are not designed for frequent power switching PARAID: A Gear-Shifting Power-Aware RAID

5. 5 Existing Work Most trade performance for energy savings directly. Most trade performance for energy savings directly. e.g. vary speed of disks e.g. vary speed of disks Most are simulated results Most are simulated results PARAID: A Gear-Shifting Power-Aware RAID

6. 6 Observations Over provisioning of resources Over provisioning of resources RAID is configured for peak performance RAID is configured for peak performance RAID keeps all drives spinning for light loads RAID keeps all drives spinning for light loads Unused storage capacity Unused storage capacity Over-provision of storage capacity Over-provision of storage capacity Unused storage can be traded for energy savings Unused storage can be traded for energy savings Fluctuating load Fluctuating load Cyclic fluctuation of loads Cyclic fluctuation of loads Infrequent on-off power transitions can be effective Infrequent on-off power transitions can be effective PARAID: A Gear-Shifting Power-Aware RAID

7. 7 Power-Aware RAID Skewed striping for energy savings Skewed striping for energy savings Preserving peak performance Preserving peak performance Maintaining reliability Maintaining reliability Evaluation Evaluation Conclusion Conclusion Questions Questions PARAID: A Gear-Shifting Power-Aware RAID

8. 8 Skewed Striping for Energy Saving Use over-provisioned spare storage Use over-provisioned spare storage Can use fewer disks for light loads Can use fewer disks for light loads PARAID: A Gear-Shifting Power-Aware RAID RAID Soft State Gears 1 2 3 12345

9. 9 Skewed Striping for Energy Saving Operate in gear 1 Operate in gear 1 Disks 4 and 5 are powered off Disks 4 and 5 are powered off PARAID: A Gear-Shifting Power-Aware RAID RAID Soft State Gears 1 2 3 12345

10. 10 Skewed Striping for Energy Saving Approximate the workload Approximate the workload Gear shift into most appropriate gear Gear shift into most appropriate gear Minimize the opportunity lost to save power Minimize the opportunity lost to save power Energy ( Powered On Disks ) Workload ( Disk Parallelism ) Conventional RAIDPARAID workload PARAID: A Gear-Shifting Power-Aware RAID

11. 11 Skewed Striping for Energy Saving Adapt to cyclic fluctuating workload Adapt to cyclic fluctuating workload Gear shift when gear utilization threshold is met Gear shift when gear utilization threshold is met time load utilization threshold gear shift PARAID: A Gear-Shifting Power-Aware RAID

12. 12 Preserving Peak Performance Operate in the highest gear Operate in the highest gear When the system demands peak performance When the system demands peak performance Uses the same disk layout Uses the same disk layout Maximize parallelism within each gear Maximize parallelism within each gear Load is balanced Load is balanced Uniform striping pattern Uniform striping pattern Delay block replication until gear shifts Delay block replication until gear shifts Capture block writes Capture block writes PARAID: A Gear-Shifting Power-Aware RAID

13. 13 Maintaining Reliability Reuse existing RAID levels (RAID-5) Reuse existing RAID levels (RAID-5) Also used in soft state Also used in soft state Drives have a limited number of power cycles Drives have a limited number of power cycles Ration number of power cycles Ration number of power cycles PARAID: A Gear-Shifting Power-Aware RAID

14. 14 Maintaining Reliability Busy disk stay powered on, idle disks stay powered off Busy disk stay powered on, idle disks stay powered off Outside disks are role exchanged with middle disks Outside disks are role exchanged with middle disks busy disks power cycled disks idle disks role exchange Disk 1 Gear 1 Gear 2 Gear 3 Disk 2Disk 3Disk 4Disk 5Disk 6 PARAID: A Gear-Shifting Power-Aware RAID

15. 15 File System RAID PARAID Block Handler Disk Device Driver User Space Linux kernel Soft RAID Reliability Manager Monitor Disk Manager Admin Tool Logical Component Design PARAID: A Gear-Shifting Power-Aware RAID

16. 16 Data Layout PARAID: A Gear-Shifting Power-Aware RAID Disk 1Disk 2Disk 3Disk 4Disk 5 Gear 1 RAID-5 (1-4)812((1-4),8,12) 1620(16,20,_)_ Gear 2 RAID-5 1234(1-4) 567(5-8)8 910(9-12)1112 13(13-16)141516 (17-20)17181920 Parity for 5 disks does not work for 4 disks Parity for 5 disks does not work for 4 disks For example, replicated block 12 on disk 3 For example, replicated block 12 on disk 3

17. 17 Data Layout PARAID: A Gear-Shifting Power-Aware RAID Disk 1Disk 2Disk 3Disk 4Disk 5 Gear 1 RAID-5 (1-4)812((1-4),8,12) 1620(16,20,_)_ Gear 2 RAID-5 1234(1-4) 567(5-8)8 910(9-12)1112 13(13-16)141516 (17-20)17181920 Cascading parity updates Cascading parity updates For example, updating block 8 on disk 5 For example, updating block 8 on disk 5

18. 18 Update Propagation Up-shift propagation Up-shift propagation Full synchronization Full synchronization On-demand synchronization On-demand synchronization For example, shifting from 3 to 5 disks For example, shifting from 3 to 5 disks Downshift propagation Downshift propagation Full synchronization Full synchronization PARAID: A Gear-Shifting Power-Aware RAID

19. 19 Asymmetric Gear-Shifting Policies Up-shift (aggressive) Up-shift (aggressive) Moving average + moving standard deviation > threshold Moving average + moving standard deviation > threshold Downshift (conservative) Downshift (conservative) Modified moving average + moving standard deviation < threshold Modified moving average + moving standard deviation < threshold Moving average modified to account for extra parity updates Moving average modified to account for extra parity updates PARAID: A Gear-Shifting Power-Aware RAID

20. 20 Implementation Prototyped in Linux 2.6.5 Prototyped in Linux 2.6.5 Open source, software RAID Open source, software RAID Implemented block I/O handler, monitor, disk manager Implemented block I/O handler, monitor, disk manager Implemented user admin tool to configure device Implemented user admin tool to configure device Updated Raid Tools to recognize PARAID level Updated Raid Tools to recognize PARAID level PARAID: A Gear-Shifting Power-Aware RAID

21. 21 Evaluation Challenges Challenges Prototyping PARAID Prototyping PARAID Commercial machines Commercial machines Conceptual barriers Conceptual barriers Benchmarks designed to measure peak performance Benchmarks designed to measure peak performance Trace replay Trace replay Time consuming Time consuming PARAID: A Gear-Shifting Power-Aware RAID

22. 22 Evaluation multimeter USB cable client server power supply 12v & 5v power lines power measurement probes SCSI cable crossover cable Xeon 2.8 Ghz, 512 MB RAM 36.7 GB 10k RPM SCSI P4 2.8 Ghz, 1 GB RAM 160 GB 7200 RPM SATA RAID BOOT PARAID: A Gear-Shifting Power-Aware RAID Measurement framework Measurement framework

23. 23 Evaluation Measurement framework Measurement framework PARAID: A Gear-Shifting Power-Aware RAID ServerClient ProcessorIntel Xeon 2.8 GhzIntel Pentium 4 2.8 Ghz Memory512 Mbytes1 Gbyte NetworkGigabit Ethernet Disks 5x36.7Gbytes 15K RPM SCSI Ultra 320 160 Gbytes 7200 RPM SATA

24. 24 Evaluation Three different workloads using two different RAID settings Three different workloads using two different RAID settings Web trace - RAID level 0 (2-disk gear 1, 5-disk gear 2) Web trace - RAID level 0 (2-disk gear 1, 5-disk gear 2) Mostly read activity Mostly read activity Cello99 - RAID level 5 (3-disk gear 1, 5-disk gear 2) Cello99 - RAID level 5 (3-disk gear 1, 5-disk gear 2) I/O intensive workload with writes I/O intensive workload with writes Postmark - RAID level 5 Postmark - RAID level 5 Measure peak performance and gear shifting overhead Measure peak performance and gear shifting overhead Speed up trace playback Speed up trace playback To match hardware To match hardware Explore range of speed up factors and power savings Explore range of speed up factors and power savings PARAID: A Gear-Shifting Power-Aware RAID

25. 25 Web Trace UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006) UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006) File system: ~32 GB (~500k files) File system: ~32 GB (~500k files) Trace replay: ~95k requests with ~4 GB data (~260 MB unique) Trace replay: ~95k requests with ~4 GB data (~260 MB unique) PARAID: A Gear-Shifting Power-Aware RAID

26. 26 Web Trace Power Savings PARAID: A Gear-Shifting Power-Aware RAID 64x – 60 requests/sec 128x – 120 requests/sec256x – 240 requests/sec 64x - 34% 128x - 28% 256x - 10% Energy Savings

27. 27 Web Trace Latency PARAID: A Gear-Shifting Power-Aware RAID 256x 128x64x 256x - within 2.7% 64x - 240% 80ms vs. 33ms Overhead

28. 28 Web Trace Bandwidth PARAID: A Gear-Shifting Power-Aware RAID 256x 128x64x 256x - within 1.3% in high gear Overhead

29. 29 Cello99 Trace Cello99 Workload Cello99 Workload HP Storage Research Labs HP Storage Research Labs 50 hours beginning on 9/12/1999 50 hours beginning on 9/12/1999 I/O intensive with 42% writes I/O intensive with 42% writes PARAID: A Gear-Shifting Power-Aware RAID

30. 30 Cello99 Power Savings PARAID: A Gear-Shifting Power-Aware RAID 128x – 1000 requests/sec 32x – 270 requests/sec 64x – 550 requests/sec 32x - 13% 64x - 8.2% 128x - 3.5% Energy Savings

31. 31 Cello99 Completion Time PARAID: A Gear-Shifting Power-Aware RAID 128x 64x32x 32x - 1.8ms, 26% slower due to time spent in low gear Overhead

32. 32 Cello99 Bandwidth PARAID: A Gear-Shifting Power-Aware RAID 64x32x 128x Overhead < 1% degra- dation during peak hours

33. 33 Postmark Benchmark Popular synthetic benchmark Popular synthetic benchmark Generates ISP-style workloads Generates ISP-style workloads Stresses peak read/write performance of storage device Stresses peak read/write performance of storage device PARAID: A Gear-Shifting Power-Aware RAID

34. 34 Postmark Performance PARAID: A Gear-Shifting Power-Aware RAID

35. 35 Postmark Power Savings PARAID: A Gear-Shifting Power-Aware RAID

36. 36 Ongoing Work Try more workloads Try more workloads Optimize PARAID gear configuration Optimize PARAID gear configuration Explore asynchronous update propagation Explore asynchronous update propagation Speed up recovery Speed up recovery Live testing Live testing PARAID: A Gear-Shifting Power-Aware RAID

37. 37 Lessons Learned Third version of design, early design not portable Third version of design, early design not portable Data alignment problems Data alignment problems Difficult to measure system under normal load Difficult to measure system under normal load Hardware and operating system optimizations Hardware and operating system optimizations Matching trace environment Matching trace environment PARAID: A Gear-Shifting Power-Aware RAID

38. 38 Conclusion PARAID reuses standard RAID-levels without special hardware while decreasing their energy use by 34%. PARAID reuses standard RAID-levels without special hardware while decreasing their energy use by 34%. Optimized version can save even more energy Optimized version can save even more energy Empirical evaluation important Empirical evaluation important PARAID: A Gear-Shifting Power-Aware RAID

39. 39 Storage Research Projects Workload characterizations Persistent-RAM-based file system Peer-replicated filing service Power-aware RAID Latency-reduction RAID Concurrency-aware RAID Secure and swift data deletion open(foo)read(bar)write(zoo) File System

40. 40 Questions PARAID: A Gear-Shifting Power-Aware RAID Contact Contact Andy Wang – awang@cs.fsu.edu Andy Wang – awang@cs.fsu.eduawang@cs.fsu.edu http://www.cs.fsu.edu/~awang/conquest-2 http://www.cs.fsu.edu/~awang/conquest-2

41. 41 Related Work Hibernator [Q. Zhu, et al., 2005] Hibernator [Q. Zhu, et al., 2005] MAID [D. Colarelli, et al. 2002] MAID [D. Colarelli, et al. 2002] PDC [E. Pinheiro, et al. 2004] PDC [E. Pinheiro, et al. 2004] EERAID [D. Li, et al. 2004] EERAID [D. Li, et al. 2004] RIMAC [X. Yao, et al. 2006] RIMAC [X. Yao, et al. 2006] BlueFS [E.B. Nightingale, et al. 2005] BlueFS [E.B. Nightingale, et al. 2005] PARAID: A Gear-Shifting Power-Aware RAID

42. 42 PARAID Recovery 2.7 times slower than conventional raid 2.7 times slower than conventional raid For example, 2 gear PARAID device For example, 2 gear PARAID device First, the soft state must recover First, the soft state must recover Second, data must be propagated Second, data must be propagated Third, conventional raid must recover Third, conventional raid must recover Recovery not as bad for read intensive workloads Recovery not as bad for read intensive workloads

43. 43 PARAID Gear-Shifting 256x128x64x Number of gear switches15.28.02.0 % time spent in low gear52%88%98% % extra I/Os for update propagations 0.63%0.37%0.21% 128x64x32x Number of gear switches6.05.65.4 % time spent in low gear47%74%88% % extra I/Os for update propagations 8.0%15%21% Web Trace Gear-Shifting Stats Cello99 Gear-Shifting Stats