Deconstructing Storage Arrays Timothy E. Denehy, John Bent, Florentina I. Popovici, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin, Madison

Gray-box Research Computer systems becoming more complex Transistors Lines of code Each component is becoming more complex Interactions between subsystems can affect Performance Reliability Power Security

Gray-box Research Interfaces remain the same Changes can be difficult and impractical Support multiple platforms or legacy systems Commercial acceptance for wide-spread adoption Hardware and software phenomenon IA-32 instruction set, POSIX OS, SCSI storage Problem: lack of information

Gray-box Solution Treat target system as a gray-box General characteristics are known Extract information from an existing interface e.g. determine cache contents Exploit information to control system behavior e.g. access cached data first

Gray-box Information Techniques Make assumptions about target system Observe system inputs and outputs Statistical methods Draw inferences about internal structure Microbenchmarks and probes Parameterize system components Observe system under controlled input

Gray-box Applications Gray-box techniques have been used to identify Memory hierarchy parameters [Saavedra and Smith] Processor cycle time [Staelin and McVoy] Low-level disk characteristics [Worthington et al.] Buffer cache replacement algorithms [Burnett et al.] File system data structures [Sivathanu et al.] storage array characteristics: Shear

Shear Software tool that automatically determines the important properties of a storage array Enables file system performance tuning with knowledge of storage array characteristics Acts as a management tool to help configure, monitor, and maintain storage arrays

Outline Introduction Shear Background Algorithm Case Studies Performance: Stripe-aligned Writes Management: Detecting Misconfiguration, Failure Conclusion

Shear Goals Determine storage array characteristics SCSI

Shear Goals Determine storage array characteristics Number of disks SCSI

Shear Goals Determine storage array characteristics Number of disks Chunk size SCSI

Shear Goals Determine storage array characteristics Number of disks Chunk size Layout and redundancy scheme RAID-0 SCSI

Shear Goals Determine storage array characteristics Number of disks Chunk size Layout and redundancy scheme RAID-1 SCSI

Shear Goals Determine storage array characteristics Number of disks Chunk size Layout and redundancy scheme PPPP PPPP PPPP PPPP RAID-5 SCSI

Shear Motivation Performance Tune file systems to array characteristics Management Verify configuration Detect failure

Shear Techniques Microbenchmarks and probes Controlled, random access read and write patterns Measure response time of access patterns Measure steady-state performance Statistical clustering Automatically classify fast and slow regimes Identify patterns that utilize only a single disk

Shear Assumptions Storage array Layout follows a repeatable pattern Composed of homogeneous disks System Able to bypass the file system and buffer cache Little traffic from other processes

Outline Introduction Shear Background Algorithm Case Studies Performance: Stripe-aligned Writes Management: Detecting Misconfiguration, Failure Conclusion

Shear Algorithm Pattern size Chunk size Layout of chunks to disks Level of redundancy

Determining the Pattern Size Find the size of the layout's repeating pattern Not always the stripe size Choose a hypothetical pattern size Perform random reads at multiples of that distance Repeat for a range of pattern sizes Cluster results and identify actual pattern size

Pattern Size Example RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 2 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 4 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 6 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 8 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 10 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 12 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 14 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 16 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 18 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 20 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 22 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 24 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 26 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 28 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 30 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example Testing 32 KB RAID-0 4 Disks 8 KB Chunks

Pattern Size Example RAID-0 4 Disks 8 KB Chunks

Pattern Size Example RAID-0 4 Disks 8 KB Chunks Actual 32 KB cluster

Shear Algorithm Pattern size Chunk size Layout of chunks to disks Level of redundancy

Determining the Chunk Size Chunk size amount of data contiguously allocated to one disk Find the boundaries between disks Choose a hypothetical boundary offset Perform random reads on both sides of that offset Repeat for all offsets in the pattern size Cluster results and identify actual chunk size

Chunk Size Example RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 0 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 2 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 4 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 6 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 8 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 10 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 12 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 14 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example Testing 16 KB RAID-0 4 Disks 8 KB Chunks

Chunk Size Example RAID-0 4 Disks 8 KB Chunks

Chunk Size Example RAID-0 4 Disks 8 KB Chunks Actual 8 KB cluster

Shear Algorithm Pattern size Chunk size Layout of chunks to disks Level of redundancy

Determining the Read Layout Find mapping of chunks to disks Choose a pair of chunks in the pattern Perform random reads to both chunks Repeat for all pairs of chunks Cluster results and identify chunks on same disk

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 0, 0 }

Read Layout Example RAID-0 ZIG-ZAG 4 Disks Testing { 0, 1 } 0 7

Read Layout Example RAID-0 ZIG-ZAG 4 Disks Testing { 0, 2 } 0 7

Read Layout Example RAID-0 ZIG-ZAG 4 Disks Testing { 0, 3 } 0 7

Read Layout Example RAID-0 ZIG-ZAG 4 Disks Testing { 0, 4 } 0 7

Read Layout Example RAID-0 ZIG-ZAG 4 Disks Testing { 0, 5 } 0 7

Read Layout Example RAID-0 ZIG-ZAG 4 Disks Testing { 0, 6 } 0 7

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 0, 7 }

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 1, 1 }

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 1, 2 } 1 6

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 1, 3 } 1 6

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 1, 4 } 1 6

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 1, 5 } 1 6

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 1, 6 } 1 6

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Testing { 1, 7 } 1 6

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks

Read Layout Example 0 7 RAID-0 ZIG-ZAG 4 Disks Actual { 0, 7 } { 1, 6 } { 2, 5 } { 3, 4} 1 6 cluster

Shear Algorithm Pattern size Chunk size Layout of chunks to disks Level of redundancy

Determining Level of Redundancy Ratio of read to write bandwidth reveals the type of redundancy in the array Expected R/W ratios: RAID-0:1 (no redundancy) RAID-1:2(mirroring) RAID-4: varies(examine write layout) RAID-5:4(parity)

Shear Experience Shear has been applied to Linux software RAID Poor RAID-5 parity updates Adaptec hardware RAID controller Implements RAID-5 left-asymmetric layout – RAID-0 – RAID-1 – Chained Declustering – RAID-4 – RAID-5 – P+Q

Outline Introduction Shear Background Algorithm Case Studies Performance: Stripe-aligned Writes Management: Detecting Misconfiguration, Failure Conclusion

RAID-5 Performance Small writes on RAID-5 are problematic Require two reads, parity calculation, two writes Writing in full stripes is more efficient PPPP PPPP PPPP PPPP RAID-5

Stripe-aligned Writes Overcome RAID-5 small write problem Modified Linux disk scheduler Groups writes into full stripes Aligns writes along stripe boundaries Approximately 20 lines of code Experiment Hardware RAID-5, 4 disks, 16 KB chunks Create 100 files of varying sizes

Stripe-aligned Writes Experiment Simple modification has a large impact

Detecting Misconfigurations Correct RAID 5-LSRAID 5-LARAID 5-RSRAID 5-RA Software RAID, 4 Disks, 8 KB Chunks What if one disk is accidentally used twice?

Detecting Misconfigurations Correct Misconfig RAID 5-LSRAID 5-LARAID 5-RSRAID 5-RA

Detecting Failures Software RAID RAID-5 LS 10 disks 8 KB chunks

Detecting Failures Software RAID RAID-5 LS 10 disks 8 KB chunks Disk 5 fails

Outline Introduction Shear Background Algorithm Case Studies Performance: Stripe-aligned Writes Management: Detecting Misconfiguration, Failure Conclusion

Gray-box research Extract / exploit information from existing interfaces Shear Extracts information Automatically determines storage array properties Exploits information File system performance tuning Storage management

Questions?