The HP AutoRAID Hierarchical Storage System John Wilkes, Richard Golding, Carl Staelin, and Tim Sullivan Hewlett-Packard Laboratories

File System Review

UNIX File System (1974) provides an addressable structure to store and retrieve files from disk simple & elegant but slow (2% bandwidth)

File System Review UNIX File System (1974) provides an addressable structure to store and retrieve files from disk simple & elegant but slow (2% bandwidth) Berkeley Fast File System (1984) modified the block size to allow bandwidth to reach up to 47% created cylinder groups that spread metadata to reduce seek times considered hardware specifics during file system parameterization

File System Review UNIX File System (1974) provides an addressable structure to store and retrieve files from disk simple & elegant but slow (2% bandwidth) Berkeley Fast File System (1984) modified the block size to allow bandwidth to reach up to 47% created cylinder groups that spread metadata to reduce seek times considered hardware specifics during file system parameterization Sprite Log-structured File System (1991) relies on increasingly large file caches to handle most reads multiple buffers multiple writes before going to disk single buffer then gets copied entirely to disk in a single write introduced the concept of extents (large continuous set of free blocks) requires cleaning(garbage collection) & requires restructuring of active/non-active data improved crash recovery with roll forward capability !!!!

What about Hardware Failure?? Redundancy FSX FFSX LFSX

What about Hardware Failure?? Redundancy FSX FFSX LFSX RAID

Redundant array of independent disks (early 80s) early days of mainframes Redundant array of inexpensive disks (1988 Patterson, et al) for smaller computer pc ( became widely popular) introduced the concept of partial redundancy Virtualization Array of Disks are viewed as a Single Virtual Disk Requires Array Controller, SCSI connector, hardware and software support  Controls Array of Disks

the many Levels of RAID Patterson introduced five levels No Standards Exist Companies are free to invent their own versions

raid0 STRIPING Pros Good performance on large requests 100% storage capacity Cons Not fault tolerant Not considered raid by many enthusiasts because nothing is redundant

raid1 MIRRORING Pros Good performance And its fault tolerant Cons 50% storage capacity Gets expensive to scale

Parity Parity is calculated using XOR Controller takes a bit from each disk if the total is even  parity = 0 If the total is odd  parity =1 Same protection as mirroring without all the overhead Increased capacity to 80% (1-1/n where n=disks) Easy to restore bits to a single failed drive For missing data, what bit makes parity correct?

raid3 Combine Striping and Redundancy Pros increased storage capacity (1 - 1/N)% high throughput for large files provides partial redundancy using parity Cons parity is at the bit level Poor performance for small I/O no parallel reads or writes possible because parity is on a single disk

raid5 Spread Parity Across All Disks Pros block level striping allows hot swappable disk replacement on failure small requests could be performed in parallel Cons small writes require reading old data, writing new data, reading corresponding old parity value, and writing new parity value(small-write problem) if workload contains too many small writes performance suffers dramatically

All these levels, how do I choose the right one? No Level fits for all occasions fast Raid1 fast but doesn’t scale well 50% storage capacity Raid5 scales but can’t handle multiple small writes

All these levels, how do I choose the right one? No Level fits for all occasions fast Raid1 fast but doesn’t scale well 50% storage capacity Raid5 scales but can’t handle multiple small writes How Can we combine the best of both Levels?

All these levels, how do I choose the right one? No Level fits for all occasions fast Raid1 fast but doesn’t scale well 50% storage capacity Raid5 scales but can’t handle multiple small writes How Can we combine the best of both Levels? Use raid1 for Active data and raid5 for Inactive data

All these levels, how do I choose the right one? No Level fits for all occasions fast Raid1 fast but doesn’t scale well 50% storage capacity Raid5 scales but can’t handle multiple small writes How Can we combine the best of both Levels? Use Raid1 for Active Data and Raid 5 for Inactive data Create a mapping that allows migration between the two

All these levels, how do I choose the right one? No Level fits for all occasions fast Raid1 fast but doesn’t scale well 50% storage capacity Raid5 scales but can’t handle multiple small writes How Can we combine the best of both Levels? Use Raid1 for Active Data and Raid 5 for Inactive data Create a mapping that allows migration between the two Assign a hierarchical preference to each level

All these levels, how do I choose the right one? No Level fits for all occasions fast Raid1 fast but doesn’t scale well 50% storage capacity Raid5 scales but can’t handle multiple small writes How Can we combine the best of both Levels? Use Raid1 for Active Data and Raid 5 for Inactive data Create mapping that allows migration between the two Assign a hierarchical preference Provide a way to migrate data between the two hierarchies

Who Manages the Migration? Not the system administrator Error prone Can not adapt fast enough to changing environment Not the file system Good idea but not a portable solution Could use an array controller if it were smart enough It would have to identify active and inactive data migrate active data to mirrored storage and inactive data to raid5 storage provide a virtual disk to the existing file system be easy to configure

HP AutoRAID Super intelligent array controller Uses Embedded software to manage hierarchy Presents virtual logical units to file system The file system is unaware of storage hierarchy active/inactive grouping data migration Have to provide a mapping to go from virtual to physical addresses!

Data Layout – placing the data on the disk PEX physical extent 1MB of disk space allocation These are the columns of data PEG physical extent group group of at least three PEX’s on different disks Spread across disks to balance data PEG States Can be assigned to the mirrored storage class Can be assigned to the raid5 storage class Can be unassigned Segment – 128KB contiguous space Included in a stripe or mirrored pair RB Relocation Block - 64KB LUN logical unit Host-visible virtual disk STRIPE row of parity & data segments in raid5 peg

LUN ptrs to PEGS PEG tables list of RB’s list of PEX’s PEX tables 1 per disk RB1 RB2 RB3 RB4 RB5 RB6 RB7 RBn LUN/ Virtual Device Tables PEG TABLES PEGn PEG2 PEG1 RB4 RB5 RB6 RB7 PEX1 PEX2 PEX3 Disk 1Disk 2 Disk3 Pex1 segment tablePex2 segment tablePex3 segment table OS File System Mapping Structure

HP AutoRAID What can it do? Initially array starts out empty Data is added to mirrored storage until it is full Some mirrored storage is immediately reallocated to raid5 storage Just re-map PEX’s in mirrored PEG’s to RAID5 PEG’s As workload changes Newly active data are promoted to mirrored storage Data that are less active are demoted to raid5 storage All of this is done in the background - no performance interference Hot-pluggable disks allow for failed component to be removed while system is running Disks can be added to the array at any time up to maximum of 12 Controller fail-over support Active hot spare to reduce the risk of having two drive failures Raid5 uses Log-Structured writes for added performance

Added redundancy Have the ability to add disks to the array on the fly We pushed control disk control from the File System to some fancy hardware with embedded software As far as the file system is concerned we have solved all the problems, right? Well, not really! RAID5 uses log-structured writes, what about the garbage collection? HP AutoRAID is very Slick!

Added redundancy Have the ability to add disks to the array on the fly We pushed control disk control from the File System to some fancy hardware with embedded software As far as the file system is concerned we have solved all the problems, right? Well, not really! RAID5 uses log-structured writes, what about the garbage collection? Same as layout balancing, garbage collection is done in the background This is done by identifying periods of idleness Cleaning requires filling the Holes left when data are promoted to the mirrored storage class HP AutoRAID is very Slick!

Compaction, cleaning, hole plugging RAID5 PEG Hole-Plugging Garbage collection If it is nearly full RB’s from almost empty PEG’s copied to fill holes Minimizes data movement If it is almost empty Those RB’s are used to fill holes in the nearly full ones If it is almost empty and no others holes are ready to be plugged valid RB’s are written to the end of the log Complete PEG is reclaimed as a unit

Performance OLTP macrobenchmark results Raid redundancy HPAutoRaid redundancy JBOD–LVM NO redundancy Striping though, so geared for speed Results are as expected Transaction rate relative to number of disks Working set to large for 5 drives Write set doesn’t fit entirely in mirrored storage Thrashing causes poor performance

Summary HP AutoRAID works well to provide performance and redundancy Extremely easy to setup and use Works in a variety of real life environments Provides outstanding general purpose storage

References Wilkes, John. et al “The HP autoraid hierarchical storage system” Hewlett-Packard Laboratories Patterson, David A “A case for redundant arrays of inexpensive disks (RAID)” Department of Electrical Engineering UC Berkeley Henson, Val “A Brief History of Unix File Systems” Rosenblum, Mendel. et al “The design and implementation of a log-structured file system” Department of Electrical Engineering UC Berkeley McKusick, Marshal K. et al “A fast file system for UNIX*” Department of Electrical Engineering UC Berkeley Raid graphics from Parity graphics from Tanenbaum, Andrew S “Modern Operating Systems 2 nd Edition” Prentice-Hall of India