CE202 December 2, 2003 David Pease Storage Architecture CE202 December 2, 2003 David Pease

Hierarchy of Storage Faster Slower Smaller Larger Higher Lower RAM Speed Smaller Larger Capacity Higher Lower Cost RAM Disk Optical Tape Cache

Storage System Components Application I/O Library File System Device Driver Host Bus Adapter Interconnect Storage Controller Devices I/O Context

Disks Microdrive, SCSI disk, RAMAC, and what’s the fourth thing?

Disk Drives “Workhorse” of modern storage systems Capacity increasing, raw price dropping can buy 1TB for only $1000! bandwidth not keeping pace reliability is actually decreasing massive systems can mean even lower availability Majority of cost of ownership in administration, not purchase price backup, configuration, failure recovery It is important for us to note that instead of increasing slowly, disk reliability is actually decreasing. We could put here that ¼ TB is now $250. That is a truly incredible thing.

Disk Architecture spindle cylinder sector track platters arms with read/write heads rotation

Disk Storage Density

Disk Capacity Growth

IBM Disk Storage Roadmap

Storage Costs

RAID Redundant Arrays of Inexpensive Disks Two orthogonal concepts: data striping for performance redundancy for reliability Striped arrays can increase performance, but at the cost of reliability (next page) redundancy can give arrays better reliability than an individual disk

Reliability of Striped Array

One-month Trace of Hardware Failures Here’s a graph showing … … As we can see, disk failure problem is really serious. My proposed topic is to make storage systems more reliable. The key problems, Trace collected from the Internet Archive (March 2003) (thanks Kelly Gottlib) -- Over 100 terabytes of compressed data -- 30 disk failures out of total 70 hardware problems

RAID Levels

RAID Levels 1 2 3 4 5 6

RAID: 4x Small Write Penalty small data write xor 3 4 1 2 5

Log-Structured File Systems Based on assumption that disk traffic will become dominated by writes Always writes disk data sequentially, into next available location on disk no seeks on write Eliminates problem of 4x write penalty all writes are “new”, no need to read old data or parity However, almost no examples in industry file systems

Tape

Tape Media Inherently sequential Subject to mechanical stress long time to first byte no random I/O Subject to mechanical stress number of read-write cycles lower than disk Problems as an archival medium: readers go away after some years most rapidly in recent years tapes (with data) remain in a salt mine 4/01 1/03 9/05 1/08 LTO Cap 100 200 400 800 GB/cart. (uncompr.) Perf 15 35 60 >100 MB/Sec transfer rate MagStar Cap 60 300 ~600 1TB Perf. 14 40 ~70 >100

Tape Media Density will always trail that of disk Tape stretches, more difficult to get higher density Alignment also an issue once it’s past the head, it’s gone more conservative techniques required Bottom line: mechanical engineering issues for tape are the difficult ones OK, so I did already say most of that. Sorry. But it’s not the media that is the sole issue, it is the mechanical control system that has to align and handle the media as well.

Optical CD, CD-R/RW, DVD, DVD-R/RW Capacities: CD: ~700MB (huge 20 years ago!) DVD: single sided, single layer: 5GB single sided, double layer: 9GB double sided, single layer: 10GB double sided, double layer: 18GB Size of cell limited by wavelength of light current lasers are red blue lasers are under development, then UV, ... List densities. CD = 700MB, DVD = 4GB, there has been a lot of discussion about magneto optical. Terastor, for example, was going to do great things. The big idea is that the laser heats the material and makes it easier to change the magnetization. This may make a return in the guise of HAMR (heat assisted magnetic recording), Seagate is working on it. The limiting factor for purely optical systems is the wavelength of the light. You can’t focus a spot smaller than the wavelength. CD uses a red laser, and I think that DVD does as well. Newer systems will use blue lasers and increase density significantly. We can then think about moving to UV light and beyond.

Optical Magneto-optical (HAMR) heat from laser makes changing direction of magnetization easier (so cell is smaller)

MEMS MicroElectroMechanical Systems 6-10 times faster than disk cost and capacity issues CMU wants to do orthogonal magnetic recording, which means the bits get written deep instead of side (longitudinal). HP wants to use electron beams to change the polarization of a polymer. IBM uses physical depressions in a polymer, like an old vinyl record. It’s called millipede.

Magnetic RAM (MRAM) Stores each bit in a magnetic cell rather than a capacitor or flip-flop data is persistent Can be read and written very quickly Read and write times 0.5 – 10 µs or less Individual bits are writeable (no block erase) Density & cost comparable to DRAM may require density/speed tradeoffs denser MRAM may have to run slower because of heat dissipation on writes Note: Space Shuttle Challenger cores readable after recovery from ocean floor! Companies: IBM, HP, LLNL, others

Magnetic RAM (MRAM) Several companies have announced partnerships to produce products ~2003 Ideas for use of MRAM in storage: Persistent cache Hot data in MRAM, cold data to disk No need to flush write cache to avoid data loss HeRMES all metadata in MRAM enough file data in MRAM to hide disk latency for first access to a file

Peripheral Buses SCSI IDE/ATA HIPPI (High Performance Parallel Intf.) IEEE 1394 (FireWire) FibreChannel (FCP) IP (e.g., iSCSI) InfiniBand Serial ATA

Peripheral Buses Parallel Serial SCSI, most printers, IBM Channels 1 or more bytes per clock Skew problems at high speeds Serial FC, RS232, IEEE1394 (FireWire) 1 bit per clock, self clocking can be run at much higher speeds than parallel bus

Networked Storage Storage attached by general-purpose or dedicated network (e.g., FibreChannel) Motivations: homogenous and heterogeneous file sharing centralized administration better resource utilization (shared storage resources, pooling) Dedicated Networks: Fibre-Channel: FCP (SCSI over FC) iSCSI: SCSI over IP InfiniBand

Networked Storage Can mean many things: NAS (Network-Attached Storage): file server appliances serving NFS and/or CIFS (for example, Network Appliance) NASD (Network-Attached Secure Disk): intelligent, network-attached drives w/ security features (also, Network-Attached Storage Device) SAN (Storage Area Network): network for attaching disks and computers, usually dedicated only to storage operations OBSD (Object-Based Storage Device): similar to NASD

A SAN File System SAN Win2K AIX Solaris Linux Meta-data Server NFS CIFS FTP HTTP Control Network (IP) IFS w/cache Win2K IFS w/cache AIX IFS w/cache Solaris IFS w/cache Linux Meta-data Server Meta-data Server Meta- data SAN Meta-data Server data Security assists Storage Management Server HSM & Backup Data Data

Additional Reading Hennessy & Patterson: Chapter 6 Chen, Lee, Gibson, Katz, & Patterson: RAID: high performance, reliable secondary storage. ACM Computing Surveys 26, June 1994, 145-185 Rosenblum & Ousterhout: The design and implementation of a log-structured file system. ACM Transactions on Computer Systems, Feb. 1992, 26-52 Gibson, Nagle, et al.: A cost-effective, high-bandwidth storage architecture. Proceedings of the Eight Conference on Architectural Support for Programming Languages and Operating Systems, 1998 http://www.almaden.ibm.com/cs/storagesystems/stortank/