1. CE202 December 2, 2003 David Pease
Storage Architecture
CE202
December 2, 2003
David Pease

2. Hierarchy of Storage Faster Slower Smaller Larger Higher Lower RAM
Speed
Smaller
Larger
Capacity
Higher
Lower
Cost
RAM
Disk
Optical
Tape
Cache

3. Storage System Components
Application
I/O Library
File System
Device Driver
Host Bus Adapter
Interconnect
Storage Controller
Devices I/O Context

4. Disks
Microdrive, SCSI disk, RAMAC, and what’s the fourth thing?

5. Disk Drives “Workhorse” of modern storage systems
Capacity increasing, raw price dropping
can buy 1TB for only $1000!
bandwidth not keeping pace
reliability is actually decreasing
massive systems can mean even lower availability
Majority of cost of ownership in administration, not purchase price
backup, configuration, failure recovery
It is important for us to note that instead of increasing slowly, disk reliability is actually decreasing.
We could put here that ¼ TB is now $250. That is a truly incredible thing.

6. Disk Architecture spindle cylinder sector track platters arms with
read/write
heads
rotation

7. Disk Storage Density

8. Disk Capacity Growth

9. IBM Disk Storage Roadmap

10. Storage Costs

11. RAID Redundant Arrays of Inexpensive Disks Two orthogonal concepts:
data striping for performance
redundancy for reliability
Striped arrays can increase performance, but at the cost of reliability (next page)
redundancy can give arrays better reliability than an individual disk

12. Reliability of Striped Array

13. One-month Trace of Hardware Failures
Here’s a graph showing … …
As we can see, disk failure problem is really serious.
My proposed topic is to make storage systems more reliable. The key problems,
Trace collected from the Internet Archive (March 2003)
(thanks Kelly Gottlib)
-- Over 100 terabytes of compressed data
-- 30 disk failures out of total 70 hardware problems

14. RAID Levels

15. RAID Levels 1 2 3 4 5 6

16. RAID: 4x Small Write Penalty
small data write
xor 3 4 1 2 5

17. Log-Structured File Systems
Based on assumption that disk traffic will become dominated by writes
Always writes disk data sequentially, into next available location on disk
no seeks on write
Eliminates problem of 4x write penalty
all writes are “new”, no need to read old data or parity
However, almost no examples in industry file systems

18. Tape

19. Tape Media Inherently sequential Subject to mechanical stress
long time to first byte
no random I/O
Subject to mechanical stress
number of read-write cycles lower than disk
Problems as an archival medium:
readers go away after some years
most rapidly in recent years
tapes (with data) remain in a salt mine
4/ / / /08
LTO
Cap GB/cart. (uncompr.)
Perf > MB/Sec transfer rate
MagStar
Cap ~ TB
Perf ~ >100

20. Tape Media Density will always trail that of disk
Tape stretches, more difficult to get higher density
Alignment also an issue
once it’s past the head, it’s gone
more conservative techniques required
Bottom line: mechanical engineering issues for tape are the difficult ones
OK, so I did already say most of that. Sorry. But it’s not the media that is the sole issue, it is the mechanical control system that has to align and handle the media as well.

21. Optical CD, CD-R/RW, DVD, DVD-R/RW Capacities:
CD: ~700MB (huge 20 years ago!)
DVD:
single sided, single layer: 5GB
single sided, double layer: 9GB
double sided, single layer: 10GB
double sided, double layer: 18GB
Size of cell limited by wavelength of light
current lasers are red
blue lasers are under development, then UV, ...
List densities. CD = 700MB, DVD = 4GB, there has been a lot of discussion about magneto optical. Terastor, for example, was going to do great things. The big idea is that the laser heats the material and makes it easier to change the magnetization. This may make a return in the guise of HAMR (heat assisted magnetic recording), Seagate is working on it.
The limiting factor for purely optical systems is the wavelength of the light. You can’t focus a spot smaller than the wavelength. CD uses a red laser, and I think that DVD does as well. Newer systems will use blue lasers and increase density significantly. We can then think about moving to UV light and beyond.

22. Optical Magneto-optical (HAMR)
heat from laser makes changing direction of magnetization easier (so cell is smaller)

23. MEMS MicroElectroMechanical Systems 6-10 times faster than disk
cost and capacity issues
CMU wants to do orthogonal magnetic recording, which means the bits get written deep instead of side (longitudinal).
HP wants to use electron beams to change the polarization of a polymer.
IBM uses physical depressions in a polymer, like an old vinyl record. It’s called millipede.

24. Magnetic RAM (MRAM)
Stores each bit in a magnetic cell rather than a capacitor or flip-flop
data is persistent
Can be read and written very quickly
Read and write times 0.5 – 10 µs or less
Individual bits are writeable (no block erase)
Density & cost comparable to DRAM
may require density/speed tradeoffs
denser MRAM may have to run slower because of heat dissipation on writes
Note: Space Shuttle Challenger cores readable after recovery from ocean floor!
Companies: IBM, HP, LLNL, others

25. Magnetic RAM (MRAM)
Several companies have announced partnerships to produce products ~2003
Ideas for use of MRAM in storage:
Persistent cache
Hot data in MRAM, cold data to disk
No need to flush write cache to avoid data loss
HeRMES
all metadata in MRAM
enough file data in MRAM to hide disk latency for first access to a file

26. Peripheral Buses SCSI IDE/ATA HIPPI (High Performance Parallel Intf.)
IEEE 1394 (FireWire)
FibreChannel (FCP)
IP (e.g., iSCSI)
InfiniBand
Serial ATA

27. Peripheral Buses Parallel Serial SCSI, most printers, IBM Channels
1 or more bytes per clock
Skew problems at high speeds
Serial
FC, RS232, IEEE1394 (FireWire)
1 bit per clock, self clocking
can be run at much higher speeds than parallel bus

28. Networked Storage
Storage attached by general-purpose or dedicated network (e.g., FibreChannel)
Motivations:
homogenous and heterogeneous file sharing
centralized administration
better resource utilization (shared storage resources, pooling)
Dedicated Networks:
Fibre-Channel: FCP (SCSI over FC)
iSCSI: SCSI over IP
InfiniBand

29. Networked Storage Can mean many things:
NAS (Network-Attached Storage): file server appliances serving NFS and/or CIFS (for example, Network Appliance)
NASD (Network-Attached Secure Disk): intelligent, network-attached drives w/ security features (also, Network-Attached Storage Device)
SAN (Storage Area Network): network for attaching disks and computers, usually dedicated only to storage operations
OBSD (Object-Based Storage Device): similar to NASD

30. A SAN File System SAN Win2K AIX Solaris Linux Meta-data Server
NFS
CIFS
FTP
HTTP
Control Network (IP)
IFS w/cache
Win2K
IFS w/cache
AIX
IFS w/cache
Solaris
IFS w/cache
Linux
Meta-data
Server
Meta-data
Server
Meta-
data
SAN
Meta-data
Server
data
Security
assists
Storage
Management
Server
HSM &
Backup
Data
Data

31. Additional Reading Hennessy & Patterson: Chapter 6
Chen, Lee, Gibson, Katz, & Patterson: RAID: high performance, reliable secondary storage. ACM Computing Surveys 26, June 1994,
Rosenblum & Ousterhout: The design and implementation of a log-structured file system. ACM Transactions on Computer Systems, Feb. 1992, 26-52
Gibson, Nagle, et al.: A cost-effective, high-bandwidth storage architecture. Proceedings of the Eight Conference on Architectural Support for Programming Languages and Operating Systems, 1998