1. COMP 740: Computer Architecture and Implementation
Montek Singh
Nov 28, 2016
Topic: Storage Systems (Disk Technology); I/O

2. Disk Systems: Characteristics
Capacity [bytes]
Mainframes typically have 3.7 GB of disk storage per MIPS
PCs tend to require 5 MB of disk storage per MIPS
Bandwidth (throughput)
Bytes transferred per unit time
Service rate
Number of service requests satisfied per unit time
Supercomputer I/O requests tend to involve large amounts of data; transaction processing systems tend to involve small ones
Response time (latency)
Time between start and completion of an event
Cost [$/MB]

3. Historical Perspective
1956 IBM Ramac — early 1970s Winchester
For mainframe computers, proprietary interfaces
Steady shrink in form factor: 27 in. to 14 in.
Form factor and capacity drives market more than perf.
1970s developments
5.25 inch floppy disk form factor
Emergence of industry standard disk interfaces
Early 1980s: PCs and first generation workstations
Mid 1980s: Client/server computing
Centralized storage on file server
Disk downsizing: 8 inch to 5.25
Mass market disk drives become a reality
industry standards: SCSI, IPI, IDE
5.25 inch to 3.5 inch drives for PCs; End of proprietary interfaces
1990s: Laptops => 2.5 inch drives
2000s: 1.8” used in media players (1” microdrive didn’t do as well)
2010s: Flash and solid-state drives (SSD)

4. Disk Figure of Merit: Areal Density
Bits recorded along a track
Metric is Bits Per Inch (BPI)
Number of tracks per surface
Metric is Tracks Per Inch (TPI)
Disk metric is bit density per unit area
Metric is Bits Per Square Inch: Areal Density = BPI x TPI

5. Example: Parameters of A Single Disk Drive

6. Technical Details (1) Platters Rotation
2-4mm thick, made of an aluminum alloy
Typically 1-6 platters in a hard disk
Both sides of each platter generally used
One side of one platter dedicated to information to guide the servomechanisms that control speed of rotation and head movement
Each recording surface has its own read-write head
Only one operational at any time
Data transfer to and from disk is bit-serial
Rotation
Speed of rotation carefully controlled by servomechanism
E.g., 7200 rpm 1% or even 1 rpm
All disk drives are synchronized in some disk arrays
Angular positions are identical (within tolerance) at any time

7. Technical Details (2) Flying height “Parking the heads”
Read/write heads do not touch rotating disk surface
Careful aerodynamic design keeps small constant distance (0.2 m)
Disk moves with approximate linear speed of 700 inches/sec
18m/s, 40 miles/hour
Smaller flying height  higher writing density
Hence, platters and heads sealed hermetically in clean space
Called Winchester drives for historical reasons
“Parking the heads”
Before disk is allowed to slow down and stop, heads are moved over an area of the disk not used for recording, where they are allowed to come into contact with the disk surface

8. Why Disks Are Bit-Serial
High TPI value makes parallel access disks unworkable
Reading heads move as rigid unit
One of them (the one over the servo disk) supposedly defines radial (track) position
Ensemble is not truly rigid, and various reasons (like thermal dilations) prevent all heads being positioned over same track simultaneously, repeatedly, and reliably
Only one head is positioned accurately at a time: servo guides the assembly to approximate position of requested track, reading head does final positioning using a high frequency signal between data tracks
Sector ID always contains track number for confirmation
Portable disks use shock sensors to prevent overwriting of adjacent tracks caused by jarring of R/W head
Bit stored in an approximate trapezoid whose sides are 0.3m and 7m

9. Track Densities
Until recently, only innermost track was recorded with maximum density
All other tracks contained same number of sectors and bytes
Recently, manufacturers are using bands of tracks
Total number of tracks (100s-1000s) divided into bands or zones (4, 8, 16, …) each containing the same number of tracks
Each band has innermost track recorded at maximum density, with other tracks having same capacity
Greatest capacity gains occur with a small number of bands
Two scheduling options
Constant rotational speed, use buffer for speed matching
Constant data rate, head spinning with rotational speed corresponding to recording density of zone

10. Sector Format
Sector is the smallest unit of data that can be read or written
Typically between 32B and 4KB, with 512B being a common size
Format of sector
ID: Identifies sector with information such as angular position and track #
Has its own error correcting code (ECC)
GAP: Permits electronics to process ECC information
DATA and its ECC ID
ECC
GAP
DATA
Total sector size
is measure of formatted capacity of disk
as fraction of nominal capacity

11. Disk Miscellanea Smaller disks tend to be more cost-effective
Smaller inertia, lower power consumption per megabyte, shorter seek distances, less vibration, less heat generated
Heat generation proportional to NRPM2.8 D4.6
N = number of platters
D = diameter
RPM = rotational speed
Disk access time = seek time + rotational latency + transfer time
Typical values: ms seek time, 8.3 ms rotational latency
Disk growth rates
Disk areal density doubling every three years
Disk transfer rate doubling every five years
Disk access time halving every ten years

12. Ensembles of Disks Key idea Used in mainframes for a long time RAID
Use collection of disk drives to improve characteristics of disk systems (storage capacity, bandwidth, etc.)
Used in mainframes for a long time
RAID
Redundant Array of Inexpensive Disks (original 1988 acronym)
Redundant Array of Independent Disks (redefined in 1992)

13. Improving Bandwidth with Disk Arrays
Arrays of independent disk drives
Similar to high-order interleaving in main memories
Each file assigned to different disk drive
Simultaneous access to files
Load balancing issues
File Stripeing/disk Stripeing/disk interleaving
Single file distributed across array of disks
Similar to low-order interleaving in main memories
Each logical I/O request corresponds to a data Stripe
Data Stripe divided into number of equal sized Stripe units
Stripe units assigned to different disk units
Two kinds of Stripeing depending on size of Stripe unit
Fine-grained Stripeing: Stripe unit chosen to balance load
Coarse-grained Stripeing: Larger Stripe unit

14. Improving Availability with Disk Arrays
MTTF of large-system disks approaches 106 hours
MTTF of PC-class disks approaches 150,000 hours
But array of 1,000 PC-class disks has MTTF of only 150 hours
All schemes to cope with low MTTF aim to “fail soft”
Operation should be able to continue while repair is made
Always depends on some form of redundancy

15. RAID-0 Striped, non-redundant
Parallel access to multiple disks
Excellent data transfer rate (for small Stripes)
Excellent I/O request processing rate (for large Stripes)
Typically used for applications requiring high performance for non-critical data

16. RAID-1 Mirrored/replicated (most costly form of redundancy)
Stripe 3
Stripe 2
Stripe 1
Stripe 0
Stripe 3
Stripe 2
Stripe 1
Stripe 0
Mirrored/replicated (most costly form of redundancy)
I/O request rate: good for reads, fair for writes
Data transfer rate: good for reads; writes slightly slower
Read can be serviced by the disk with the shorter seek distance
Write must be handled by both disks
Typically used in system drives and critical files
Banking, insurance data
Web (e-commerce) servers

17. Combining RAID-0 and RAID-1
Stripe 12
Stripe 8
Stripe 4
Stripe 0
Stripe 13
Stripe 9
Stripe 5
Stripe 1
Stripe 14
Stripe 10
Stripe 6
Stripe 2
Stripe 15
Stripe 11
Stripe 7
Stripe 3
Stripe 12
Stripe 8
Stripe 4
Stripe 0
Stripe 13
Stripe 9
Stripe 5
Stripe 1
Stripe 14
Stripe 10
Stripe 6
Stripe 2
Stripe 15
Stripe 11
Stripe 7
Stripe 3
Can combine RAID-0 and RAID-1:
Mirrored Stripes (RAID 0+1, or RAID 01)
Example: picture above
Striped Mirrors (RAID 1+0, or RAID 10)
Data transfer rate: good for reads and writes
Reliability: good
Efficiency: poor (100% overhead in terms of disk utilization)

18. RAID-3 b0 b1 b2 b3 P(b) Fine-grained (bit) interleaving with parity
E.g., parity = sum modulo 2 (XOR) of all bits
Disks are synchronized, parity computed by disk controller
When one disk fails… (how do you know?)
Data is recovered by subtracting all data in good disks from parity disk
Recovering from failures takes longer than in mirroring, but failures are rare, so is okay
Hot spares used to reduce vulnerability in reduced mode
Performance:
Poor I/O request rate
Excellent data transfer rate
Typically used in large I/O request size applications, such as imaging or CAD

19. RAID-2 Hamming codes capable of correcting two or more erasures
f0(b)
f1(b)
f2(b)
Hamming codes capable of correcting two or more erasures
E.g., single error-correcting, double error-detecting (SEC-DED)
Problem with small writes (similar to DRAM cycle time)
Poor I/O request rate
Excellent data transfer rate

20. RAID-4 Blk 12 Blk 8 Blk 4 Blk 0 Blk 13 Blk 9 Blk 5 Blk 1 Blk 14 Blk 10
P(12-15)
P(8-11)
P(4-7)
P(0-3)
Coarse-grained Striping with parity
Unlike RAID-3, not all disks need to be read on each write
New parity computed by computing difference between old and new data
Drawback:
Like RAID-3, parity disk involved in every write; serializes small reads
I/O request rate: excellent for reads, fair for writes
Data transfer rate: good for reads, fair for writes

21. RAID-5 Blk 12 Blk 8 Blk 4 Blk 0 P(12-15) Blk 9 Blk 5 Blk 1 Blk 13
Key Idea: reduce load on parity disk
Block-interleaved distributed parity
Multiple writes can occur simultaneously
Block 0 can be accessed in parallel with Block 5
First needs disks 1 and 5; second needs disks 2 and 4
I/O request rate: excellent for reads, good for writes
Data transfer rate: good for reads, good for writes
Typically used for high request rate, read-intensive data lookup

22. Removable Media Magnetic Tapes: Optical Disks: CD’s and DVD’s
Have been used in computer systems as long as disks
Similar magnetic technology as disks
have followed similar density improvements as disks
Long Stripes wound on removable spools
Benefits: can be essentially of “unlimited” length; removable
Drawbacks: only offer sequential access; wear out
Used for archival backups; but on the way out
Optical Disks: CD’s and DVD’s
CD-ROM/DVD-ROM: removable, inexpensive, random access
CD-R/RW, DVD-R/RAM: writable/re-writable
Becoming immensely popular: replacing floppy drives, tape drives
Flash: flash cards, memory sticks, digital film, SSDs
Microdrives: small portable hard disks (up to 1GB)

23. Interfacing I/O Subsystem to CPU
Where do we connect I/O devices?
Cache?
Memory? 
Low-cost systems: I/O bus = memory bus!
CPU and I/O compete for bus
How does CPU address I/O devs.?
Memory-mapped I/O
Portions of address space assigned to I/O devices
I/O opcodes (not very popular)
How does CPU synch with I/O?
Polling vs. interrupt-driven
Real-time systems: hybrid
Clock interrupts periodically
CPU polls during interrupt

24. Bus: Different Options
Bus width
High-perf: separate address and data lines
Low-cost: multiplex address and data lines
Data width
High-perf: wider is faster (e.g., 64 bits)
Low-cost: narrower is cheaper (e.g., 8 bits)
Transfer size
High-perf: multiple words have less bus overhead
Low-cost: single-word transfer is simpler
Bus masters
High-perf: multiple (requires arbitration)
Low-cost: single master (simpler)
Split transaction
High-perf: Yes (request and reply in separate “packets”)
Low-cost: No (continuous connection is cheaper)
Clocking
High-perf: synchronous (though it is changing)
Low-cost: asynchronous

25. Examples of Bus Standards
How about SATA?
IDE/Ultra ATA
SCSI
PCI
PCI-X
Data width
16 bits
8 or 16 bits
32 or 64 bits
Clock rate
Up to 100 MHz
MHz
33-66 MHz
MHz
# Bus masters 1
Multiple
Peak Bandwidth
200 MB/sec
320 MB/sec
533 MB/sec
1066 MB/sec
Clocking
Async
Sync

26. Does I/O Performance Matter?
The “No” Argument:
“There is always another process to run while one process waits for I/O to complete”
Counter-arguments:
Above argument applies only if throughput is sole goal
Interactive software and personal computers lay more emphasis on response time (=latency), not throughput
Interactive multimedia, transaction processing
Process switching actually increases I/O (paging traffic)
Desktop/Mobile computing: only one person/computer  fewer processes than in time-sharing
Amdahl’s Law: benefits of making CPU faster taper off if I/O becomes the bottleneck

27. Does CPU Performance Matter?
Consequences of Moore’s Law:
Large, fast CPU’s
Small, cheap CPU’s
Main goal: keeping I/O devices busy, not keeping CPU busy
bulk of the hardware cost is in I/O peripherals, not CPU
Shift in focus:
From computation to communication
Reflected in change of terminology:
1960’s-80’s: Computing Revolution
1990’s-present: Information Age

28. Does Performance Matter?
Performance is not the problem it once was!
15 years of doubling CPU speed every 18 months (x1000)
Disks also have steadily become bigger and faster
Most people would prefer more reliability than speed:
Today’s speeds come at a cost: frequent crashes
Program crashes  frustration
Disk crashes  hysteria
Reliability, availability, dependability, etc. are the key terms
Client-server model of computing has made reliability the key criterion for evaluation
E.g., UNC-COMPSCI IMAP server (from 1990s to 2010 or so):
300 MHz or something, 99.9… % uptime