1. Slide 1 ISTORE: A server for the PostPC Era Aaron Brown, Dave Martin, David Oppenheimer, Noah Trauhaft, Dave Patterson,Katherine Yelick University of California at Berkeley Patterson@cs.berkeley.edu UC Berkeley ISTORE Group istore-group@cs.berkeley.edu October 2000

2. Slide 2 ISTORE as Storage System of the Future Availability, Maintainability, and Evolutionary growth key challenges for storage systems –Maintenance Cost ~ >10X Purchase Cost per year, –Even 2X purchase cost for 1/2 maintenance cost wins –AME improvement enables even larger systems ISTORE also cost-performance advantages –Better space, power/cooling costs ($@colocation site) –More MIPS, cheaper MIPS, no bus bottlenecks –Compression reduces network $, encryption protects –Single interconnect, supports evolution of technology, single network technology to maintain/understand Match to future software storage services –Future storage service software target clusters

3. Slide 3 Lampson: Systems Challenges Systems that work –Meeting their specs –Always available –Adapting to changing environment –Evolving while they run –Made from unreliable components –Growing without practical limit Credible simulations or analysis Writing good specs Testing Performance –Understanding when it doesn’t matter “Computer Systems Research -Past and Future” Keynote address, 17th SOSP, Dec. 1999 Butler Lampson Microsoft

4. Slide 4 Jim Gray: Trouble-Free Systems Manager –Sets goals –Sets policy –Sets budget –System does the rest. Everyone is a CIO (Chief Information Officer) Build a system –used by millions of people each day –Administered and managed by a ½ time person. »On hardware fault, order replacement part »On overload, order additional equipment »Upgrade hardware and software automatically. “What Next? A dozen remaining IT problems” Turing Award Lecture, FCRC, May 1999 Jim Gray Microsoft

5. Slide 5 Jim Gray: Trustworthy Systems Build a system used by millions of people that –Only services authorized users »Service cannot be denied (can’t destroy data or power). »Information cannot be stolen. –Is always available: (out less than 1 second per 100 years = 8 9’s of availability) »1950’s 90% availability, Today 99% uptime for web sites, 99.99% for well managed sites (50 minutes/year) 3 extra 9s in 45 years. »Goal: 5 more 9s: 1 second per century. –And prove it.

6. Slide 6 Hennessy: What Should the “New World” Focus Be? Availability –Both appliance & service Maintainability –Two functions: »Enhancing availability by preventing failure »Ease of SW and HW upgrades Scalability –Especially of service Cost –per device and per service transaction Performance –Remains important, but its not SPECint “Back to the Future: Time to Return to Longstanding Problems in Computer Systems?” Keynote address, FCRC, May 1999 John Hennessy Stanford

7. Slide 7 The real scalability problems: AME Availability –systems should continue to meet quality of service goals despite hardware and software failures Maintainability –systems should require only minimal ongoing human administration, regardless of scale or complexity: Today, cost of maintenance = 10-100 cost of purchase Evolutionary Growth –systems should evolve gracefully in terms of performance, maintainability, and availability as they are grown/upgraded/expanded These are problems at today’s scales, and will only get worse as systems grow

8. Slide 8 VAX crashes ‘85, ‘93 [Murp95]; extrap. to ‘01 Sys. Man.: N crashes/problem, SysAdmin action –Actions: set params bad, bad config, bad app install HW/OS 70% in ‘85 to 28% in ‘93. In ‘01, 10%? Rule of Thumb: Maintenance 10X HW –so over 5 year product life, ~ 95% of cost is maintenance Is Maintenance the Key?

9. Slide 9 Principles for achieving AME No single points of failure, lots of redundancy Performance robustness is more important than peak performance Performance can be sacrificed for improvements in AME –resources should be dedicated to AME »biological systems > 50% of resources on maintenance –can make up performance by scaling system Introspection –reactive techniques to detect and adapt to failures, workload variations, and system evolution –proactive techniques to anticipate and avert problems before they happen

10. Slide 10 Hardware Techniques (1): SON SON: Storage Oriented Nodes Distribute processing with storage –If AME really important, provide resources! –Most storage servers limited by speed of CPUs!! –Amortize sheet metal, power, cooling, network for disk to add processor, memory, and a real network? –Embedded processors 2/3 perf, 1/10 cost, power? –Serial lines, switches also growing with Moore’s Law; less need today to centralize vs. bus oriented systems Advantages of cluster organization –Truly scalable architecture –Architecture that tolerates partial failure –Automatic hardware redundancy

11. Slide 11 Hardware techniques (2) Heavily instrumented hardware –sensors for temp, vibration, humidity, power, intrusion –helps detect environmental problems before they can affect system integrity Independent diagnostic processor on each node –provides remote control of power, remote console access to the node, selection of node boot code –collects, stores, processes environmental data for abnormalities –non-volatile “flight recorder” functionality –all diagnostic processors connected via independent diagnostic network

12. Slide 12 Hardware techniques (3) On-demand network partitioning/isolation –Internet applications must remain available despite failures of components, therefore can isolate a subset for preventative maintenance –Allows testing, repair of online system –Managed by diagnostic processor and network switches via diagnostic network Built-in fault injection capabilities –Power control to individual node components –Injectable glitches into I/O and memory busses –Managed by diagnostic processor –Used for proactive hardware introspection »automated detection of flaky components »controlled testing of error-recovery mechanisms

13. Slide 13 “Hardware” culture (4) Benchmarking –One reason for 1000X processor performance was ability to measure (vs. debate) which is better »e.g., Which most important to improve: clock rate, clocks per instruction, or instructions executed? –Need AME benchmarks “what gets measured gets done” “benchmarks shape a field” “quantification brings rigor”

14. Slide 14 Example single-fault result Compares Linux and Solaris reconstruction –Linux: minimal performance impact but longer window of vulnerability to second fault –Solaris: large perf. impact but restores redundancy fast Linux Solaris

15. Slide 15 Deriving ISTORE What is the interconnect? –FC-AL? (Interoperability? Cost of switches?) –Infiniband? (When? Cost of switches? Cost of NIC?) –Gbit Ehthernet? Pick Gbit Ethernet as commodity switch, link –As main stream, fastest improving in cost performance –We assume Gbit Ethernet switches will get cheap over time (Network Processors, volume, …)

16. Slide 16 Deriving ISTORE Number of Disks / Gbit port? Bandwidth of 2000 disk –Raw bit rate: 427 Mbit/sec. –Data transfer rate: 40.2 MByte/sec –Capacity: 73.4 GB Disk trends –BW: 40%/year –Capacity, Areal density,$/MB: 100%/year 2003 disks –~ 500 GB capacity (<8X) –~ 110 MB/sec or 0.9 Gbit/sec (2.75X) Number of Disks / Gbit port = 1

17. Slide 17 ISTORE-1 Brick Webster’s Dictionary: “brick: a handy-sized unit of building or paving material typically being rectangular and about 2 1/4 x 3 3/4 x 8 inches” ISTORE-1 Brick: 2 x 4 x 11 inches (1.3x) –Single physical form factor, fixed cooling required, compatible network interface to simplify physical maintenance, scaling over time –Contents should evolve over time: contains most cost effective MPU, DRAM, disk, compatible NI –If useful, could have special bricks (e.g., DRAM rich) –Suggests network that will last, evolve: Ethernet

18. Slide 18 ISTORE-1 hardware platform 80-node x86-based cluster, 1.4TB storage –cluster nodes are plug-and-play, intelligent, network- attached storage “bricks” »a single field-replaceable unit to simplify maintenance –each node is a full x86 PC w/256MB DRAM, 18GB disk –more CPU than NAS; fewer disks/node than cluster Intelligent Disk “Brick” Portable PC CPU: Pentium II/266 + DRAM Redundant NICs (4 100 Mb/s links) Diagnostic Processor Disk Half-height canister ISTORE Chassis 80 nodes, 8 per tray 2 levels of switches 20 100 Mbit/s 2 1 Gbit/s Environment Monitoring: UPS, redundant PS, fans, heat and vibration sensors...

19. Slide 19 Common Question: RAID? Switched Network sufficient for all types of communication, including redundancy –Hierarchy of buses is generally not superior to switched network Veritas, others offer software RAID 5 and software Mirroring (RAID 1) Another use of processor per disk

20. Slide 20 A Case for Intelligent Storage Advantages: Cost of Bandwidth Cost of Space Cost of Storage System v. Cost of Disks Physical Repair, Number of Spare Parts Cost of Processor Complexity Cluster advantages: dependability, scalability 1 v. 2 Networks

21. Slide 21 Cost of Space, Power, Bandwidth Co-location sites (e.g., Exodus) offer space, expandable bandwidth, stable power Charge ~$1000/month per rack (~ 10 sq. ft.) –Includes 1 20-amp circuit/rack; charges ~$100/month per extra 20-amp circuit/rack Bandwidth cost: ~$500 per Mbit/sec/Month

22. Slide 22 Cost of Bandwidth, Safety Network bandwidth cost is significant –1000 Mbit/sec/month => $6,000,000/year Security will increase in importance for storage service providers XML => server format conversion for gadgets => Storage systems of future need greater computing ability –Compress to reduce cost of network bandwidth 3X; save $4M/year? –Encrypt to protect information in transit for B2B => Increasing processing/disk for future storage apps

23. Slide 23 Cost of Space, Power Sun Enterprise server/array (64CPUs/60disks) –10K Server (64 CPUs): 70 x 50 x 39 in. –A3500 Array (60 disks): 74 x 24 x 36 in. –2 Symmetra UPS (11KW): 2 * 52 x 24 x 27 in. ISTORE-1: 2X savings in space –ISTORE-1: 1 rack (big) switches, 1 rack (old) UPSs, 1 rack for 80 CPUs/disks (3/8 VME rack unit/brick) ISTORE-2: 8X-16X space? Space, power cost/year for 1000 disks: Sun $924k, ISTORE-1 $484k, ISTORE2 $50k

24. Slide 24 Cost of Storage System v. Disks Hardware RAID box ~ 5X cost of disks

25. Slide 25 Disk Limit: Bus Hierarchy CPU Memory bus Memory External I/O bus (SCSI) (PCI) Internal I/O bus Data rate vs. Disk rate –SCSI: Ultra3 (80 MHz), Wide (16 bit): 160 MByte/s –FC-AL: 1 Gbit/s = 125 MByte/s n Use only 50% of a bus m Command overhead (~ 20%) m Queuing Theory (< 70%) (15 disks/bus) Storage Area Network (FC-AL) Server Disk Array Mem RAID bus

26. Slide 26 Physical Repair, Spare Parts ISTORE: Compatible modules based on hot- pluggable interconnect (LAN) with few Field Replacable Units (FRUs): Node, Power Supplies, Switches, network cables –Replace node (disk, CPU, memory, NI) if any fail Conventional: Heterogeneous system with many server modules (CPU, backplane, memory cards, …) and disk array modules (controllers, disks, array controllers, power supplies, … ) –Store all components available somewhere as FRUs –Sun Enterprise 10k has ~ 100 types of spare parts –Sun 3500 Array has ~ 12 types of spare parts

27. Slide 27 ISTORE: Complexity v. Perf Complexity increase: –HP PA-8500: issue 4 instructions per clock cycle, 56 instructions out-of-order execution, 4Kbit branch predictor, 9 stage pipeline, 512 KB I cache, 1024 KB D cache (> 80M transistors just in caches) –Intel Xscale: 16 KB I$, 16 KB D$, 1 instruction, in order execution, no branch prediction, 6 stage pipeline Complexity costs in development time, development power, die size, cost –550 MHz HP PA-8500 477 mm 2, 0.25 micron/4M $330, 60 Watts –1000 MHz Intel StrongARM2 (“Xscale”) @ 1.5 Watts, 800 MHz at 0.9 W, … 50 Mhz @ 0.01W, 0.18 micron (old chip 50 mm 2, 0.35 micron, $18) => Count $ for system, not processors/disk

28. Slide 28 ISTORE: Cluster Advantages Architecture that tolerates partial failure Automatic hardware redundancy –Transparent to application programs Truly scalable architecture –Given maintenance is 10X-100X capital costs, cluster size limits today are maintenance, floor space cost - generally NOT capital costs As a result, it is THE target architecture for new software apps for Internet

29. Slide 29 ISTORE: 1 vs. 2 networks Current systems all have LAN + Disk interconnect (SCSI, FCAL) –LAN is improving fastest, most investment, most features –SCSI, FC-AL poor network features, improving slowly, relatively expensive for switches, bandwidth –FC-AL switches don’t interoperate –Two sets of cables, wiring? –SysAdmin trained in 2 networks, SW interface, …??? Why not single network based on best HW/SW technology? –Note: there can be still 2 instances of the network (e.g. external, internal), but only one technology

30. Slide 30 Initial Applications ISTORE-1 is not one super-system that demonstrates all these techniques! –Initially provide middleware, library to support AME Initial application targets –information retrieval for multimedia data (XML storage?) »self-scrubbing data structures, structuring performance-robust distributed computation »Example: home video server using XML interfaces –email service »self-scrubbing data structures, online self-testing »statistical identification of normal behavior

31. Slide 31 A glimpse into the future? System-on-a-chip enables computer, memory, redundant network interfaces without significantly increasing size of disk ISTORE HW in 5-7 years: –2006 brick: System On a Chip integrated with MicroDrive »9GB disk, 50 MB/sec from disk »connected via crossbar switch »From brick to “domino” –If low power, 10,000 nodes fit into one rack! O(10,000) scale is our ultimate design point

32. Slide 32 Conclusion: ISTORE as Storage System of the Future Availability, Maintainability, and Evolutionary growth key challenges for storage systems –Maintenance Cost ~ 10X Purchase Cost per year, so over 5 year product life, ~ 95% of cost of ownership –Even 2X purchase cost for 1/2 maintenance cost wins –AME improvement enables even larger systems ISTORE has cost-performance advantages –Better space, power/cooling costs ($@colocation site) –More MIPS, cheaper MIPS, no bus bottlenecks –Compression reduces network $, encryption protects –Single interconnect, supports evolution of technology, single network technology to maintain/understand Match to future software storage services –Future storage service software target clusters

33. Slide 33 Questions? Contact us if you’re interested: email: patterson@cs.berkeley.edu http://iram.cs.berkeley.edu/ “If it’s important, how can you say if it’s impossible if you don’t try?” Jean Morreau, a founder of European Union

34. Slide 34 Clusters and TPC Software 8/’00 TPC-C: 6 of Top 10 performance are clusters, including all of Top 5; 4 SMPs TPC-H: SMPs and NUMAs –100 GB All SMPs (4-8 CPUs) –300 GB All NUMAs (IBM/Compaq/HP 32-64 CPUs) TPC-R: All are clusters –1000 GB :NCR World Mark 5200 TPC-W: All web servers are clusters (IBM)

35. Slide 35 Clusters and TPC-C Benchmark Top 10 TPC-C Performance (Aug. 2000) Ktpm 1. Netfinity 8500R c/s Cluster441 2. ProLiant X700-96P Cluster262 3. ProLiant X550-96P Cluster230 4. ProLiant X700-64P Cluster180 5. ProLiant X550-64P Cluster162 6.AS/400e 840-2420SMP 152 7. FujitsuGP7000F Model 2000SMP 139 8. RISC S/6000 Ent. S80SMP139 9. BullEscala EPC 2400 c/sSMP 136 10.Enterprise 6500 ClusterCluster135

36. Slide 36 Cost of Storage System v. Disks Examples show cost of way we build current systems (2 networks, many buses, CPU, …) Disks Disks DateCostMain.Disks/CPU /IObus –NCR WM: 10/97$8.3M--131210.25.0 –Sun 10k: 3/98$5.2M--66810.47.0 –Sun 10k: 9/99$6.2M$2.1M173227.012.0 –IBM Netinf: 7/00$7.8M$1.8M704055.09.0 =>Too complicated, too heterogenous And Data Bases are often CPU or bus bound! –ISTORE disks per CPU: 1.0 –ISTORE disks per I/O bus: 1.0

37. Slide 37 Common Question: Why Not Vary Number of Processors and Disks? Argument: if can vary numbers of each to match application, more cost-effective solution? Alternative Model 1: Dual Nodes + E-switches –P-node: Processor, Memory, 2 Ethernet NICs –D-node: Disk, 2 Ethernet NICs Response –As D-nodes running network protocol, still need processor and memory, just smaller; how much save? –Saves processors/disks, costs more NICs/switches: N ISTORE nodes vs. N/2 P-nodes + N D-nodes –Isn't ISTORE-2 a good HW prototype for this model? Only run the communication protocol on N nodes, run the full app and OS on N/2

38. Slide 38 Common Question: Why Not Vary Number of Processors and Disks? Alternative Model 2: N Disks/node –Processor, Memory, N disks, 2 Ethernet NICs Response –Potential I/O bus bottleneck as disk BW grows –2.5" ATA drives are limited to 2/4 disks per ATA bus –How does a research project pick N? What’s natural? –Is there sufficient processing power and memory to run the AME monitoring and testing tasks as well as the application requirements? –Isn't ISTORE-2 a good HW prototype for this model? Software can act as simple disk interface over network and run a standard disk protocol, and then run that on N nodes per apps/OS node. Plenty of Network BW available in redundant switches

39. Slide 39 SCSI v. IDE $/GB Prices from PC Magazine, 1995-2000

40. Slide 40 Grove’s Warning “...a strategic inflection point is a time in the life of a business when its fundamentals are about to change.... Let's not mince words: A strategic inflection point can be deadly when unattended to. Companies that begin a decline as a result of its changes rarely recover their previous greatness.” Only the Paranoid Survive, Andrew S. Grove, 1996

41. Slide 41 Availability benchmark methodology Goal: quantify variation in QoS metrics as events occur that affect system availability Leverage existing performance benchmarks –to generate fair workloads –to measure & trace quality of service metrics Use fault injection to compromise system –hardware faults (disk, memory, network, power) –software faults (corrupt input, driver error returns) –maintenance events (repairs, SW/HW upgrades) Examine single-fault and multi-fault workloads –the availability analogues of performance micro- and macro-benchmarks

42. Slide 42 Results are most accessible graphically –plot change in QoS metrics over time –compare to “normal” behavior? »99% confidence intervals calculated from no-fault runs Benchmark Availability? Methodology for reporting results

43. Slide 43 ISTORE-2 Improvements (1): Operator Aids Every Field Replaceable Unit (FRU) has a machine readable unique identifier (UID) => introspective software determines if storage system is wired properly initially, evolved properly »Can a switch failure disconnect both copies of data? »Can a power supply failure disable mirrored disks? –Computer checks for wiring errors, informs operator vs. management blaming operator upon failure –Leverage IBM Vital Product Data (VPD) technology? External Status Lights per Brick –Disk active, Ethernet port active, Redundant HW active, HW failure, Software hickup,...

44. Slide 44 ISTORE-2 Improvements (2): RAIN ISTORE-1 switches 1/3 of space, power, cost, and for just 80 nodes! Redundant Array of Inexpensive Disks (RAID): replace large, expensive disks by many small, inexpensive disks, saving volume, power, cost Redundant Array of Inexpensive Network switches: replace large, expensive switches by many small, inexpensive switches, saving volume, power, cost? –ISTORE-1: Replace 2 16-port 1-Gbit switches by fat tree of 8 8-port switches, or 24 4-port switches?

45. Slide 45 ISTORE-2 Improvements (3): System Management Language Define high-level, intuitive, non-abstract system management language –Goal: Large Systems managed by part-time operators! Language interpretive for observation, but compiled, error-checked for config. changes Examples of tasks which should be made easy –Set alarm if any disk is more than 70% full –Backup all data in the Philippines site to Colorado site –Split system into protected subregions –Discover & display present routing topology –Show correlation between brick temps and crashes

46. Slide 46 ISTORE-2 Improvements (4): Options to Investigate TCP/IP Hardware Accelerator –Class 4: Hardware State Machine –~10 microsecond latency, full Gbit bandwidth + full TCP/IP functionality, TCP/IP APIs Ethernet Sourced in Memory Controller (North Bridge) Shelf of bricks on researchers’ desktops? SCSI over TCP Support Integrated UPS

47. Slide 47 IStore-2 Deltas from IStore-1 Geographically Disperse Nodes, Larger System –O(1000) nodes at Almaden, O(1000) at Berkeley –Bisect into two O(500) nodes per site to simplify space problems, to show evolution over time? Upgraded Storage Brick –Two Gbit Ethernet copper ports/brick Upgraded Packaging –32?/sliding tray vs. 8/shelf –User Supplied UPS Support –8X-16X density for ISTORE-2 vs. ISTORE-1

48. Slide 48 Why is ISTORE-2 a big machine? ISTORE is all about managing truly large systems - one needs a large system to discover the real issues and opportunities –target 1k nodes in UCB CS, 1k nodes in IBM ARC Large systems attract real applications –Without real applications CS research runs open-loop The geographical separation of ISTORE-2 sub-clusters exposes many important issues –the network is NOT transparent –networked systems fail differently, often insidiously

49. Slide 49 UCB ISTORE Continued Funding New NSF Information Technology Research, larger funding (>$500K/yr) 1400 Letters 920 Preproposals 134 Full Proposals Encouraged 240 Full Proposals Submitted 60 Funded We are 1 of the 60; starts Sept 2000

50. Slide 50 NSF ITR Collaboration with Mills Mills: small undergraduate liberal arts college for women; 8 miles south of Berkeley –Mills students can take 1 course/semester at Berkeley –Hourly shuttle between campuses –Mills also has re-entry MS program for older students To increase women in Computer Science (especially African-American women): –Offer undergraduate research seminar at Mills –Mills Prof leads; Berkeley faculty, grad students help –Mills Prof goes to Berkeley for meetings, sabbatical –Goal: 2X-3X increase in Mills CS+alumnae to grad school IBM people want to help? Helping teach, mentor...

51. Slide 51 Number of Disks for Blue Gene? According to NSIC Tape Roadmap, in 2003 a tape reader + 100 tapes = $1/GB Today a disk is $10/GB; in 2003, ~$1/GB Tape libraries seem bad choice future: unreliable, no longer cost competitive Blue Gene checkpoint every 20 minutes: –0.5 MB x 1M processors = 0.5 TB per snapshot –25,000 snapshots/year => 25,000 disks –Alternative calculation: 100 MB/sec for 1 year => 100 MB/s x 365 x 24 x 60 x 60 = 3150 TB => 6300 disks

52. Slide 52 Deriving ISTORE Implication of Ethernet network? Need computer associated with disk to handle network protocol stack Blue Gene I/O using a processor per disk? Compare checkpoints across multiple snapshots to reduce disks storage: 2X? 4X? 6X? –Reduce cost of purchase, cost of maintenance, size Anticipate disk failures: 25000  2 fail/day –Record history of sensor logs/disk –History allows 95% error prediction > 1 day in advance Check accuracy of snapshot? (Assertion tests) Help with maintenance: despite hope, likely many here will run it; expensive SysAdmin!