1 Put Everything in Future (Disk) Controllers (it’s not “if”, it’s “when?”) Jim Gray Acknowledgements : Dave Patterson explained this to me a year ago Kim Keeton Erik Riedel Catharine Van Ingen Helped me sharpen these arguments

2 Remember Your Roots

3 Technology Drivers: Disks Disks on track 100x in 10 years 2 TB 3.5” drive Shrink to 1” is 200GB Disk replaces tape? Disk is super computer! Kilo Mega Giga Tera Peta Exa Zetta Yotta

4 Data Gravity Processing Moves to Transducers Move Processing to data sources Move to where the power (and sheet metal) is Processor in » Modem » Display » Microphones (speech recognition) & cameras (vision) » Storage: Data storage and analysis

5 It’s Already True of Printers Peripheral = CyberBrick You buy a printer You get a » several network interfaces » A Postscript engine cpu, memory, software, a spooler (soon) » and… a print engine.

6 Tera Byte Backplane TODAY » Disk controller is 10 mips risc engine with 2MB DRAM » NIC is similar power SOON » Will become 100 mips systems with 100 MB DRAM. They are nodes in a federation (can run Oracle on NT in disk controller). Advantages » Uniform programming model » Great tools » Security » economics (CyberBricks) » Move computation to data (minimize traffic) All Device Controllers will be Cray 1’s Central Processor & Memory

7 Basic Argument for x-Disks Future disk controller is a super-computer. » 1 bips processor » 128 MB dram » 100 GB disk plus one arm Connects to SAN via high-level protocols » RPC, HTTP, DCOM, Kerberos, Directory Services,…. » Commands are RPCs » management, security,…. » Services file/web/db/… requests » Managed by general-purpose OS with good dev environment Move apps to disk to save data movement » need programming environment in controller

8 The Slippery Slope If you add function to server Then you add more function to server Function gravitates to data. Nothing = Sector Server Everything = App Server Something = Fixed App Server

9 Why Not a Sector Server? (let’s get physical!) Good idea, that’s what we have today. But » cache added for performance » Sector remap added for fault tolerance » error reporting and diagnostics added » SCSI commends (reserve,.. are growing) » Sharing problematic (space mgmt, security,…) Slipping down the slope to a 2-D block server

10 Why Not a 1-D Block Server? Put A LITTLE on the Disk Server Tried and true design » HSC - VAX cluster » EMC » IBM Sysplex (3980?) But look inside » Has a cache » Has space management » Has error reporting & management » Has RAID 0, 1, 2, 3, 4, 5, 10, 50,… » Has locking » Has remote replication » Has an OS » Security is problematic » Low-level interface moves too many bytes

11 Why Not a 2-D Block Server? Put A LITTLE on the Disk Server Tried and true design » Cedar -> NFS » file server, cache, space,.. » Open file is many fewer msgs Grows to have » Directories + Naming » Authentication + access control » RAID 0, 1, 2, 3, 4, 5, 10, 50,… » Locking » Backup/restore/admin » Cooperative caching with client File Servers are a BIG hit: NetWare™ » SNAP! is my favorite today

12 Why Not a File Server? Put a Little on the Disk Server Tried and true design » Auspex, NetApp,... » Netware Yes, but look at NetWare » File interface gives you app invocation interface » Became an app server Mail, DB, Web,…. » Netware had a primitive OS Hard to program, so optimized wrong thing

13 Why Not Everything? Allow Everything on Disk Server (thin client’s) Tried and true design » Mainframes, Minis,... » Web servers,… » Encapsulates data » Minimizes data moves » Scaleable It is where everyone ends up. All the arguments against are short-term.

14 The Slippery Slope If you add function to server Then you add more function to server Function gravitates to data. Nothing = Sector Server Everything = App Server Something = Fixed App Server

15 Disk = Node has magnetic storage (100 GB?) has processor & DRAM has SAN attachment has execution environment OS Kernel SAN driverDisk driver File SystemRPC,... ServicesDBMS Applications

16 Technology Drivers: System on a Chip Integrate Processing with memory on one chip » chip is 75% memory now » 1MB cache >> 1960 supercomputers » 256 Mb memory chip is 32 MB! » IRAM, CRAM, PIM,… projects abound Integrate Networking with processing on one chip » system bus is a kind of network » ATM, FiberChannel, Ethernet,.. Logic on chip. » Direct IO (no intermediate bus) Functionally specialized cards shrink to a chip.

17 Technology Drivers: What if Networking Was as Cheap As Disk IO? TCP/IP » Unix/NT 100% 40MBps Disk » Unix/NT 8% 40MBps Why the Difference? Host Bus Adapter does SCSI packetizing, checksum,… flow control DMA Host does TCP/IP packetizing, checksum,… flow control small buffers

18 Technology Drivers: The Promise of SAN/VIA:10x in 2 years Today: » wires are 10 MBps (100 Mbps Ethernet) » ~20 MBps tcp/ip saturates 2 cpus » round-trip latency is ~300 us In the lab » Wires are 10x faster Myrinet, Gbps Ethernet, ServerNet,… » Fast user-level communication tcp/ip ~ 100 MBps 10% of each processor round-trip latency is 15 us

19 Gbps Ethernet: 110 MBps SAN: Standard Interconnect PCI: 70 MBps UW Scsi: 40 MBps FW scsi: 20 MBps scsi: 5 MBps LAN faster than memory bus? 1 GBps links in lab. 100$ port cost soon Port is computer RIP FDDI RIP ATM RIP SCI RIP SCSI RIP FC RIP ?

20 Technology Drivers: 100 GBps Ethernet replaces SCSI Why I love SCSI » Its fast (40MBps) » The protocol uses little processor power Why I hate SCSI » Wires must be short » Cables are pricy » pins bend

21 Functionally Specialized Cards Storage Network Display M MB DRAM P mips processor ASIC Today: P=50 mips M= 2 MB In a few years P= 200 mips M= 64 MB

22 Technology Drivers Plug & Play Software RPC is standardizing: (DCOM, IIOP, HTTP) » Gives huge TOOL LEVERAGE » Solves the hard problems for you: naming, security, directory service, operations,... Commoditized programming environments » FreeBSD, Linix, Solaris,…+ tools » NetWare + tools » WinCE, WinNT,…+ tools » JavaOS + tools Apps gravitate to data. General purpose OS on dedicated ctlr can run apps.

23 Basic Argument for x-Disks Future disk controller is a super-computer. » 1 bips processor » 128 MB dram » 100 GB disk plus one arm Connects to SAN via high-level protocols » RPC, HTTP, DCOM, Kerberos, Directory Services,…. » Commands are RPCs » management, security,…. » Services file/web/db/… requests » Managed by general-purpose OS with good dev environment Move apps to disk to save data movement » need programming environment in controller