Slide 1 ISTORE: An Introspective Storage Architecture for Network Service Applications Aaron Brown, David Oppenheimer, Kimberly Keeton, Randi Thomas, Jim Beck, John Kubiatowicz, and David Patterson Winter IRAM Retreat

Slide 2 Agenda Overview of ISTORE: Motivation and Architecture Hardware Details and Prototype Plans Software Architecture Discussion and Feedback

Slide 3 Motivation Emerging paradigm of network-based services –today: e-commerce, online database services, search engines, and web servers –tomorrow: above, plus thin-client/PDA infrastructure support These services have unique demands –vast storage needs (large operational and historical DSS databases) –high access rates –mix of transactional (ACID) and non-transactional accesses –easy incremental scaling of storage and processing *minimal administration burden »easy setup, scaling, maintenance, repair, tuning,... »automatic dynamic adaptation to failures, workload changes,...

Slide 4 Scaleable Storage Appliances Best architecture to meet these demands is a scaleable self-maintaining storage appliance –appliance: a single-function device optimized to provide one application or service »consists of software and hardware constructed with that service in mind »interface to WAN/LAN via standard high-level protocols »internal implementation hidden –storage appliance: integrates an application with its data storage in one unit –scaleable appliance: incrementally scaleable with automatic integration of new resources (e.g., disks) –self-maintaining appliance: runs autonomously, without human intervention, once initially configured Examples: e-commerce, decision-support DB, PDA proxy, video server, or online voting appliances

Slide 5 Today’s Appliances Some storage appliances exist today: –Network Appliance NFS and proxy servers –SNAP! Server file server –Qube web server –SonicWall firewall –... But none are truly self-maintaining, and most are not scaleable

Slide 6 Why Aren’t There Appliances That Match Our Criteria? 1) Self-maintenance is tricky and application-dependent *can’t rely on human administrators to configure, monitor, and tune system for its application –failure management »devices must fail fast without interrupting service »failures should not require immediate human intervention –performance management »system must adapt to changes in workload or access patterns –system upgrades and scaling »new hardware resources must be automatically incorporated without interruption of service »new devices should be used immediately to improve performance or repair prior failures

Slide 7 More Appliance Challenges 2) Appliance designs are inextricably tied to single applications –each service or application requires a custom-designed appliance »new systems software must be written, tested, debugged »often can share hardware, but entire system must be retested »especially difficult to design new self-maintaining appliances –performance and manageability suffer when an appliance is used for other than its intended purpose »e.g., if workload doesn't quite match appliance design goals »example: using a file-server appliance as the storage subsystem for a database application

Slide 8 The ISTORE Solution: An Introspective Meta-Appliance A meta-appliance is a generic hardware and software platform that can be specialized into a single- application, focused appliance An introspective system uses hardware and software monitoring to tune itself and to achieve self- maintenance The ISTORE meta-appliance is a hardware/software architecture for building scaleable self-maintaining storage appliances

Slide 9 ISTORE Hardware Innovations 1) ISTORE uses intelligent hardware –ISTORE is a shared-nothing distributed system of intelligent device bricks interconnected by an intelligent chassis –intelligent device brick: a device (e.g., a disk) plus a fast embedded CPU, memory, and NI (e.g., an IRAM) –intelligent chassis: scaleable, redundant, fast network + UPS Intelligent Chassis Intelligent Device Brick Device CPU, memory, NI

Slide 10 Benefits of ISTORE’s Hardware Architecture Decentralized processing (shared-nothing) –system can withstand partial failure Intelligent devices monitor their own “health,” test themselves, manage failures, collect application- specified performance data, and execute application worker code –provides the foundation for self-maintenance and self-tuning Plug & play, hot-swappable bricks ease configuration, scaling –hardware is specialized/scaled by selecting an appropriate collection of devices »disks, DRAMs, WAN/LAN interfaces,...

Slide 11 ISTORE Software Innovations 2) ISTORE’s system software is designed to deliver self-maintenance and allow for easy specialization –layered mechanism libraries facilitate introspection, self- maintenance –extensibility via domain-specific languages (DSLs) allows customization of the runtime system with application-specific policies Benefits of ISTORE’s software architecture –appliance designer inherits functionality of supplied base mechanisms for monitoring and adaptation –easy for designer to customize mechanisms and define adaptation policies by writing domain-specific code

Slide 12 ISTORE and IRAM ISTORE relies on intelligent devices IRAM is an easy way to add intelligence to a device –embedded, low-power CPU meets size and power constraints –integrated DRAM reduces chip count –fast network interface (serial lines) meets connectivity needs Initial ISTORE prototype won’t use IRAM –will use collection of commodity components that approximate IRAM functionality, not size/power –but results will be applicable to IRAM-based systems »success with ISTORE will encourage integration of IRAM-like processors into standard devices

Slide 13 Agenda Overview of ISTORE: Motivation and Architecture Hardware Details and Prototype Plans Software Runtime Architecture Discussion and Feedback

Slide 14 Backup Slides

Slide 15 Related Work ISTORE extends several recent research efforts –Intelligent/Active Disks, NASD (UCB, UCSB, CMU) »used as a component in ISTORE; ISTORE adds a global runtime system, introspection, and self-maintenance –network service appliances (NetApp, Snap!, Qube, Swarm,...) »ISTORE adds the idea of a meta-appliance & self-maintenance –self-maintaining systems (Sun Project StoreX, Tandem,...) –extensible operating systems (SPIN, Exokernel, VINO, Scout,...) –domain-specific languages (OGI Microlanguages, Prolac, VHDL,...) –adaptive systems (HP AutoRAID, MS AutoAdmin, MS Millennium) –plug-and-play system construction (Sun Jini, PC Plug&Play,...) ISTORE leverages Berkeley’s experience with reliable, large, distributed storage systems and applications –RAID, Tertiary Disk, XFS, Sprite, NOW, PostGres/GiST,...