Scalable Sharing of Network Storage Micah Beck, Research Assoc. Professor Director, Logistical Computing & Internetworking (LoCI) Lab Computer Science Department University of Tennessee CERNMay 30, 2002

A Broad View of Networking A network is a facility for sharing a pool of resources within a community. Sharing is enabled by –Common goals –Standard protocols –Low allocation overhead –Mutual protection

IP Routing as Scalable Sharing of Bandwidth The Internet is a collection of links connected by routers Internet Protocol (IP) datagrams flow from sender to reciever along some path of links The choice of paths is made by routers according to topological considerations (tempered by policy)

Wide Area Network Services Are Like the Network Itself Intermittently inaccessible Vulnerable to partition Prone to corruption in transit Unpredictable latencies/jitter End-to-End: Never require a network service to be bigger, better or more complex than wide area access allows

Why Is It Called “End-to-End”? If the end-points requires a service that is stronger than the network can provide, they must implement it on top of the network. The fundamental tools are –Aggregation (retransmission, fragmentation) –Timers –Protocol state maintained at the end-points

How Can Storage Be Shared Scalably? Don’t require it to be –Unbounded in size –Unbounded in duration –Perfectly available –Perfectly reliable Best effort network storage Implement these properties end-to-end

Sharing Storage Enables Advanced Network Apps Caching Staging (Replication) Localization (Rendezvous) Distribution (Multicast) Spooling ( ) Temporary State Management (Checkpointing)

“Logistical” Networking Analogy to military or industrial logistics Distributing goods requires not only highways but also warehouses (depots) Data transmission is the highway Storage servers are the depots This is not Storage Networking Enabling new users and applications

Dimensions in Communication Routing (spatial)Storage (temporal) Logistical Networking

Principle Investigators –Micah Beck –James S. Plank Graduate Students –Erica Fuentes –Xiang Li Funding –DoE SciDAC –NSF Next Gen SW –NSF Internet Tech. Research Staff –Scott Atchley –Alex Bassi –Ying Ding –Hunter Hagewood –Sharmila Kancherla –Jeremy Millar –Terry Moore –Stephen Soltesz –Yong Zheng

Internet Backplane Protocol (IBP) Servers that make allocation of primitive “byte arrays” available to clients Byte arrays are not blocks (more abstract) –Network capabilities (primitive security) –Variable extents Byte arrays are not files (weaker semantics) –Size & duration are limited –“Volatile” allocations –Best effort reliability and availability –No directory structure, accounting –No caching, replication An end-to-end approach to network storage

IBP Software Structure IBP Depots (servers) are daemons that serve local storage to IBP clients. IBP clients only link to a protocol library. Clients talk to depots using TCP/IP. –Alternatives are being investigated Design is for high-performance/scalability.

Allocation Attributes Permanent vs. Time-Limited Volatile vs. Stable Read/Write semantics: –Byte Array –Pipe –Circular Queue –Completely Destructive Write

Impressions of Networking for High Energy Physics (?) Reliable throughput is the limiting factor Modes of sharing can be “modified” The use of specialized data movement middleware is acceptable The data transfer network has very few nodes and small user communities Little patience for fancy architectural ideas

IBP Enables Heterogeneity The tyranny of “TCP-friendly” IP –Datagrams and usage policy must be homogeneous end-to-end –Non-standard IP policies in the middle can leak out onto public networks Good fences make good neighbors IBP can terminate a flow and regenerate it in different network, with localized control

Example:Tsunami Uses very aggressive UDP transfer with TCP back-link for control Runs using Abilene less-than-best-effort “Scavenger Service” Tsunami must not be routed onto a link where Scavenger Service is not supported! Solution: do not route the packets, require explicit buffering and retransmission

Scheduling Control Over Sharing Without IP QoS Consider a high speed link that operates at 10Gb/s, connected to two types of links: –100Mb/s and 1Gb/s Jobs are 1TB transfers –~20 hours at 100Mb/s, ~2 hours at 1Gb/s Suspending a slow job to allow a fast job through delays it by only 10%

Scheduled Sharing of Network Links Is In Your Future! Ignoring buffer control won’t make the need for scheduled sharing go away When was more control over time & space ever a bad thing? Are we revisiting CERNET/UUCP/BITNET/DEC net? A Short History of Internet Protocols at CERN Ben Segal / CERN PDP-NS April, 1995 IBP is designed according to end-to-end principles (see SIGCOMM ’02 paper)

The Alternative to Interoperability In Buffering High performance data transfer is an isolated service, with “job scheduling” and “data staging” like at computation centers As more high performance links are built, concatenation is based on “gateways” rather than a scalable routing infrastructure Should networking solutions be built without the benefit of network architecture?

Application Areas Source routing Bandwidth adaption Reducing ( BW  delay) Reliable multicast Content Distribution Remote access to structured data Managing computation state (NetSolve caching) Temporary storage Very large data sets Collaborative computing & visualization

Building on IBP Many applications assume file semantics –Unbounded size & duration –High reliability & availability –Caching & replication In a layered architecture, these are implemented through aggregation and additional intelligence at the next level

The Network Storage Stack Physical Access (OS) IBP (Internet Backplane Protocol) Logistical Runtime System Logistical Tools Applications L-BoneexNode

ExNode vs inode exNode inode IBP Allocations the network local system disk blocks kernel capabilities block addresses user

ExNode Mobility XML Serialization The exNode serialization is a portable soft link

Sameple exNodes Services: Fragmentation, Replication 123

Building a Logistical Operating Environment Resource Discovery –Logistical Backbone registry (LBone: Plank) –Can also use DNS, could use MDS Programming Tools –exNode library, primitive commands –policy modules (replication, fragmentation) –cool applications (SC demo: IBPster MP3)

Related Work Peer-to-Peer Storage –Napster, Gnutella, etc –OceanStore (UC Berkeley) Storage Area Networking –Fiber Channel, Ultraband, iSCSI –Network Attached Storage (CMU) Overlay Networking –Caching –Multicast –Data Grids Multimedia Protocols –Content Delivery –Streaming –Video on Demand

Logistical Networking: E2E Storage and Bandwidth Complex paths and buffers are exposed Assumptions are appropriately weak Resource management is passive Spatial and temporal control Applic. autonomy, network transparency Resources in the hands of end users Futures: more control, greater scope

Logistical Computing and Internetworking Laboratory Documents and software downloads