Internet2 Distributed Storage Infrastructure Update Micah Beck Univ. of Tennessee, Knoxville Bert Dempsey Univ. of North Carolina, Chapel Hill Web Caching Workshop BOF 31 March 1999, San Diego

I2-DSI Participants zUT Knoxville / ICL yMicah Beck yTerry Moore yMartin Swany yJudi Talley zUNC Chapel Hill /SILS yBert Dempsey yPaul Jones (MetaLab) yDebra Weiss yZhiwei Xiao z GigaPOP and Campus Site Managers z UCAID/Internet2 yNetwork Storage Working Group yTed Hanss Applications Director z NC Networking Initiative z Digital Library Federation

A Word From Our Sponsors zCiscoDNS redirection zEllemtel engineering effort zIBM large storage & DCE servers zNovellstorage & directory servers zStarburst reliable multicast software zStorageTeklarge storage servers zSun design collaboration

Single Server Model zHigh performance locally zUnacceptable performance across commodity backbone

Relying on Wide Area QoS zHigh performance access with reserved bandwidth zEssential for real-time communication zTechnically difficult, expensive, not generally available

I2-DSI Model: Replicated Services zClients access nearby server zEveryone gets performance zLocal resources implement a global service

I2-DSI Service Architecture zReplication yRsynch+, Omnicast, AFS/DFS Novell Replication zResolution ySonar DNS, Distributed Director zDelegation yCache prefetch general users replicated core delegated server local users

Internet Content Channels zA channel is a collection of content which can be transparently delivered to end user communities at a chosen (price,performance) point through a flexible, policy-based application of resources

Server Channel Examples zReplicated Web Servers yAPIs: Standard HTML, Active Server Pages yChannels: Web sites zStreaming Media yAPIs: MPEG-2, proprietary file formats yChannels: collections of multimedia presentations zExecutable content yAPIs: Java byte code, Tcl, Perl yChannels: CGI programs

Current Server Deployment

IBM Web Cache Manager RS/6000 AIX Server 1 GB RAM 72 GB Disk / 900 GB Tape ADSM Heirarchical Storage Mgt.

I2-DSI Server Operations zProject Operations Coordinator yJudi Talley, University of Tennessee at Knoxville zSite Managers yDave Vernon, Indiana University yDavid Lassner, University of Hawaii at Manoa yMark Johnson, NC Networking Initiative yMichael Rechtenbaugh, EROS Data Center

Infrastructure Expansion zStorageTek y2 PC/Linux Servers y700GB disk, tape backup (hot!) zNovell y6 PC/NetWare Servers y100GB disk ySmaller institutions or departments

Infrastructure Development Efforts zProximity Resolution xMartin Swany: SonarDNS xGeoff Carpenter, German Goldszmidt: Narwhal (IBM) zReplication Mechanisms and Modeling yBert Dempsey & students: xDebra Weiss: Batch rsync + multicast xZhiwei Xiao: Network metrics and modeling zChannel Representation and Server xLeif Abrahamsson, Christophe Achouiantz, Patrik Johansson (Ellemtel)

I2-DSI Applications Workshop Chapel Hill, NC March 4 &5, 1999 z10 applications yIndiana: Digital music and media library yUNC-CH: Instructional Management System ySan Jose State: Art history images yVanderbilt: zoomable medical images yViagenie: Network docs database yColumbia: Earth sciences environment yUNC-CH: Virtual Laboratories yOhio Supercomputer Center: High Volume Datasets yCalTech: Globally Interconnected Databases yUniv. of Kent: National Software Archive yRed Hat: pan-Linux source distribution

I2-DSI Applications Workshop Chapel Hill, NC March 4 &5, 1999 z4 technologies xMinnesota: Scalable Video xIBM Research: Multicast, Filter and Store xMoscow Ctr. for New Info. Tech. in Med. Ed.: Semantic Text Analysis xIBM Research: Narwhal Resolution Proxy  zSpecial issue of the Journal of Network and Computer Applications (Academic Press)

Application Management Partner MetaLab.unc.edu zThe site formerly known as SunSITE.unc.edu zFearless Leader Paul Jones z“A cool, tall glass of sweet tea on a hot day.” z 2 M HTTP & 1/3 M FTP file transfers daily zCollections policy yteaching, research, or public service yuse technology in innovative and unique ways ynon-commercial or not-for-profit

Application Strategy zChose initial applications yAvailable or easily ported services yLow update demands zPort to an I2-DSI server yOur development effort is limited yApp developers can have access to the servers zDistribute to homogeneous core zDerive service abstractions

The Need for Channel Representation Standards Origin Server proxy Replicated Server Standard-based Web traffic Replication of source files Web clients locally interpreted files Replicated Server replicated files

Replication Performance and Scalability Issues zServer placement zServer resources zServer description (metadata) zServer Channel description (metadata) zObject representation zCharacterization of replication mechanisms zChannel-to-server mapping (subscription)

NetStore ‘99 Workshop zNetwork Storage Technical Workshop yKnoxville, TN, October 1999 yhttp://dsi.internet2.edu/netstore99 zScope yI2-DSI implementation yI2-DSI applications yRelated networking projects yStorage technology

Conclusions zA server platform is in place zInfrastructure development yService abstractions (search, computation) yPublication and replication protocols yPortable representation and API yHeterogeneous servers zSix months to show results from initial application development efforts