The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago

ARGONNE CHICAGO On-demand creation of powerful virtual computing systems The Grid: The Web on Steroids Web: Uniform access to HTML documents Grid: Flexible, high-perf access to all significant resources Sensor nets Data archives Computers Software catalogs Colleagues

ARGONNE CHICAGO Why Now? l The Internet as infrastructure –Increasing bandwidth, advanced services l Advances in storage capacity –You can buy a Terabyte for < $15,000 l Increased availability of compute resources –Clusters, supercomputers, etc. l Advances in application concepts –Simulation-based design, advanced scientific instruments, collaborative engineering,...

ARGONNE CHICAGO Talk in a Nutshell … l Where this all started –Special purpose networks, stunts l Where we are –Grid services as unifying concept –Limited but growing application set –Persistent testbeds l Where were going –Production grids –New user communities and applications –Convergence of Grid and Info Infrastructure –High-level services and development tools

ARGONNE CHICAGO Disclaimers l Describing work of many people –ANL, ISI, NCSA, CIT, SDSC, NASA, EVL, etc. –In particular, Carl Kesselman, Globus Co-PI –In U.S., Europe, Asia, Australia l Much important work by others is left out –I have picked just a representative sample

Where This All Started

ARGONNE CHICAGO High-Performance Networks: E.g., CASA Gigabit Testbed l One of four U.S. testbeds in early 90s l Dedicated network –HIPPI over SONET, custom hardware –Caltech, SDSC, LANL l Focus on distributed supercomputing applications –Large distributed memory & vector supercomputers CASA: Paul Messina et al.

ARGONNE CHICAGO The SC95 Conference l Focus on application demonstrations –60+ groups l OC-3 backbone l Large-scale use of immersive displays –CAVE and I-Desk l I-Soft programming environment –Pioneered security, scheduling ideas I-WAY leaders: Tom DeFanti, Tim Kuhfuss, Rick Stevens, Maxine Brown

ARGONNE CHICAGO The I-Soft Software Environment l Kerberos authentication –I-POP initiated rsh to local resources l Andrew File System to distribute software & state l Central scheduler –Dedicated I-WAY nodes on resource –Interface to local scheduler l Nexus-based comm libraries –MPI, CAVEcomm, CC++ I-Soft development: Warren Smith, Jonathon Geisler, Steve Tuecke I-POP AFSKerberosScheduler ATM Switch Resource 1 Resource N Possible Firewall ATM Internet

ARGONNE CHICAGO Combustion System Modeling l A shared collaborative space –Link people at multiple locations –Share and steer scientific simulations on supercomputer l Prototypical of new applns –Traditional HPC an element of a more complex whole l Substantial development effort –Yet insecure, no QoS Chicago San Diego Lori Freitag et al., Argonne National Laboratory and NALCO DC

Where We Are Now

ARGONNE CHICAGO The Need for Grid Services Remote access Remote monitor Information services Fault detection... Resource mgmt Collaboration Tools Data Mgmt Tools Distributed simulation... net

ARGONNE CHICAGO A Defining Framework: Grid Services Architecture Grid-enabled archives, networks, computers, display devices, etc.; associated local services Protocols, authentication, policy, resource management, instrumentation, discovery, etc., etc. Grid Fabric Grid Services Appln Toolkits Applns... … a rich variety of applications... Remote viz toolkit Remote comp. toolkit Remote data toolkit Remote sensors toolkit Async. collab. toolkit

ARGONNE CHICAGO The Grid Services Concept l Standard services that –Provide uniform, high-level access to a wide range of resources (including networks) –Address interdomain issues: security, policy –Permit application-level management and monitoring of end-to-end performance l Broadly deployed, like Internet Protocols l Enabler of application-specific tools as well as applications themselves l Integrate across domains to support the creation of scalable virtual organizations

ARGONNE CHICAGO The Globus Project: Developing a Grid Services Architecture l Developed as a key enabling mechanism the Grid Security Infrastructure –Uniform authentication & authorization mechanisms in multi-institutional setting –Single sign-on, delegation, identity mapping –Public key technology, SSL, X.509, GSS-API l Used to construct Grid resource managers that provide secure remote access to –Computers: GRAM server (HTTP), secure shell –Storage: storage resource managers, GSIFTP –Networks: differentiated service mechanisms Globus project Co-PI: Carl Kesselman

ARGONNE CHICAGO Developing a Grid Services Architecture l Developed as a 2nd key enabling mechanism the Grid Information Service (aka MDS) –Lightweight Directory Access Protocol (LDAP) used to access info about specific resources –Index servers provide for flexible indexing –Uniform resource discovery & characterization mechanisms in highly distributed systems –A basis for resource brokering, adaptation, autoconfiguration l Other services include fault detection and communication GIS: Steve Tuecke, Steve Fitzgerald, Gregor von Laszewski

ARGONNE CHICAGO Status of Grid Services l Core Grid services & others (e.g., Network Weather Service) have been deployed in large-scale testbeds l Availability of these services is enabling tool & application development projects l Major challenges certainly remain: e.g. –Advance reservation, policy, accounting –End-to-end application adaptation (events?) –Integration with commodity technology l Grid Forum:

ARGONNE CHICAGO Examples of Grid-Enabled Tools l Message Passing –MPICH-G2 l Distributed collaboration –CAVERNsoft, Access Grid l High-throughput computing –Condor-G, Nimrod-G l Distributed data management & analysis –Data Grid toolkits l Desktop access to Grid resources –Commodity Grid Toolkits (CoG Kits)

ARGONNE CHICAGO tomographic reconstruction real-time collection wide-area dissemination desktop & VR clients with shared controls Advanced Photon Source Online Instrumentation archival storage DOE X-ray grand challenge: ANL, USC/ISI, NIST, U.Chicago

ARGONNE CHICAGO Distributed Supercomputing SF-Express Distributed Interactive Simulation: Caltech, USC/ISI l Starting point: SF-Express parallel simulation code l Globus mechanisms for –Resource allocation –Distributed startup –I/O and configuration –Fault detection l 100K vehicles using 13 computers, 1386 nodes, 9 sites (5 years early!) NCSA Origin Caltech Exemplar CEWES SP Maui SP

ARGONNE CHICAGO l Problem solving environment for comp. chemistry l Globus services used for authentication, remote job submission, monitoring, and control l Future: distributed data archive, resource discovery, charging Problem Solving Environments ECCE: Pacific Northwest National Laboratory

Where We Are Going

ARGONNE CHICAGO Emerging Production Grids NASA Information Power Grid NSF National Technology Grid

ARGONNE CHICAGO Emerging User Communities l NSF Network for Earthquake Engineering Simulation (NEES) –Integrated instrumentation, collaboration, simulation l Grid Physics Network (GriPhyN) –ATLAS, CMS, LIGO, SDSS –World-wide distributed analysis of Petascale data l Access Grid: supporting group based collaboration

ARGONNE CHICAGO Todays Information Infrastructure l Network-centric: simple, fixed end systems; few embedded capabilities; few services; no user-level quality of service O(10 6 ) nodes

ARGONNE CHICAGO Tomorrows Information Infrastructure: Not Just Faster and More Reliable l Application-centric: heterogeneous, mobile end-systems; many embedded capabilities; rich services; user-level quality of service QoS Resource Discovery Caching O(10 9 ) nodes

ARGONNE CHICAGO Optical MREN (OM) ANL 3 pr 1 pr (2) OC-3 3 pr OC-3 2 pr Bell Nexxia to CA*net4 I-WIRE OC-3 ? Optical STAR TAP (AUP free) AADS switch 6 pr Franklin CO Central Office UIUC UIC UCIIT MREN NU Chicago BN NU Evanston GigE IU Bloom- ington UIPUI Purdue STAR LIGHT

ARGONNE CHICAGO I-WIRE: Computing Urbana ArgonneChicago Topology Research Areas Latency-Tolerant Algorithms Interaction of SAN/LAN/WAN technologies Cluster Architectures

ARGONNE CHICAGO Scalable Global Services: The InterGrid l Do for the Grid what routers do for the network l Wide variety of services –Resource trading –Policy enforcement –Agent based scheduling –Agent based monitoring –Virtual data l Exploit emerging network services, e.g. QoS Data Caching Schedulin g Fault Recover y Routing

ARGONNE CHICAGO l Megacomputing (Larry Smarr) –Very large numbers of computers –Opportunistic, dynamic behaviors –Bandwidths variable, ultimately to Gb/s l distributed.net, entropia.com

ARGONNE CHICAGO The Emergence of the Application Service Provider l ASP model: rent access to applications that execute on remote computers l Decouples interface, application, computing, hardware –Each can be optimized & sold separately l Highly disruptive impact on all aspects of the computer industry l May also be key to a dramatic increase in the penetration of HPC –E.g., see NetSolve (U.Tenn), PUNCH (Purdue)

ARGONNE CHICAGO Summary l Grids will change the way we do science and engineering l Key services and concepts have been identified, although major challenges remain l Transition of services and applications to production use is starting to occur l Future will see increased sophistication and scope of services, tools, and applications l Potentially disruptive trends: ASPs, megacomputing

ARGONNE CHICAGO For More Information l Book published by Morgan Kaufman – l Globus – l Grid Forum –

ARGONNE CHICAGO We are Hungry for the Grid