Perspectives on Grid Technology Ian Foster Argonne National Laboratory The University of Chicago

ARGONNE  CHICAGO Ian Foster Background: Why “Grids”? l Because the resources needed to solve complex problems are rarely colocated u Advanced scientific instruments u Large amounts of storage u Large amounts of computing u Groups of smart people l For a variety of reasons u Resource allocations not optimized for 1 appln u Required resource configurations change u Different views of priorities and truth

ARGONNE  CHICAGO Ian Foster Grid Application Examples Teleimmersion/distance collaboration Record-setting distributed supercomputing TransAtlantic remote visualization/steering Parameter studies with deadline scheduling Online analysis of instrument data

ARGONNE  CHICAGO Ian Foster Hence, Grids l Next-gen infrastructure u Advanced network services u Computers, storage, etc. u New tools, methodologies l Foundation for advanced network applications, e.g. u Data-intensive (Data Grid) u Collaborative (Access Grid) u Compute-intensive u Online instrumentation Akamai’s server network (Jan 2000)

ARGONNE  CHICAGO Ian Foster Grid R&D: A Brief History l Late 80s/early 90s: Isolated experiments u Gigabit testbeds, metacomputing expts l Mid 90s: first attempts at integration u E.g., 1995 I-WAY and I-Soft software l Late 90s: emergence of infrastructure and identification of new usage modalities u Globus toolkit, Access Grid, Data Grid u Grid Forum, production Grid infrastructures l 2000+: Grids go mainstream u “Grid services” integrated into network

ARGONNE  CHICAGO Ian Foster Creating a Usable Grid : Grid Services (“Middleware”) l Standard grid services that u Provide uniform, high-level access to a wide range of resources (including networks) u Address interdomain issues of security, policy, etc. u Permit application-level management and monitoring of end-to-end performance l Middleware-level and higher-level APIs and tools targeted at application programmers u Map between application and Grid

ARGONNE  CHICAGO Ian Foster Grid Services Architecture: An Emerging Grid Computing Framework Archives, networks, computers, display devices, etc.; associated local services Protocols, authentication, policy, resource management, instrumentation, discovery, etc., etc. Grid Fabric Grid Services Appln Toolkits Applns... … a rich variety of applications... Remote viz toolkit Remote comp. toolkit Remote data toolkit Remote sensors toolkit Async. collab. toolkit

ARGONNE  CHICAGO Ian Foster Remote comp. toolkit Grid Fabric Remote data toolkit Archives, networks, computers, display devices, etc.; associated local services Remote data for climate Data Grid Toolkit Protocols, authentication, policy, resource management, instrumentation, discovery, etc., etc. Grid Services Appln Toolkits Applns... Remote sensors toolkit Async. collab. toolkit AuthenticationReservationInformationFault detection Res. mgmtAccountingInstrumentation... Remote viz toolkit Remote viz for CFD Grid MPI Co-allocator Advantages of GSA u Single infrastructure u Avoid redundant development u Encourage code sharing

ARGONNE  CHICAGO Ian Foster National and International Grid Testbeds I-WAY NASA’s Information Power Grid NSF PACI’s National Technology Grid

ARGONNE  CHICAGO Ian Foster European Grid Testbeds

ARGONNE  CHICAGO Ian Foster

ARGONNE  CHICAGO Ian Foster

ARGONNE  CHICAGO Ian Foster A Current Focus: Data Grids l Integrate data archives & computers into a distributed data management & analysis “Grid” l More than storage, computing, network: also u Caching and mirroring to exploit locality u Intelligent scheduling to determine appropriate replica, site for (re)computation, etc. u Coordinated policy-driven resource management for performance guarantees u Embedded security, policy, agent technologies for effective distributed analysis

ARGONNE  CHICAGO Ian Foster Grid Services and the Data Grid: Resource Management Architecture “10 GFlops, EOS data, 20 Mb/sec -- for 20 mins” Grid Information Service GRAM Resource Broker Info service: location + selection Resource Managers GRAM Fork LSF EASYLL Condor etc. “What computers?” “What speed?” “When available?” “50 processors + storage from 10:20 to 10:40 pm” “20 Mb/sec”

ARGONNE  CHICAGO Ian Foster Scheduling Bulk Transfer and High-Priority Transfers

ARGONNE  CHICAGO Ian Foster

ARGONNE  CHICAGO Ian Foster Grids and HEP Computing l HEP computing: a Grid project par excellence u Tight integration of computing, storage, networking; demanding requirements u Focus on services for a large community l HEP computing brings new problems, e.g. u Big increase in scale u Object database technology u Complex policy issues l HEP also contributes interesting technologies u E.g., MONARC simulator, NILE, GIOD

ARGONNE  CHICAGO Ian Foster Summary l Grids promise to enable qualitatively new approaches to science and engineering l Key enabler is the integration of advanced resources with new Grid services and tools l To date, wonderful application demonstrations, considerable progress in key technologies, some early attempts at real deployment l Timely for scientific communities such as HEP to investigate and apply for real

ARGONNE  CHICAGO Ian Foster For More Information... l Globus: l (European) Grid Forum: l Grid book “The Grid: Blueprint for a Future Computing Infrastructure,” I. Foster & C. Kesselman (Eds), Morgan-Kaufmann,