National Collaboratories Program Overview Mary Anne ScottFebruary 7, rd DOE/NSF Meeting on LHC and Global Computing “Infostructure”

Background 1993 The term “Collaboratories” was coined in an NRC report ” National Collaboratories - Applying Information Technology for Scientific Research," Committee on a National Collaboratory, National Research Council, National Academy Press, Washington, D. C., DOE activities in Collaboratories Distributed Collaborative Experiment Environment Project (testbeds and supporting technology projects ) DOE 2000 Program (pilot collaboratories and technology projects) 2000-present National Collaboratories Program Distributed Computing/Metacomputing 1999 “Grid” term popularized by publication of The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Grids are a subset of Collaboratories

Why is this important to the Office of Science? Scientific problems of strategic importance typically Involve physical scales that range over 5 to 50 orders of magnitude; Couple scientific disciplines, for example, chemistry and fluid dynamics to understand combustion; Must be addressed by teams of mathematicians, computer scientists, and application scientists; Utilize facilities that generate millions of gigabytes of data shared among scientists throughout the world; and Utilize facilities that are closely coupled to computational devices. The resources committed to solving these problems (laboratories, scientists and engineers, computational and data storage facilities, unique, one-of-a-kind facilities) are distributed throughout the enterprise (and for a given problem are seldom co-located). Requirement—coordinated, dynamic access to distributed compute and data resources, as well as collaborators

Grid Services: secure and uniform access and management for distributed resources Science Portals: collaboration and problem solving Application building services Supercomputing and Large-Scale Storage ESnet: High Speed Networking Spallation Neutron Source High Energy Physics Advanced Photon Source Macromolecular Crystallography Advanced Engine Design Computing and Storage of Scientific Groups Supernova Observatory Advanced Chemistry An End-to-End Problem for Applications “ Grids” are primarily the middleware designed to facilitate the routine interactions of resources (people, data, facilities, computers,…) in order to support widely distributed, multi-institutional science and engineering.

The National Collaboratories Program: Three Strongly Linked Efforts  Deployment and operation of a DOE Science Grid  Provides basic “infrastructure”  ESnet for directory and certificate services  Requires coordinated effort by multiple labs  Manages basic software plus imports other R&D work as available  Application partnerships linking computer scientists and application groups  Test and validate enabling technologies for discipline-specific applications  Explore how to exploit Grid infrastructure to facilitate DOE applications  Enabling R&D - middleware  Extending technology base for Grids  Extending technology base for collaboration  Packaging Grid software for deployment  Developing application toolkits  Web services for science portals

High Performance Network Environment Today Adequate bandwidth for most applications Advanced services becoming available (eg, security) Grid and collaboration technologies deployed and in use by a few applications at limited sites for limited resources In five years Ubiquitous, easy to use services and infrastructure Research in Middleware and Network technology is needed to realize this vision

Middleware Research Secure access over who does what Today: PKI/Kerberos integrated with core software Open Issues: deployable security models acceptable to site security requirements, diverse policies, etc. Information integration and access Today: mechanisms for seamless interface to multiple storage systems; mechanisms for replication management Open Issues: data provenance; metadata standards, mechanisms to resolve global names into access mechanisms etc. Middleware is the software that connects or mediates between two otherwise separate programs

Middleware Research, cont’d Coscheduling and quality of service Today: some tools available but not implemented outside closely controlled networks Open Issues: middleware to monitor bulk tranfers and adapt as needed; cross-domain policies, etc. Effective network caching and computing Today: research only, no functional tools other than web caching, distributed content delivery available Open Issues: new approaches to configuration, operation, and use needed Services to support collaborative work Today: A wide variety of “community services” such as Access Grid, etc. Open Issues: scaleable, easy-to-use primitives to support multiple modes of secure, wide-area collaboration Monitoring and problem diagnosis Today: minimal research tools such as Netlogger Open issues: no comprehensive approach exists to top-to-bottom monitoring and diagnosis for distributed science applications

Pilot Collaboratories: Early implementations of virtual laboratories NC Pilots are … Focused on a problem of national scientific or engineering significance clearly related to the mission of DOE and have high visibility Involve geographically separated groups of personnel and/or facilities that are inherently required to collaborate or be used remotely for success of the project Implementations to test and validate that scientific research programs can integrate unique and expensive DOE research facilities and resources for remote collaboration, experimentation, simulation and analysis Utilize middleware technology to enable ubiquitous access to remote resources – computation, information and expertise Demonstrate capabilities that make it easier for distributed teams to work together, over the short and long term

NC Middleware Program is focused on… Technology to enable ubiquitous access to remote resources – computation, information and expertise Capabilities that make it easier for distributed teams to work together, over the short and long term Standard services and protocols for access to networked resources, that aid software development/interoperability Middleware advances that enable scientific computing, e.g., high performance for scientific applications Goal: It’s routine; it’s easy.

SciDAC National Collaboratories National Fusion Grid General Atomics, MIT, Princeton, Univ. Utah, PPPL ANL, LBNL Contact: David P. Schissel (GA) Particle Physics Data Grid ANL, BNL, Caltech, FNAL, JLab, LBNL, SDSC, SLAC, U. Wisconsin, UCSD, USC/ISI Doug Olson (LBNL), Ruth Pordes (FNAL) The Earth System Grid ANL, LBNL, LLNL, NCAR, ORNL, USC Contact: Don E. Middleton (NCAR) DOE Science Grid LBNL, PNNL, ANL, ORNL, NERSC Contact: William E. Johnston (LBNL)

SciDAC National Collaboratories Middleware Projects A High-Performance Data Grid Toolkit ANL, USC, U. Wisconsin Contact: Bill Allcock (ANL) Middleware Technology to Support Science Portals Indiana University Contact: Dennis Gannon (IU)

Supporting Base Projects A Pilot Collaboratory and 11 Middleware Projects Collaboratory for Multi-Scale Chemical Science SNL, PNNL, ANL, LLNL, LANL, NIST, MIT, UC Berkeley Contact: Larry Rahn (SNL) Commodity Grid (CoG) Kits ANL (Gregor von Laszewski), LBNL Storage Resource Management for Data Grid Applications LBNL (Arie Shoshani), FNAL Pervasive Collaborative Computing Environment LBNL (Deb Agarwal), U. Wisconsin Scientific Annotation Middleware PNNL (Jim Myers), ORNL Middleware to Support Group to Group Collaboration ANL (Rick Stevens)

Supporting Base Projects Continued Reliable and Secure Group Communication LBNL (Deb Agarwal) Distributed Security Architectures LBNL (Mary Thompson) Grid Services ANL (Kate Keahey), LBNL (Keith Jackson) Distributed Monitoring Framework LBNL (Brian Tierney) A Scalable and Secure Peer to Peer Information Sharing Tool LBNL (Karlo Berket) Portal Web Services UT Austin (Mary Thomas), Indiana U., SDSC, GA