Fox 2 AISRP April 4-6, 2005  Earth System Grid  Grid-enabled OPeNDAP  Architecture - Server and Application access  Framework experience  Summary  Plans for the coming year Overview

Fox 3 AISRP April 4-6, 2005  The goal of ESG is to make climate data – particularly climate model data – an easily accessible community resource. The project is funded by the SciDAC program: Scientific Discovery through Advanced Computing.  Enabling researchers to understand and make effective use of very large, distributed climate datasets is critical. The broad strategy is to develop a collection of server-side capabilities – minimize the amount of data movement.  Multiple interfaces to ESG will allow researchers to focus on science rather than issues of data transfer, format, and data set manipulation.  Foundation is Globus Grid technology Earth System Grid Overview

Fox 4 AISRP April 4-6, 2005 Earth System Grid Portal

Fox 5 AISRP April 4-6, 2005 ESG: U.S. Collaborations & Development ORNL: Climate storage & computational resources ORNL: Climate storage & computational resources LANL: Next generation coupled models & computing LANL: Next generation coupled models & computing ANL: Computational grids, & grid-based applications ANL: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications NCAR: Climate change predication and scenarios NCAR: Climate change predication and scenarios LBNL: Climate storage facility LBNL: Climate storage facility LLNL: Model diagnostics & inter-comparison LLNL: Model diagnostics & inter-comparison

Fox 6 AISRP April 4-6, 2005 ESG areas of development âAuthentication and Authorization services : application of Globus technologies for secure data management and access (PKI certificates, proxy delegation, Community Authentication Services, web interfaces) âData Transport Services: based on gridFTP protocol and implementation (high speed, tunable, multi-stream, reliable), extensions for multi-file management and connection to offline storage systems (Hierarchical Storage Management), and for transparent data access and operations (grid-enabled OPeNDAP) âMetadata services (for data management, access, search & discovery, annotation, analysis, etc.) âOther services: Data Analysis and Visualization, Task Management, Monitoring and Control, Replication of data, etc.

Fox 7 AISRP April 4-6, 2005 ESG: ESG-II Architecture

Fox 8 AISRP April 4-6, 2005 TOMCAT Servlet engine TOMCAT Servlet engine MCS Metadata Cataloguing Services MCS Metadata Cataloguing Services RLS Replica Location Services RLS Replica Location Services SOAP RMI MyProxy server MyProxy server MCS client RLS client MyProxy client GRAM gatekeeper GRAM gatekeeper CAS Community Authorization Services CAS Community Authorization Services CAS client disk MSS Mass Storage System HPSS High Performance Storage System disk HPSS High Performance Storage System disk SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server LBNL LLNL ISI NCAR ORNL ANL Striped gridFTP client Striped gridFTP client gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP LAS Live Access Server LAS Live Access Server

Fox 9 AISRP April 4-6, 2005 NCAR LBNL LLNL ISI ANL ORNL GSI CAS server CAS client MyProxy clientMyProxy server TOMCAT SECURITY services GRAM METADATA services FRAMEWORK services Auth metadata RLS MySQL RLS MySQL RLS MySQL RLS MySQL NERSC HPSS NCAR MSS DISK ORNL HPSS DATA storage The Earth System Grid THREDDS catalogs Xindice MySQL OGSA-DAISMCS TRANSPORT services gridFTP server/client HRM openDAPg server ANALYSIS & VIZ services NCL openDAPg clientLAS server CDAT openDAPg client MONITORING services SLAMON daemon TOMCAT AXIS

Fox 10 AISRP April 4-6, 2005 Metadata-centric view of ESG services METADATA SERVICES METADATA SERVICES USER AUTHENTICATION AND AUTHORIZATION USER AUTHENTICATION AND AUTHORIZATION ACCESS AND AUTHORIZATION METADATA DATA TRANSPORT LOCATION METADATA SYSTEM MONITORING AND CONTROL SYSTEM MONITORING AND CONTROL LOGGING METADATA DATA SEARCH & DISCOVERY CONTENT METADATA ANNOTATION & HISTORY METADATA DATA ANALYSIS & VISUALIZATION DATA ANALYSIS & VISUALIZATION AGGREGATION METADATA DATA BROWSING CATALOGUING METADATA

Fox 11 AISRP April 4-6, 2005 OPeNDAP and Grid systems âDODS since ~ 1995 was based on http and cgi-style architecture âTwo concerns âApplication support and performance of HTTP âHousekeeping abilities of cgi architecture âSolution: evolve OPeNDAP, the discipline neutral aspect of DODS

Fox 12 AISRP April 4-6, 2005 OPeNDAP ctd. âData transport protocol and access protocol separated âRevised server architecture âAddress Grid-style authentication âMemory management âException handling âAll these changes and retain interoperation with HTTP and cgi âAdvanced requirements: URL should support more than one dataset, or object, i.e. aggregation

Fox 13 AISRP April 4-6, 2005 OPeNDAP 3.x vs OPeNDAP-g Architecture Simple and easy to install One CGI process per URL request Limited memory management – external Limited scalability Limited status reporting to web server Returns data stream from one format Standalone server or httpd module Can manage multiple daemon processes Strong memory management – internal Reuse processes, scales Coupled to OPeNDAP server for status Returns multiple formats in a single stream, multiple protocols

Fox 14 AISRP April 4-6, 2005

Fox 15 AISRP April 4-6, 2005 Application development

Fox 16 AISRP April 4-6, 2005 Status âOperational/production release of standalone OPeNDAP server (no dependence on web server) for ESG âRun OPeNDAP server as a client to GridFTP or HTTP server âMulti-protocol support: file, http, GridFTP, ftp, etc. âFile format support: netCDF, CDF, FITS, CEDAR, … âRe-architected for aggregation support and performance âPortal application client in production, netCDF client operational âAuthentication is handled outside OPeNDAP server framework âURL syntax is more complex but more expressive âWill become part of community OPeNDAP release very soon

Fox 17 AISRP April 4-6, 2005 ESG: Framework experience  ESG is a highly collaborative effort allowing users to quickly access data (petabytes of raw or processed data in an application independent manner).  Payoffs of this distributed collaborative infrastructure have included: Distributed data-sharing, RLS works! SRM/HRM work! OPeNDAP-g works! Simplified data discovery of climate data, the work on metadata paid off! Scalability? Large-scale climate data processing and analysis via highly integrated portal Increased collaboration among climate research scientists, people use it! Aid in climate assessments and estimates of future climate variability and trends, IPCC!

Fox 18 AISRP April 4-6, 2005 ESG: Framework experience âTransport - GridFTP versus HTTP 3Server to server 3Very good performance 7Depends on a very specific version of GRIDftp server (stripped) 7Clients are not as capable due to ‘weight’ of globus, revert to HTTP âScalability and response times (data AND metadata) 3Framework architecture supports re-layered for tuning âService monitoring 3to support the distributed collaborative infrastructure 7need lots or all services to really make a production environment work  Try out ESG by visiting the website at:

Fox 19 AISRP April 4-6, 2005 Success?  Users are generally happy, developers are very happy  Exploited new technology components  Integration - when and how does it work and scale? 3XML -> SQL 3DODS -> OPeNDAP and OPeNDAP-g  Globus provides a suite of framework components, some are easier to integrate than others, some just don’t fit our use-cases and architecture  Data framework - e.g. OPeNDAP has been extremely successful  Carrying this to space science (solar-terrestrial)

Fox 20 AISRP April 4-6, 2005 Vision for building science cyberinfrastructure  Use-case, then requirements  Then derive architecture and choose technology components  Build a working system for users from the start  Get your funding source and community to commit to an evolving architecture  If you choose a major framework technology, e.g. Globus, OPeNDAP, THREDDS, partner with them

Fox 21 AISRP April 4-6, 2005 One paradigm Goal - find the right balance of data/model holdings, portals and client software that a researchers can use without effort or interference as if all the materials were available on his/her local computer. E.g. The Virtual Solar-Terrestrial Observatory (VSTO) is proposed to be: a distributed, scalable education and research environment for searching, integrating, and analyzing observational, experimental and model databases in the fields of solar, solar-terrestrial and space physics Comprises: a system-like framework which provides virtual access to specific data, model, tool and material archives containing items from a variety of space- and ground-based instruments and experiments, as well as individual and community modeling and software efforts bridging research and educational use

Fox 22 AISRP April 4-6, 2005 Virtual Observatory? Need better glue Basic problem: schema are categorized rather than developed from an object model/class hierarchy -> significantly limits non-human use. However, they all form the basis to organize catalog interfaces for all types of data, images, etc. This limits data systems utilizing frameworks and prevents frameworks from truly interoperating (SOAP, WSDL only a start) Directories, e.g. NASA GCMD, CEDAR catalog, FITS (flat) keyword/ value pairs, are being turned into ontologies (SWEET, VSTO) Markup languages, e.g. ESML, SPDML, ESG/ncML are excellent bases Evolve, recast, merge (where appropriate) using formal processes, tools with intended use in mind - for interface specifications, reasoning, validation, etc. beyond the usual search and access

Fox 23 AISRP April 4-6, 2005 Summary  Basic success in both data systems and data frameworks  Satisfying user and sponsor needs (from ‘just’ to ‘outstanding’)  Experience with Globus ranges from very good, to not ready for our need  Experience with OPeNDAP is very good, esp. with core services  Scalability and performance require an adaptable architecture which is something system-level interfaces can still hide from the user  Challenge - to bring these attributes to a framework, i.e. in which the user is more exposed

Fox 24 AISRP April 4-6, 2005 Plans  IDL application level access to new OPeNDAP server framework  Outreach to NASA communities/data centers to install and test new capabilities (server and client)  Joint development of accompanying semantic catalogs for Sun-Earth Connection datasets within the OPeNDAP framework  SPDML-enabled OPeNDAP server