The Earth System Grid (ESG) Computer Science and Technologies DOE SciDAC ESG Project Review Argonne National Laboratory, Illinois May 8-9, 2003

May 8, 2003Earth System Grid2 Computer Science Perspective: Why is ESG Important? Application needs help formulate new frameworks and information technologies –Scientific apps good indicator of future trends –Climate community leading IT consumer Experimentation key to (computer) science –Needs robust instantiation of new technology –Needs an engaged community of consumers Multi disciplinary (intra-CS and CS-apps) teams are key to IT advances

May 8, 2003Earth System Grid3 We’re Particularly Interested in the Following Aspects of ESG “Enable [a community of] researchers to understand and make effective use of large, distributed climate datasets” –Dataset federation—physical and semantic –Security: who can do how much of what –Efficient analysis: distribution and placement of computation and data Within the context of real data centers, real data, real analyses, and real users

May 8, 2003Earth System Grid4 The Computer Science Team ESG engages CS people at every institution Four groups act as Grid technology providers –Argonne National Laboratory (Globus Toolkit, etc.) –Lawrence Berkeley National Laboratory (SRM) –USC Information Sciences Institute (Globus Toolkit, etc.) –Oak Ridge National Laboratory (monitoring) Two groups act as climate data analysis tech providers –NCAR (data delivery and analysis software) –PCMDI (data delivery and analysis software) Integration, application, experimentation are highly collaborative activities

May 8, 2003Earth System Grid5 Our Hammers … Storage Resource Managers, Multiple File Transfer service Grid Security Infrastructure, Community Authorization Service GRAM Job Management GridFTP data movement, Reliable File Transfer service, Metadata and replica management Monitoring technologies

May 8, 2003Earth System Grid6 ESG CS Mission Work closely with application groups to –Integate these (and other) components to provide end-to-end application solutions –Identify and, if possible, develop missing pieces –Evaluate what happens when real users apply our “solution” at scale Iterate to improve both Grid technologies and climate solutions

May 8, 2003Earth System Grid7 Contribution to IT New functionality: New features have been added to Globus Toolkit & SRM to meet climate community needs Robustness: “Production” deployment exposed limitations (functional and/or scale) and bugs in software tools Research: ESG requirements limitations exposed by deployment triggered new IT research directions

May 8, 2003Earth System Grid8 ESG Achievements Real value has been delivered to users –Mike Wehner, LLNL: “This has changed my life” Significant buy-in from climate scientists Middleware is more robust & easier to use Real interdisciplinary CS-climate scientist teams established National and international visibility for, and interest in, our work

May 8, 2003Earth System Grid9 Observations We are building a middleware and people infrastructure w/o long term commitment –How do we persuade the community to engage? Scope of the demand for ESG solutions is enormous, we can easily be overwhelmed –What is needed is an international environmental sciences Grid –How can ESG contribute to its realization, via leadership and technology development?

The Earth System Grid (ESG) Architecture DOE SciDAC ESG Project Review Argonne National Laboratory, Illinois May 8-9, 2003

May 8, 2003Earth System Grid11

May 8, 2003Earth System Grid12 ESG Architecture Metadata Catalog Replica Catalog Tape Library Disk Cache Attribute Specification Logical Collection and Logical File Name Disk ArrayDisk Cache Application Replica Selection Multiple Locations NWS Selected Replica gsiftp commands Performance Information and Predictions Replica Location 1Replica Location 2Replica Location 3 MDS

May 8, 2003Earth System Grid13 Metadata Catalog Replica Catalog Tape Library Disk Cache Attribute Specification Logical Collection and Logical File Name Disk ArrayDisk Cache Application Replica Selection Multiple Locations NWS Selected Replica gsiftp commands Performance Information and Predictions Replica Location 1Replica Location 2Replica Location 3 MDS ESG Architecture Remote Data Tookit Remote Calc. Toolkit Remote Viz Toolkit Generic Apps Grid Infrastructure BrokersInfoScheduleDataMonitorSecurity Grid Application Toolkit (Middleware) User Adm. Portals Applications Generic U.S. Users CDAT UsersFerret Users U.K. UsersClimate Community Commercial Users Community Outreach University Users Sponsors Networks ESG Grid U.K. NERC DataGrid CEOS Grid Other Grids

May 8, 2003Earth System Grid14 NCAR LBNL LLNL ISI ANL ORNL GSI CAS server CAS client MyProxy clientMyProxy server TOMCAT SECURITY services GRAM METADATA services FRAMEWORK services Auth metadata RLS NCAR MSS ORNL HPSS DATA storage The Earth System Grid THREDDS catalogs OGSA-DAISMCS TRANSPORT services gridFTP server/client TRM+DRM DRM openDAPg server ANALYSIS & VIZ services NCL openDAPg clientLAS server CDAT openDAPg client MONITORING services SLAMON daemon TOMCAT AXIS NERSC HPSS DISK mySQL xindice mySQLxindice mySQL

May 8, 2003Earth System Grid15 Typical Application Data (local) netCDF lib Application Data (remote) OPeNDAP Client Application OPeNDAP Via http Big Data (remote) ESG client Application ESG + DODS OPeNDAP Server ESG Server Distributed Application data OPeNDAP Via Grid Distributed Data Access Protocols Gridded Application

May 8, 2003Earth System Grid16 Data Movement Data (local) Data (remote) Big Data (remote) ESG client Application ESG + DODS ESG Server Distributed Analysis OPeNDAP Via Grid Additional Scenarios Big Data (remote) ESG Server

May 8, 2003Earth System Grid17 Grid and Network Infrastructure Grid-enabled storage systems Computational resources ? R CAS ESG services: information, replica, metadata, community authorization M Data consumers Data producers ESG: Collaboration Network