The Earth System Grid: A Visualisation Solution Gary Strand

Introduction

Acknowledgments ESG Development Team PI’s Veronika Nefedova (ANL) Ann Chervenak (ISI/USC) Carl Kesselman (ISI/USC) David Bernholdt (ORNL) Kasidit Chanchio (ORNL) Line Pouchard (ORNL) Alex Sim (LBNL) Arie Shoshani (LBNL) Bob Drach (LLNL) Dave Brown (NCAR) Gary Strand (NCAR) Jose Garcia (NCAR) Luca Cinquini (NCAR) Peter Fox (NCAR) PI’s Ian Foster (ANL) Don Middleton (NCAR) Dean Williams (LLNL)

Current Practices Scientist (or others) wants a visualisation Visualisation person gets appropriate data after verifying with data manager as to the name, location, total size, etc. Data moved to “local” machine that has visualisation tools Visualization created on “local” machine Hopefully, someone remembers to archive the visualisation

Simple Vis Example

Problems in the process What if the data cannot be found (e.g. we have 1.2 million files, 73 TB of data), or the data manager is unavailable? What if there isn’t enough disk space or sufficient other resources? What if a better visualisation tool is located elsewhere? What if the visualisation should be shared? What if the visualisation is lost? ESG is part of the answers to these questions

What is ESG? ANL: Computational grids, & grid-based applications LBNL: Climate storage facility LLNL: Model diagnostics & inter-comparison USC/ISI: Computational grids, & grid-based applications NCAR: Climate change predication and scenarios LANL: Next generation coupled models & computing ORNL: Climate storage & computational resources

ESG Architecture LBNL ANL NCAR LLNL ORNL ISI HPSS High Performance Storage System disk ANL CAS Community Authorization Services NCAR SRM Storage Resource Management gridFTP Striped server gridFTP server openDAPg server Tomcat servlet engine MyProxy server LLNL disk MCS client MyProxy client CAS client RLS client SRM Storage Resource Management Note the many services needed to build such a Grid. gridFTP server GRAM gatekeeper gridFTP server ORNL gridFTP SRM Storage Resource Management ISI gridFTP gridFTP server SRM Storage Resource Management MCS Metadata Cataloguing Services SOAP HPSS High Performance Storage System RLS Replica Location Services RMI MSS Mass Storage System disk disk

Solutions Metadata catalogue service (MCS) What happens when data cannot be found, or the data manager is unavailable? Metadata catalogue service (MCS) Replica location service (RLS)

MCS and RLS and Metadata Services EXTRACTION DISPLAY BROWSING QUERY ESG CLIENTS API & USER INTERFACES Data & Metadata Catalog Dublin Core Database COARDS mirror XML Files COMMENTS METADATA HOLDINGS ANNOTATION VALIDATION METADATA ACCESS (update, insert, delete, query) SERVICE TRANSLATION LIBRARY CORE METADATA SERVICES AGGREGATION DISCOVERY METADATA & DATA REGISTRATION PUBLISHING HIGH LEVEL METADATA SERVICES SEARCH & DISCOVERY ADMINISTRATION BROWSING & DISPLAY ANALYSIS & VISUALIZATION

Solutions (contd.) Hierarchical Resource Manager (HRM) What if there isn’t enough disk space or sufficient other resources? Hierarchical Resource Manager (HRM)

HRM

Solutions (contd.) Distributed visualization What What if a better visualization tool is located elsewhere? Distributed visualization

CDAT Example of an ESG Script Access The next-generation language, Python, is used to access the Earth System Grid (ESG) at LLNL Import cdms, vcs db = cdms.open(“ldap://localhost:389/database=demo,ou=PCMDI,o=LLNL,c=US”) f = db.open( “ncep_reanalysis_mo”) ds = f(‘ts’) x=vcs.init( ) x.plot(ds)

CDAT: Example of an ESG GUI Client Access

Solutions (contd.) What if the visualization should be shared? Access Grid plus Visualisation Tool

Collaborative Environments Science Portals + AccessGrid: University of Michigan (Knoop, Hardin) Vegetation & Ecosystem Mapping Program (VEMAP) NCAR/SCD VETS/KEG Argonne National Labs

Conclusions Visualisation can require as many services and resources as the initial computation Many sites do not offer sufficient resources for the visualisations earth sciences require ESG provides, and will provide, the tools that enable visualisation on a grander scale

Conclusions (contd.) ESG tools enable better data access, better data knowledge, and the processes of collaboration for the needs of investigating, visualising, and learning