NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid METADATA DEVELOPMENT for the EARTH SYSTEM GRID Luca Cinquini (SCD/NCAR) for the Earth System Grid collaboration

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid Metadata-centric view of ESG services METADATA SERVICES METADATA SERVICES USER AUTHENTICATION AND AUTHORIZATION USER AUTHENTICATION AND AUTHORIZATION ACCESS AND AUTHORIZATION METADATA DATA TRANSPORT LOCATION METADATA SYSTEM MONITORING AND CONTROL SYSTEM MONITORING AND CONTROL LOGGING METADATA DATA SEARCH & DISCOVERY CONTENT METADATA ANNOTATION & HISTORY METADATA DATA ANALYSIS & VISUALIZATION DATA ANALYSIS & VISUALIZATION AGGREGATION METADATA DATA BROWSING CATALOGUING METADATA

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid ESG Metadata Services Goal Functionality Services responsible for the creation, management and utilization of metadata associated with geophysical data Functionality:  Metadata extraction (automatically, from files in different format and according to various possible metadata standards)  Metadata conversion (from one standard to another)  Metadata aggregation (associated with data collections)  Metadata annotation (manually by humans)  Metadata validation (basic quality control of metadata)  Registration (population of metadata holdings)  Harvesting (combination of metadata from different repositories)  Metadata browsing and display (for humans)  Search and discovery of data through metadata  Metadata query (by agents or clients for data analysis and visualization)

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid ESG Metadata Services Architecture 3-layers architecture: Metadata Holdings: physical metadata content, stored in a system of relational and/or XML native databases Core Metadata Services: modules and libraries that mediates all access to the Metadata Holdings (insert, update, delete, query) – expose an API that hides the specific implementation of the databases and query languages High Level Metadata Services: system of applications that make use of the Core Metadata Services to fulfill a specific atomic functionality – will be invoked by external clients

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid ORNL: Climate storage & computational resources ORNL: Climate storage & computational resources LANL: Next generation coupled models & computing LANL: Next generation coupled models & computing ANL: Computational grids, & grid-based applications ANL: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications NCAR: Climate change predication and scenarios NCAR: Climate change predication and scenarios LBNL: Climate storage facility LBNL: Climate storage facility LLNL: Model diagnostics & inter-comparison LLNL: Model diagnostics & inter-comparison The Earth System Grid

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid The Earth System Grid “Synergistic collaboration” among several US national labs and research centers (ANL, ISI, LBNL, LLNL, NCAR, ORNL) involved in atmospheric science and scientific computing 3 year project funded by the DOE Scientific Discovery through Advanced Computing (SciDAC) Goal: build the next generation computational and data management environment for the geosciences: a system of geographically distributed data and computational centers which will allow seamless access for earth scientists to data repositories, analysis tools and computational resources Strategy: application and extension of Grid technologies (and other IT innovations) to the geosciences Initial focus on next generation climate modeled data (CCSM)

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid ESG areas of development Authentication and Authorization services : application of Globus technologies for secure data management and access (PKI certificates, proxy delegation, Community Authentication Services, web interfaces) Data Transport Services: based on gridFTP protocol and implementation (high speed, tunable, multi-stream, reliable), extensions for multi-file management and connection to offline storage systems (Hierarchical Storage Management), and for transparent data access and operations (grid-enabled DODS) Metadata services (for data management, access, search & discovery, annotation, analysis, etc.) Other services: Data Analysis and Visualization, Task Management, Monitoring and Control, etc.

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid METADATA EXTRACTION METADATA EXTRACTION METADATA DISPLAY METADATA DISPLAY METADATA BROWSING METADATA BROWSING METADATA SEARCH, QUERY & DISCOVERY METADATA SEARCH, QUERY & DISCOVERY ESG CLIENTS API & USER INTERFACES Replica Location Services Metadata Cataloguing Services XML DB THREDDS catalogs METADATA HOLDINGS METADATA ANNOTATION METADATA ANNOTATION METADATA VALIDATION METADATA VALIDATION METADATA ACCESS (update, insert, delete, query) METADATA ACCESS (update, insert, delete, query) SERVICE TRANSLATION LIBRARY SERVICE TRANSLATION LIBRARY CORE METADATA SERVICES METADATA AGGREGATION METADATA AGGREGATION METADATA CONVERSION METADATA CONVERSION METADATA & DATA REGISTRATION METADATA & DATA REGISTRATION PUBLISHING HIGH LEVEL METADATA SERVICES SEARCH & DISCOVERY ADMINISTRATION BROWSING & DISPLAY ANALYSIS & VISUALIZATION

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid ESG Metadata Services Current Development Currently developing or evaluating the following technologies : Replica Location Services : database to manage and index multiple copies of the same data stored at different centers Metadata Cataloguing Services : relational database to store scientific metadata (developed for high energy physics and geophysical data) XML native databases (Apache Xindice) THREDDS (by Unidata ) : system for hierarchical cataloguing of datasets and associated metadata ( NcML (Netcdf Markup Language) : XML language for encoding of metadata associated with data in netcdf format (and more…)

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid ESG Metadata Policy Premise : geophysical sciences are too broad and complex to impose a single, omnicomprehensive metadata standard to capture the relevant information for all datasets, projects, instruments, scientists ESG will not mandate use of any metadata schema or convention Allow data providers, scientists to use their metadata of choice, provide technologies and tools to store and access metadata through common services (MCS, XML DB, THREDDS catalogs) Encourage development and reuse of a limited set of domain- specific standards (climate data, radar data, airborn instrumentation etc), encoding in XML (according to community developed schemas), interoperability and combination of schemas (XML namespaces, RDF, ontologies)

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid Netcdf Markup Language (NcML) Work in progress, collaboration between ESG, Unidata and the University of Florence Definition: XML representation for data following the netcdf model Features:  Express metadata associated with data in netcdf format  Definition of coordinates and coordinate systems (capturing netcdf conventions)  Aggregation/subsetting  Definition of new data, restracturing of existing data (virtual datasets)  Interoperability with openGIS and ISO  Also, possibly extend the model to other data formats (HDF, Grib etc.) Strategy: develop a system of XML schemas each covering a specific domain (advantages: more flexible, mantainable and extensible). Keep it simple!

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML: schemas architecture Netcdf core (generic netcdf data) Netcdf core (generic netcdf data) Netcdf Coordinate Systems (netcdf conventions for coord, coord systems) Netcdf Coordinate Systems (netcdf conventions for coord, coord systems) Netcdf (virtual) dataset (operations on data) Netcdf (virtual) dataset (operations on data) Netcdf Geo Coordinate Systems (geo-referenced coord systems) Netcdf Geo Coordinate Systems (geo-referenced coord systems) openGIS-ISO Reference Coordinate Systems openGIS-ISO Reference Coordinate Systems Other schemas for openGIS- ISO

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML: core schema For XML encoding of metadata (and data) of any generic netcdf file Objects: Netcdf, Dimension, Variable, Attribute Beta version reference implementation as Java library (

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid E xample : two-dimensional latitude, longitude coordinate variables (CDL) dimensions:  xc = 128;  yc = 64;  lev = 18; variables:  float T(lev,yc,xc);  T:long_name = "temperature"; T:units = "K"; T:coordinates = "lon lat";  float xc(xc);  xc:long_name = "x-coordinate in Cartesian system"; xc:units = "m";  float yc(yc);  yc:long_name = "y-coordinate in Cartesian system"; yc:units = "m";  float lev(lev);  lev:long_name = “altitude levels"; lev:units = “km";  float lon(yc,xc);  lon:long_name = "longitude"; lon:units = "degrees_east";  float lat(yc,xc);  lat:long_name = "latitude"; lat:units = "degrees_north";

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML core schema

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML core schema

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML: coordinate systems schema Generalization and unification of netcdf conventions for coordinates and coordinate systems

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid Coordinate Systems extension to NcML

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid Coordinate Systems extension to NcML

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid Coordinate Systems extension to NcML <nc:variable name="T" shape="lev yc xc" type="float” coordinateSystems=“implicit geo pressure”>

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid Aggregation in NcML XML naturally suited to represent aggregation of netcdf data Rules for representing an aggregation hierarchy:  Allow netcdf nodes to contain other netcdf nodes  Factor out (i.e. in the parent netcdf node) all common structure between two nodes  Structure defined in a netcdf node overrides that defined in a parent netcdf node

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML aggregation over existing coordinate (time)

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML aggregation over variables

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML double aggregation

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid NcML double aggregation

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid Other NcML planned development Subsetting of data Compute derived data Extensions for interoperability with openGIS and ISO standards :  Establish a bond between Atmospheric Research and Geo-spatial communities  Allows import of NcML data into GIS tools, export of GIS data in netcdf format

NIEeS Workshop, Cambridge (UK), Sep 2002 Luca Cinquini for the Earth System Grid Conclusions ESG is very active in the research and development of metadata schemas, services and technologies We are very interested in collaborating with other projects and institutions to the definition and adoption of metadata standards for the geosciences and to work at interoperability technologies among standards