Embedding Live Access Server into GFDL Data Portal Infrastructure K.O’Brien (PMEL), S.Nikonov (GFDL), R.Schweitzer (PMEL), S.Hankin (PMEL), V.Balaji (GFDL) The 6 th GO-ESSP Workshop June Jussieu, Paris

Outlines  Curator metadata DB, part of GFDL FMS Runtime Environment (FRE) is a centralized metadata storage for entire modeling process.  Live Access Server (LAS) is important component of GFDL Data Portal.  FRE –> Curator DB –> LAS metadata stream.  Benefits of symbiosis. The 6 th GO-ESSP Workshop June Jussieu, Paris

Live Access Server + Curator = Live Access Server + Curator = LASurator (not LACerater !!!) The 6 th GO-ESSP Workshop June Jussieu, Paris   Symbiotic self configuring system combining LAS and Curator metadata DB.   Part of FRE   Essential part of GFDL Data Portal FRE Data Portal LASurator

 Technical part of modeling process consists of three important parts: assembling model, configuring it and running simulation.  Two ways of conducting these stages: 1.write once and forever sophisticated universal scripts with a lot of input parameters needed for running them; 2.write tools which generate disposable scripts with configurations’ parameters needed for simulation hardcoded in these scripts.  FRE was design using the 2 nd way avoiding scientists the long road of configuring model experiment every time. 1 st version offers XML metadata file as user interface for configuring model and experiment.  The 2 nd version will give convenience and automation to scientists in controlling the process of model building through more user friendly interface then XML file based on centralized storage – Curator DB. Curator DB is important part of FRE The 6 th GO-ESSP Workshop June Jussieu, Paris

Modes of working Curator DB within FRE  Research mode (component oriented) - modeler introduces new physical processes / parameterization / algorithmizations / components from newly developed modules. New entities have to be described in database.  Production mode (simulation oriented) – experimenter composes coupled model from available components described in database, builds scenario, postprocessing plan and runs experiment. All this activity is recorded in database. Thoroughly elaborated very friendly GUI is the critical thing for these modes otherwise users will avoid the database load stage => DB will be empty. Thoroughly elaborated very friendly GUI is the critical thing for these modes otherwise users will avoid the database load stage => DB will be empty.  Automatic mode – applications write metadata into database harvesting it from model output data files or queries it from DB during data searching.  Publishing mode – metadata is extracted from DB by publishing tools for describing public data. The 6 th GO-ESSP Workshop June Jussieu, Paris

Curator DB on different stages of modeling process Metadata Curator DB Component Building Model Assembling Experiment Preparation Postprocessing Data Portal Service The 6 th GO-ESSP Workshop June Jussieu, Paris FMS Runtime Environment (currently developed version) in development currently

Curator DB Design Compartments  Process Domains & Physical Process: decomposition of physical reality into homogeneous domains with descriptions of theoretical approaches for processes there considered for modeling.  Algorithmization: describes program modules of elementary physical processes  Composition: components, couplers; drivers; technical environment needed for assembling model as computer application  Simulation: metadata on simulations and model output data  Publishing: all metadata on data available for public; includes descriptions needed for Data Portal software (OPeNDAP, LAS). The 6 th GO-ESSP Workshop June Jussieu, Paris

Curator database   Contains a lot of metadata information needed for entire modeling process   Well ordered “information architecture” Live Access Server   Configurable metadata access   Configurable data browse/access The 6 th GO-ESSP Workshop June Jussieu, Paris

The GFDL IPCC hierarchy, for example: IPCC (Project) GFDL (Institution) CM2.0 (Model number) Climate of the 20 th Century (Scenario) Realization 1 (Which realization) Run 1 (Which run) 3 hourly data (Temporal domain) Jan 1991 – Dec 2000 (Dates of on-line data) The 6 th GO-ESSP Workshop June Jussieu, Paris Atmosphere (Domain)

GFDL CM2.0 CM2.1 Climate of the 20 th Century 1%/year CO2 increase scenario (to quadrupling) Climate of the 20 th Century Run 2 Run 1 Run 2 Run 1 AtmosOceanIce AtmosOceanIce AtmosOceanIce AtmosOceanIce AtmosOceanIce annual monthlydaily 3hrmonthlydaily 3hr annual Jan 2046 – Dec 2050 Jan 2096 – Dec 2100 Jan 2196 – Dec 2200 Jan 2146 – Dec 2150 Precipitation Surface Air Temperature Surface Latent Heat Flux Sea Level Pressure The 6 th GO-ESSP Workshop June Jussieu, Paris Metadata Tree

Challenge:   How to configure LAS to allow efficient user interaction and reflecting all variety of available data   How to minimize work for LAS installer Solution:   Use Curator DB to pull together pertinent metadata on experiments   Modify LAS utility addXML to read metadata from mySQL database rather than individual or aggregated files   Create generic velocity templates to present in friendly HTML for users The 6 th GO-ESSP Workshop June Jussieu, Paris

Simple but effective architecture GFDL Data Storage LAS THREDDS server Aggregations’ URLs metadata addXML Categories The 6 th GO-ESSP Workshop June Jussieu, Paris metadata Extractor, XMLGenerator metadata Extractor, XMLGenerator metadata Extractor/ XMLGenerator metadata Extractor/ XMLGenerator metadata

Behind the scenes The 6 th GO-ESSP Workshop June Jussieu, Paris Data Curator LAS XML Aggregation XMLs OPeNDAP access to data LAS access to products metadata

Tools populating Curator  Populating Curator DB is automatic.  One piece of software scans data storage following the list of public experiments from Curator DB. Traversing data files it extracts metadata and write it in Curator. In next version of FRE metadata will populate DB automatically in course of all stages from assembling model through experiment running.  Another tool analyses metadata from Curator and creates aggregations records in accordance to the aggregation criterion. Currently, Experiment is used as a sorting criterion for upper level category.  Last stage is generating THREDDS configuration XML files based on prepared metadata in Curator. The 6 th GO-ESSP Workshop June Jussieu, Paris

Interaction FRE ↔ Curator → LAS (final goal) The 6 th GO-ESSP Workshop June Jussieu, Paris Model Configuration XML metadata FMS Config data metadata FRE LAS XML LAS Aggregation XMLs in development currently GUI

LAS oriented Curator DB design features Created specific tables for Inventorying metadata about time spans and variables of available data; Inventorying metadata about time spans and variables of available data; Describing projections in details; Describing projections in details; Storing THREDDS aggregations descriptions including such fields as type of averaging, domain of variable (atmos, land,…), time limits, URL, aggregation criterion. Storing THREDDS aggregations descriptions including such fields as type of averaging, domain of variable (atmos, land,…), time limits, URL, aggregation criterion. Dynamic hierarchy Implemented flexibility of criteria for categories hierarchy giving to user freedom in choosing his own interface layout preferences. The 6 th GO-ESSP Workshop June Jussieu, Paris

Curator DB table samples The 6 th GO-ESSP Workshop June Jussieu, Paris

Imagine a search interface with Ajax talking to Curator DB or machine oriented auxiliary DB automatically designed based on adopted ontology and populated by RDF triples from main metadata DB - Curator (like it’s proposed and explored in ESG project) …. The 6 th GO-ESSP Workshop June Jussieu, Paris Future development –Discovery/Navigation Interface As the user constrains the search, output and menu lists adjust immediately.

Imagine a metadata summary of this experiment here … Experiment 2 The 6 th GO-ESSP Workshop June Jussieu, Paris

The 6 th GO-ESSP Workshop June Jussieu, Paris

Desktop:Matlab, IDL, IDV, Ferret, GrADS, … Information Products Files:netCDF,binary,spreadsheet, GIS layer, …

Benefits  Curator DB contains extensive metadata (aggregations, gridspec descriptions, time spans of data availability).  LAS configuration can be pulled directly from Curator DB. Simplifies the configuration of the complicated information hierarchy for the LAS installer. (using LAS “categories”) Simplifies the configuration of the complicated information hierarchy for the LAS installer. (using LAS “categories”) Simplifies the complex mosaic of datasets which are presented to the LAS user. Simplifies the complex mosaic of datasets which are presented to the LAS user.  LAS addXML tool works faster. The 5 th GO-ESSP Workshop June , LLNL

Thanks!Questions? The 6 th GO-ESSP Workshop June Jussieu, Paris

Process Domain Process Algorithmization Assembling Simulation Domains Atmosphere Ocean Ice Land Surf_Boundary Layer Rivers Lakes Compartment Structure of Curator Database Dynamics Radiation IceProc BiotaProc Hydrology CloudProc Chemistry Convection ProcCodeBase NumArtificies GridSpecs BoundCond TracerModels NameLists DataSets Components CmpPMIOD CmpDrivers Services Versioning Compiling PlatformEnv Others...Projects Experiments Scenarios PostProc Variables Fields/Files Aggregations Realizations DomConstituents The 5 th GO-ESSP Workshop June , LLNLInitCond CouplModels