SURA IT Meeting – 22 March 2005 SURA IT Committee Meeting March 22, 2005 Sara J. Graves, Ph.D. Director, Information Technology and Systems Center University Professor, Computer Science Department University of Alabama in Huntsville Director, Information Technology Research Center National Space Science and Technology Center SCOOP Status

SURA IT Meeting – 22 March 2005 Whereas, the Southeastern Universities Research Association has proposed the creation of an open-access network of distributed sensors, linked via an ultra-fast network to state-of-the-art computing systems that track and model the southeastern coastal zone in real time and provide components of a more comprehensive coastal security infrastructure — known as Southeastern Coastal Ocean Observing Program (SCOOP); now, therefore, be it Resolved, That the Southern Governors’ Association supports SURA’s Southeastern Coastal Ocean Observing Program to bring more effective protection of life and property to the increasingly developed coastal zone, to offer a vehicle for bringing the extensive and widely dispersed intellectual talent of the ocean sciences community to address program of homeland security via an integrated and spatially distributed program, and to aid in addressing the ecological and environmental concerns endangering health and safety of inhabitants and marine resources.

SURA IT Meeting – 22 March 2005 SURA’s Southeastern Coastal Ocean Observing Program (SCOOP) will facilitate the assimilation of observational data into community models and provide a distributed data ingestion and support grid with broad band connectivity. This is expected to become a coastal counterpart to the Global Ocean Data Assimilation Experiment (GODAE) with emphasis on the southeast.

SURA IT Meeting – 22 March 2005 Board of Trustees Meeting Nov 2002 Data fusion is critical Modeled and observed fields must have equal representation Use GODAE (Global Data Assimilation Experiment) as a guide for CODAE (Coastal Ocean Data Assimilation Experiment) SURA is a strong brand (we should use it) Focused sub-regional efforts with specified deliverables which would be new and exciting Broad SURA effort targeted on building a culture supporting region-wide collaboration in shared scientific goals

SURA IT Meeting – 22 March 2005 Integrated Ocean Observing System (IOOS) Serves national needs for: Detecting and forecasting oceanic components of climate variability Facilitating safe and efficient marine operations Ensuring national security Managing resources for sustainable use Preserving and restoring healthy marine ecosystems Mitigating natural hazards Ensuring public health

SURA IT Meeting – 22 March 2005 National Federation of Regional Systems National Backbone Satellite remote sensing In situ sensing reference & sentinel station-network Link to global ocean component Data standards & exchange protocols Regional Systems Regional priorities Effects of climate change & Effects of land-based sources  Resolution,  Variables

SURA IT Meeting – 22 March A national coastal observing program will necessarily consist of regional and sub-regional components. 2.National, regional and sub-regional observing systems must consist of three interconnected aspects: (i) spatially distributed sensor arrays; (ii) data management and dissemination hubs; and (iii) nowcasting and forecasting models that are fused with assimilated observational data. 3. The creation and long-term viability of nested integrated and sustained coastal observing systems will depend on a high level of interagency coordination. Overarching Principles for Coastal Observing Programs

SURA IT Meeting – 22 March 2005 The SURA Coastal Ocean Observing and Prediction (SCOOP) program is an initiative to create a open-access, distributed national laboratory for scientific research and coastal operations. SCOOP is designed to complement the efforts of both Ocean.US - the organization responsible for implementing the national Integrated Ocean Observing System (IOOS)- and the coastal component of NSF’s Ocean Research Interactive Observatory Networks (ORION) project. The SCOOP emphasis is on interoperability in order to create a real-time observations system for both monitoring and prediction. Through SURA Universities, SCOOP will provide the expertise and IT infrastructure to integrate observing systems that currently exist, and incorporate emerging systems. This will promote the effective and rapid fusion of observed data with numerical models, and facilitate the rapid dissemination of information to operational, scientific, and public and private users. SCOOP Vision Statement

SURA IT Meeting – 22 March System of Systems Ocean Observing = IOOS & OOI & ORION Coastal Ocean Component of the Global Earth Observing System of Systems (GEOSS) Components: (i) Sensor arrays, (ii) Data management & communication, (iii) Predictive models 2.Distributed National Lab for Research & Applications IT Glue...Bricks & Mortar Research to Operations Academic & Federal Agency & Industry partnership 3.IT Enabling Big Science Environmental prediction Standards enable innovation Interoperable community solving the really big problems Overarching Goals for SCOOP

SURA IT Meeting – 22 March 2005 Planned Capabilities Validate accurate and timely short and long-term predictions Simultaneous measurements of winds, waves, currents, water density, nutrients, water quality, biological indices, and fish stocks under all conditions Focus on storm surge, wind waves, and surface currents, with special attention to predicting and visualizing phenomena that cause damage and inundation of coastal regions during severe storms, hurricanes and possibly tsunamis Bridge the gap between scientific research and coastal operations

SURA IT Meeting – 22 March 2005 SCOOP Science Goals Assess and predict the coastal response to extreme atmospheric events – focus on storm surge, flooding & waves Modular modeling tools for regional issues (wave coupling, sediment suspension, etc.) Standardized interfaces for data and (coupled) model interoperability Ensemble prediction – forecasts based on many independent models runs

SURA IT Meeting – 22 March 2005 SCOOP Research Goals Measure, understand and predict environmental conditions Provide R&D support for operational agencies including NOAA, the U.S. Navy, and others Include outreach and education components that assure relevance of their observing activities

SURA IT Meeting – 22 March 2005 SCOOP Objectives Develop and deploy standards and protocols for data management, exchange, translation and transport Implementation of existing standards and protocols (e.g. FGDC, OGC, web services, etc.) Application of Grid Technologies Deployment of the communications infrastructure to link ocean sensors operating in extreme environmental conditions to people who need timely information Cultivation of industry partners

SURA IT Meeting – 22 March 2005 Coordination is Key Ocean.US - National Office for Integrated and Sustained Ocean Observations coordinates development of an operational, integrated and sustained Ocean Observing System (created by NOPP) Integrated Ocean Observation System (IOOS) a national effort to create an Integrated Ocean Observing System National Oceanographic Partnership Program (NOPP) 15 federal agencies providing leadership and coordination of national research and education programs National Federation of Regional Associations provide a framework for orchestrating regional collaborations NSF Ocean Research Interactive Observatory Networks (ORION) an emerging network of science-driven ocean observing systems

SURA IT Meeting – 22 March 2005 Interoperability is Key Ocean.US Data Management and Communications (DMAC) Plan provides the framework for interoperability Open Geographic Information Systems (GIS) Consortium (OGC) an open consortium of industry, government, and academia developing interface specifications to support interoperability Marine Metadata Interoperability a community effort to make marine metadata easier to find, access and use

SURA IT Meeting – 22 March 2005 Interoperability Demonstration NOAA and ONR grant recipients collaboration

SURA IT Meeting – 22 March 2005 Funding provided by ONR, NOAA 2004 List of SCOOP Partners: SCOOP System Development Consortium for Oceanographic Research and Education Gulf of Maine Ocean Observation System (GoMOOS) Louisiana State University, Center for Computation &Technology Louisiana State University, Coastal Studies Institute Southeast Atlantic Coastal Ocean Observing System (SEACOOS) Southeast Coastal Ocean Observations Regional Association (SECOORA) Texas A&M University & Gulf Coast Ocean Observing System (GCOOS) University of Alabama in Huntsville University of Delaware (Mid-Atlantic Regional Association (MACOORA) University of Florida University of Miami, Center for Southeastern Tropical Advanced Remote Sensing University of North Carolina Virginia Institute of Marine Science

SURA IT Meeting – 22 March 2005 SCOOP Program Elements 1.Data Standards Metadata standards – compliant with existing and emerging standards Standard data models – to facilitate aggregation 2.Data Grid OGC Web services – for distributed maps and data Augmenting with new data, e.g., surface currents 3.Model Grid Storm surge & wave prediction Modular, standardized prediction system

SURA IT Meeting – 22 March 2005 SCOOP Data Architecture (high level) NOAA Web Browsers GIS Clients NDBC Regional Association Data Center (Archive) OGC Protocols Other Regional Association Data Centers HTTP & HTML LDM…??? NDBC MODEM HTTP & HTML SCOOP Modeling Partners TBD…??? Regional Data Provider #1 Regional Data Provider #2 Regional Data Provider #N SCOOP Modeling Partners TBD…??? Transport Mediums

SURA IT Meeting – 22 March 2005 SCOOP Prediction System All Versions 1.Standard naming conventions – Adopt existing community standards where appropriate (e.g., CF or NCEP) and add our own conventions only when necessary. 2.Mechanisms for tracking metadata, e.g., provenance, forcing, source of OBCs, forcing used to create OBCs, etc. 3.Portals – entry point for access to models & model output. Deals with authentication & authorization.

SURA IT Meeting – 22 March 2005 SCOOP Prediction System, Version 1.0 Modular wind forcing Modular embedded regional models Coupled models – for existing groups Using existing computational resources Verification – real time model-data comparisons Model-GIS interface & OGC Web services Web mapping with roads, etc. Web mapping with time sequences (WMS) Standardized time-series verification Openioos.org for displaying results Other…?

SURA IT Meeting – 22 March 2005 SCOOP Operational Prediction System Version 1.0 Regional Model Center #1 Operational Wave Predictions (BIO/GoMOOS) Operational Tide/Surge Predictions (SABLAM) Coupled Wave-Surge Predictions (Miami) Large Scale Response NOAA/NCEP (ETA) NOAA/NCEP/UNC? (EDAS) Archive Enhanced Winds (Miami) Forcing Regional Response Standardize model interfaces Regional Model Center #2 Standardize Transport/Encapsulation XML, FTP, LDM, OPeNDAP…?

SURA IT Meeting – 22 March 2005 SCOOP Verification & Visualization Prediction System Standardize model-GIS interfaces Regional Web Server #1 Modeling Center #1 (Regional or otherwise) Modeling Center #2 (ditto) Regional Data Center #1 Information Providers Regional Web Server #2 OGC, RSS…? Regional Data Center #2 Standardize verification tools & data Data System

SURA IT Meeting – 22 March 2005 Prediction System Task Elements for Version 1.0 Task:Lead Partner: Data standardsTAMU Data transport UAH Data translation & mgmtUAH Coupled modelingMiami Nested ModelingVIMS Customized configurationTAMU Visualization ServicesLSU Verification & validationMiami Computing & storage resourcesLSU SecurityTAMU Grid management middlewareLSU Web Mapping DemonstrationGoMOOS

SURA IT Meeting – 22 March 2005 Users, Modeling Partners, other Data Centers, etc. Regional Association Data Centers Regional Data Providers Data Provider Data Translation Services Data Provider SCOOP Data Architecture: High level Services Metadata only data Data and Metadata data Modeler / Data Provider Observation Data Model Data Access Services Metadata Services SCOOP Catalog data Archive/Repository Broker Data Access Services Model or Application User Interface GeoSpatial OneStop / FGDC Clearinghouse

SURA IT Meeting – 22 March 2005 Regional Association Data CentersRegional Data Providers Data Provider Data Provider SCOOP Data Architecture Specifics: Data Acquisition – example technologies to support dynamic transport and metadata cataloging Metadata only data Metadata and Data data Modeler / Data Provider Observation Data Model Data Access Services Metadata Services data Archive/Repository Broker Data Access Services Data Transport Metadata Cataloging LDM e.g., LDM e.g., XML, Metadata Harvest SCOOP Catalog

SURA IT Meeting – 22 March 2005 Users, Modeling Partners, other Data Centers, etc. Data Translation Services data Modeler / Data Provider FTP OPeNDAP OGC Model or Application User Interface GeoSpatial OneStop / FGDC Clearinghouse SCOOP Data Architecture Specifics: Data Discovery & Access – example technologies to support dynamic transport, analysis and visualization Metadata Query Services data Archive/Repository Broker FTP OPeNDAP OGC ESML IOOS Interoperability Demo Z39.50 SOAP OGC WMS XML HTTP FTP Data Discovery Data Access Regional Association Data Centers Regional Data Providers SCOOP Catalog

SURA IT Meeting – 22 March 2005 data Modeler / Data Provider User GeoSpatial OneStop SCOOP Information Architecture: Example metadata exchange technologies to support Data Discovery IOOS Interoperability Demo Z39.50 XML Regional Association Data/Service Centers Regional Data Providers Data Provider data Local Metadata SCOOP Data Dictionary SCOOP Catalog SOAP OGC Get Capabilities Metadata Harvest Metadata Harvest SCOOP Interactive Search U/I Model or Application FGDC Records HTTP WMS data list & metadata Automated Data Discovery SOAP XML ? Metadata Harvest Metadata Services Data Discovery Metadata Population Users, Modeling Partners, other Data Centers, etc. Ingest SvcsQuery Svcs Manual Updates SCORE

SURA IT Meeting – 22 March 2005 SCORE: Accomplishments & Plans SCORE is the catalogs and services infrastructure for SCOOP data management Data & Model Survey provided initial snapshot of partners’ data (observations and model results) Developed database schema for SCORE to support –Strawman SCOOP Catalog: requesting input on improved capabilitiesStrawman SCOOP Catalog –IOOS demo: working with GoMOOS team to integrate catalog with demoIOOS demo –FGDC Clearinghouse to support Geospatial One-Stop: plan to create FGDC metadata records from SCOREFGDCGeospatial One-Stop Issues –What data management functionality is needed within SCOOP? Metadata services for data collections, data files/streams, general model information, information on specific model runs,… –How to coordinate metadata and data management across sites? –How to automate population of SCORE?

SURA IT Meeting – 22 March Environmental Prediction Prediction systems fuse models & observations Nonlinear dynamics limits predictability – Lorentz’s seagull Probability and statistics – ensemble modeling 2.Hurricane Surge & Waves Biggest uncertainty in the winds Ensemble of winds: different models or different simulations New paradigm & new metrics for skill assessment 3.Research to Operations Improving upon SLOSH – a good idea 30 years ago GIS compatibility enabling application & visualization OpenIOOS.org is the high visibility “front end” Science Goals for Version 2.0

SURA IT Meeting – 22 March 2005 Regional Archives Ensemble wind fields from varied and distributed sources ADCirc ElCirc WAM/SWAN Ensemble of models run across distributed resources Archive Verification Visualization Analysis, storage and cataloguing of output data Select region and time range Transform and transport data Wind Forcing Wave and/or Surge Models Result Dissemination Synthetic Wind Ensemble NCEP MM5 NCAR or OpenIOOS Version 2.0

SURA IT Meeting – 22 March 2005 SCOOP: Data-to-Model (D2M) Realtime Transport and Translation (Nested Model) Scenario UNC TAMU VIMS TAMU, UF, Others? ADCIRC MM5 Translation Services subset subsample re-format re-grid ELCIRC Model X LDM, OPeNDAP, FTP Push/pull POC: Rick Luettich, Brian Blanton POC: Gerry Creager POC: Harry Wang LDM-push POC: Matt Smith, Ken Keiser (UAH) LDM-push Water levels NCEP (NAM) Wind Forecasts Atmospheric Models Regional Oceanic/Coastal Models Localized Models, Users and Archives High-Res Wind Forecasts LDM-push D2M Node Translated Water levels (1) (2) (3) (4) (1a) WRF (future) (1)Atmospheric Model products are “translated” through D2M to the form requested by the client model. Currently, using ftp-pull, all NAM grids 0-84h for the 4 runs (00, 06, 12, 18 UTC) of AWIP12 and AWIP32 are sent to a D2M node and translated. (2)Via LDM, UNC, TAMU, & UF have access to the raw and translated model data. (3)Partners use translated ob/model data in their models. Then push their results to a D2M node. Currently, ADCIRC output files (text and netCDF) are being pushed to a D2M node (for translation) and other modeling partners via LDM. (4)Resulting translated data products area pushed to a client model’s site and made available for other transport vehicles (FTP, OPeNDAP, OGC, etc) for use in retrospective studies and other applications. Likewise the output of other models can be processed through D2M for translation steps requested by other client models. Translated Winds and fluxes Translation Services subset subsample re-format re-grid D2M Node ESML LDM-push (FTP-pull) Alternate

SURA IT Meeting – 22 March 2005 Data Management Goals Version 1 Provided a high-level data catalog for SCOOP data discovery, providing descriptions of partner data holdings and pointers to partner data access points (web, ftp, OPeNDAP, etc.) –Based initial catalog on Data & Model Survey results Coordinated with Data Transport (Task 2) to develop initial LDM network to exchange data in near real time among SCOOP partners. Coordinated with Data Standards (Task 1) on development of metadata keywords for SCOOP Version 2 Expand SCOOP data discovery capabilities based on evolving data management practices of SCOOP partners. –Support IOOS Demo –Field an FGDC Clearinghouse node for SCOOP Monitor Marine Metadata Interoperability activities and their potential interaction with SCOOPMarine Metadata Interoperability –Assist SCOOP partners in developing standard metadata to describe their data collections Continue coordination with all partners on data management issues

SURA IT Meeting – 22 March 2005 GLOBE AMSU-A Knowledge Base ITSC Coastlines Countries MCS Events Cyclone Events AMSU-A Channel 01 AMSU-A data overlaid with MCS and Cyclone events, merged with world boundaries from GLOBE. Merged data product for on-demand visualization Distributed Data Integration

SURA IT Meeting – 22 March 2005 Heterogeneity Leads to Data Usability Problems Science Data Characteristics Many different formats, types and structures (18 and counting for atmospheric science alone!) Different states of processing (raw, calibrated, derived, modeled or interpreted) Enormous volumes

SURA IT Meeting – 22 March 2005 Interoperability: Accessing Heterogeneous Data The Problem DATA FORMAT 1 DATA FORMAT 1 DATA FORMAT 2 DATA FORMAT 2 DATA FORMAT 3 DATA FORMAT 3 READER 1 READER 2 FORMAT CONVERTER APPLICATION ESML LIBRARY APPLICATION DATA FORMAT 1 DATA FORMAT 1 DATA FORMAT 2 DATA FORMAT 2 DATA FORMAT 3 DATA FORMAT 3 The Solution ESML FILE ESML FILE ESML FILE ESML FILE ESML FILE ESML FILE One approach: Enforce a standard data format, but… Difficult to implement and enforce Can’t anticipate all needs Some data can’t be modeled or is lost in translation Converting legacy data is costly A better approach: Interchange Technologies Earth Science Markup Language

SURA IT Meeting – 22 March 2005 What is ESML? It is a specialized markup language for Earth Science metadata based on XML - NOT another data format. It is a machine-readable and -interpretable representation of the structure, semantics and content of any data file, regardless of data format ESML description files contain external metadata that can be generated by either data producer or data consumer (at collection, data set, and/or granule level) ESML provides the benefits of a standard, self-describing data format (like HDF, HDF-EOS, netCDF, geoTIFF, …) without the cost of data conversion ESML is the basis for core Interchange Technology that allows data/application interoperability ESML complements and extends data catalogs such as FGDC and GCMD by providing the use/access information those directories lack.

SURA IT Meeting – 22 March 2005 ESML IN ACTION: Ingest surface skin temperature data in Numerical Models Reanalysis GRIB files Reanalysis GRIB files MM5 GOES ESML file ESML file ESML file ESML Library NUMERICAL WEATHER MODELS (MM5, ETA, RAMS) Scientists can: Select remote files across the network Select different observational data to increase the model prediction accuracy Purpose: Use ESML to incorporate observational data into the numerical models for simulation Skin temperatures come in a variety of data formats GOES – McIDAS Reanalysis Data - GRIB MM5 Model - Binary AVHRR – HDF MODIS - EOS-HDF