Configuring NVODS Datasets in LAS Joseph McLean TMAP/PMEL/NOAA and JISAO/UW OAR WebShop 2002.

Slides:



Advertisements
Similar presentations
Characterisation of observations François Bonnarel, Mireille Louys, Anita Richards, Alberto Micol, Jonathan McDowell, Igor Chilingarian, et al.
Advertisements

Weathertop Consulting, LLC Server-side OPeNDAP Analysis - A General Approach Utilizing Legacy Applications through TDS Roland Schweitzer Weathertop Consulting,
A Coherent and Practical End-to-End Metadata Strategy using Existing Standards and Tools for Clinical Research Stephane AUGER Danone Research, FRANCE.
DataTools Models Data, models and tools: Dealing with any complex hydraulic engineering problem invariable use is made of: data, models and tools.
Recent Work in Progress
® Page 1 Intel Compiler Lab – Intel Array Visualizer HDF Workshop VI December 5, 2002 John Readey
Environmental GIS Nicholas A. Procopio, Ph.D, GISP Some slides from Lyna Wiggins (Rutgers University)
The MetaDater Model and the formation of a GRID for the support of social research John Kallas Greek Social Data Bank National Center for Social Research.
Exploring large marine datasets using an interactive website and Google Earth Jon Blower, Dan Bretherton, Keith Haines, Chunlei Liu, Adit Santokhee Reading.
LAS & NVODS S.Hankin -- Sep NVODS and the Live Access Server (LAS) Steve Hankin, PI (NOAA/PMEL) Jon Callahan (U of WA/JISAO) Ansley Manke (NOAA/PMEL)
® Page 1 Intel Compiler Lab – Intel Array Visualizer HDF Workshop VIII October 27, 2004 John Readey
CEOS/WGISS 20, Kyev, September 12-16, WTF-CEOP Implementation Plan #1 Status (WTF-CEOP first prototype, by JAXA) September 12, 2005 Osamu Ochiai.
Coordinated Energy and water-cycle Observations Peroject A Well Organized Data Archive System Data Integrating/Archiving Center at University of Tokyo.
NetCDF-4 The Marriage of Two Data Formats Ed Hartnett, Unidata June, 2004.
Module 14: WCF Send Adapters. Overview Lesson 1: Introduction to WCF Send Adapters Lesson 2: Consuming a Web Service Lesson 3: Consuming Services from.
MapServer-OGR-OPeNDAP: An Integrated System for Uniform Access to Land and Oceanographic Datasets Frank Warmerdam Consultant Thomas E. Burk University.
Message Brokers and B2B Application Integration Chap 13 B2B Application Integration Sungchul Hong.
FinCEN Suspicious Activity Report by Depository Institutions
1 © 2008 Avaya Inc. All rights reserved. IPOffice Configuration Service Emil Ratnam.
The IRI Climate Data Library: translating between data cultures Benno Blumenthal International Research Institute for Climate Prediction Columbia University.
Supporting High- Performance Data Processing on Flat-Files Xuan Zhang Gagan Agrawal Ohio State University.
© Blackboard, Inc. All rights reserved. Deploying a complex building block Andre Koehorst Learning Lab Universiteit Maastricht, the Netherlands July 18.
NcBrowse A Graphical netCDF/OPeNDAP Browser Donald Denbo 1 & John Osborne 2 1 UW/JISAO-NOAA/PMEL, 2 OceanAtlas Software
Integrated Model Data Management S.Hankin ESMF July ‘04 Integrated data management in the ESMF (ESME) Steve Hankin (NOAA/PMEL & IOOS/DMAC) ESMF Team meeting.
1 Dapper and Argo Joe Sirott PMEL/NOAA. 2 What is Dapper? Web server that provides distributed access to in-situ data via OPeNDAP protocol Clients include.
Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept
WDC-MARE – World Data Center for Marine Environmental Sciences Data portal based on Open Archives Initiative Protocols and Apache Lucene Uwe Schindler,
Easily Serving and Accessing HDF-EOS2 Datasets Using DODS Technologies Richard Chinman, UCAR-IITA, DODS Project Manager
1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA.
UAF/OSMC Presenters: Kevin O’Brien and Eugene Burger Abstract: Kevin O’Brien and Eugene Burger are from NOAA’s Pacific Marine Environmental Laboratory.
A Data Access Framework for ESMF Model Outputs Roland Schweitzer Steve Hankin Jonathan Callahan Kevin O’Brien Ansley Manke.
EPICS EPICS Limitations Bob Dalesio Marty Kraimer.
OPeNDAP Hyrax Harnessing the power of the BES OPeNDAP Hyrax Back-End Server Patrick West
Hellenic Centre for Marine Research (HCMR) MedOBIS - Ocean Biogeographic Information System for the Eastern Mediterranean and Black Sea.
00/XXXX 1 Data Processing in PRISM Introduction. COCO (CDMS Overloaded for CF Objects) What is it. Why is COCO written in Python. Implementation Data Operations.
1 GFDL Data Portal Current Status, Achievements and Future Development NOAATECH-2006 K.Dixon, V.Balaji, S.Nikonov GFDL, Princeton.
Data Interoperability at the IRI: translating between data cultures Benno Blumenthal International Research Institute for Climate Prediction Columbia University.
Weathertop Consulting, LLC Server-side OPeNDAP Analysis – Concrete steps toward a generalized framework via a reference implementation using F-TDS Roland.
ISO 191** Overview A “Family” of Standards. Resources ISO Standards Web Page – Technical.
LAS and THREDDS: Partners for Education Roland Schweitzer Steve Hankin Jonathan Callahan Joe Mclean Kevin O’Brien Ansley Manke Yonghua Wei.
LCG Distributed Databases Deployment – Kickoff Workshop Dec Database Lookup Service Kuba Zajączkowski Chi-Wei Wang.
Improving User Access to Metadata for Public and Restricted Use US Federal Statistical Files William C. Block Jeremy Williams Lars Vilhuber Carl Lagoze.
Transitioning unique NASA data and research technologies to operations SPoRT AWIPS II Activities Sixth Meeting of the Science Advisory Committee 28 February.
XML Exchange – the future of moving data. Purpose… Moving product data to Retail portal. Moving product data from one WERCS database to another Moving.
CoRD Meeting 12 March 2003 STIPES (Lot 4) STIPES = Statistical Inquiries from Popular European Software.
Ewa Deelman, Virtual Metadata Catalogs: Augmenting Existing Metadata Catalogs with Semantic Representations Yolanda Gil, Varun Ratnakar,
Institute for the Protection and Security of the Citizen HAZAS – Hazard Assessment ECCAIRS Technical Course Provided by the Joint Research Centre - Ispra.
Java OceanAtlas A Cross-Platform Application for Visualization and Selection of Oceanographic Profile Data John R. Osborne ftp://odf.ucsd.edu/pub/OceanAtlas/
Pan-European infrastructure for Ocean & Marine Data management An EU Integrated research Infrastructure Initiative (I3) How to implement CDI ? M. Fichaut,
The Virtual Observatory and Ecological Informatics System (VOEIS): Using RESTful architecture and an extensible data model to provide a unique data management.
February 12, 2002Tom McGlynn ADEC Interoperability Technical Working Group Report.
NcBrowse: A Graphical netCDF File Browser Donald Denbo NOAA-PMEL/UW-JISAO
1. Gridded Data Sub-setting Services through the RDA at NCAR Doug Schuster, Steve Worley, Bob Dattore, Dave Stepaniak.
NcBrowse: OPeNDAP Server Access and 3-D Graphics Presented by Nancy N. Soreide NOAA/PMEL Donald W. Denbo UW/JISAO-NOAA/PMEL.
Shawn McClure, Rodger Ames and Doug Fox - CIRA
Data Browsing/Mining/Metadata
Moving from HDF4 to HDF5/netCDF-4
HDF5 Metadata and Page Buffering
Outline RTQC goals to achieve Description of current proposal Tests
Tom Rink Tom Whittaker Paolo Antonelli Kevin Baggett.
WEL-COME Outlook Technical Support Number Outlook Technical Support Number Call Now : ( Toll Free ) For More Visit on:
Signposting your information - MEDIN Discovery Metadata
CEE 6440 GIS in Water Resources Fall 2004 Term Paper Presentation
Developing a Data Model
Metadata The metadata contains
Live Access Server (LAS)
Metadata use in the Statistical Value Chain
Publishing image services in ArcGIS
Supporting High-Performance Data Processing on Flat-Files
3rd Party Widgets & Custom Code
Presentation transcript:

Configuring NVODS Datasets in LAS Joseph McLean TMAP/PMEL/NOAA and JISAO/UW OAR WebShop 2002

Part 1 -Getting a Dataset "into" LAS- Best Case - COARDS Compliant NetCDF File The metadata ingestion process. The relational database and user interface. Ready to access and process. The LAS Data Configuration XML File operations/institution information data set attributes variable attributes back-end properties lon/lat/depth/time axis definitions

-Data Configuration XML File- ]> <institution name="Pacific Marine Environmental Lab“ url=" &StdOperations; <coads_climatology_cdf name="COADS Climatology" url=" doc="doc/coads_climatology.html"> light_centered COADS_clim_init LAS attributes Data set attributes variable attributes back-end properties Axis definitions

Part 1 -Getting a Dataset "into" LAS- Configuring the XML File when needed custom palettes or contour levels variable name conflicts virtual variables incomplete/incorrect metadata Worst Case - No Metadata frequently missing or inadequate supply metadata in XML file by hand need adequate documentation explore the dataset custom initialization scripts

Part 1 -Getting a Dataset "into" LAS- Back-end Scripts multiple back-ends "NCL","CDAT",GrADS, Matlab,IDL Ferret is free and distributed with LAS. NVODS LAS(via Ferret) uses the NetCDF API to access DODS datasets. Custom User Interfaces high resolution maps data organization

Part 2 -Examples of Dataset Problems- Missing Attributes bad dataflag/missing_value attribute Usually this is straightforward. Dimensional Metadata COARDS Coordinate Variables Fixing this can be time consuming. need documentation. custom scripts Time axes some time units not supported "modulo" attribute Wrong Metadata Freeform file format descriptor Ancillary DAS Files

-DODS.das File with Missing Attributes- Attributes { time { } level { } mean_freq { } NC_GLOBAL { String dtg " “ String model_name "WAM“ String dset_name "fcst_ops“ String geom_name "adriatic“ Float32 southern_latitude 40.; Float32 northern_latitude 46.; Float32 western_longitude 12.; Float32 eastern_longitude 20.; Float32 delta_x ; Float32 delta_y ; } DODS_EXTRA { String Unlimited_Dimension "time" }

Part 3 -NVODS/DODS Dataset Statistics How does Metadata affect Performance? A summary of Tests on a sample of the DODS Dataset List Dataset Ready to Use as accessed - 30% Dataset Usable after Ferret repairs - 35% Unable to use Dataset - 35% Please Note that not all of the problems with opening and using datasets were related with unavailable use metadata. Some of the reasons for not being able to use a Dataset included password needs, bad links, and incompatible file types. With the use of back-end scripts to enhance and correct use metadata, we were able to successfully access 65% of the DODS datasets tested.

- Conclusion - How can the Ancillary Information Service Help? The NVODS LAS is Not Unique Fix the Dataset, Not the Application Allow a User to Determine the Fix