Improving Data Catalogs with Free and Open Source Software Kevin O’Brien University of Washington Joint Institute for the Study of the Atmosphere and Ocean.

Slides:

Advertisements

Similar presentations

1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.

Advertisements

The Live Access Server (Access to observational data) Jonathan Callahan (University of Washington) Steve Hankin (NOAA/PMEL – PI) Roland Schweitzer, Kevin.

The Model Output Interoperability Experiment in the Gulf of Maine: A Success Story Made Possible By CF, NcML, NetCDF-Java and THREDDS Rich Signell (USGS,

Climate Analytics on Global Data Archives Aparna Radhakrishnan 1, Venkatramani Balaji 2 1 DRC/NOAA-GFDL, 2 Princeton University/NOAA-GFDL 2. Use-case 3.

Integrating NOAA’s Unified Access Framework in GEOSS: Making Earth Observation data easier to access and use Matt Austin NOAA Technology Planning and Integration.

The Unified Access Framework (UAF) Philosophy, progress, and plans DAARWG Meeting, Seattle, Nov Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO),

Kevin O’Brien University of Washington/JISAO NOAA/PMEL Interoperable Access to Near Real Time Ocean Observations with the Observing System Monitoring Center.

Summary previous session 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data.

1 Lightning Products and Services at NOAA’s National Climatic Data Center Steve Ansari, Stephen Del Greco, Neal Lott (NOAA / NCDC)

Ocean data dissemination Jon Blower, University of Reading, UK Steve Hankin, Bob Keeley, Sylvie Pouliquen, Jeff de la Beaujardière, Edward Vanden Berghe,

The GEON Integrated Data Viewer (IDV) and IRIS DMC Services: CyberInfrastructure Support for Seismic Data Visualization and Interpretation Charles Meertens.

TPAC Digital Library Talk Overview Presenter:Glenn Hyland Tasmanian Partnership for Advanced Computing & Australian Antarctic Division Outline: TPAC Overview.

Steve Rutz NOAA/NESDIS National Oceanographic Data Center NODC Observing Systems Team Leader June 21, 2011.

Unidata TDS Workshop THREDDS Data Server Overview October 2014.

HYCOM Data Service New Datasets, Functionality and Future Development Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL) Major contributors: Jon Callahan.

ToolMatch: Discovering What Tools can be used to Access, Manipulate, Transform, and Visualize Data Patrick West 1 Nancy Hoebelheinrich.

ISO-2 and more NGDC Geoportal January 4 th, 2012.

VrRBO with THREDDS data store. Paths & URLs THREDDS server THREDDS data directory.

Metadata Guides for Smarties Marine Metadata Initiative URL:

Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned Rich Signell USGS Woods Hole, MA / NOAA Silver Spring USA Model Data.

Observing System Monitoring Center Integrating data and information across observing system networks.

Bringing it All Together: NODC’s Geoportal Server as an Integration Tool for Interoperable Data Services Kenneth S. Casey, Ph.D. YuanJie Li NOAA National.

DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.

2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

THREDDS Data Server Ethan Davis GEOSS Climate Workshop 23 September 2011.

Weathertop Consulting, LLC Wednesday, January 14, 2009 IIPS 11A.2 1 A General Purpose System for Server-side Analysis of Earth Science Data Roland Schweitzer.

U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center.

Contrasting styles of Web UI Development: GWT vs Native JavaScript Roland Schweitzer Weathertop Consulting, LLC Jeremy Malczyk JISAO.

Enhancements to a Community Toolset for Ocean Model Data Interoperability: Unstructured grids, NCTOOLBOX, and Distributed Search Rich Signell (USGS), Woods.

Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.

IOOS Model Data Interoperability Design ROMS POM WW3 WRF ECOM NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF)

Integrated Model Data Management S.Hankin ESMF July ‘04 Integrated data management in the ESMF (ESME) Steve Hankin (NOAA/PMEL & IOOS/DMAC) ESMF Team meeting.

Opendap dev - meeting, Boulder, Feb 2007 OPeNDAP infrastructure in European Operational Oceanography T Loubrieu (IFREMER) T Jolibois (CLS)

IOOS Modeling Testbed Cyberinfrastructure Rich Signell, USGS, Woods Hole, MA IOOS-RA-Briefing, Feb 14, 2012.

1 HYCOM Data Service HYCOM Data Service An overview Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL)

IOOS Coastal Ocean Modeling Testbed (COMT) Cyberinfrastructure Oceans 12 Becky Baltes, IOOS Liz Smith, SURA Rich Signell, USGS Eoin Howlett, Kyle Wilcox,

Unidata TDS Workshop THREDDS Data Server Overview

1 NASA CEOP Status & Demo CEOS WGISS-24 Oberpfaffenhofen, Germany October 15, 2007 Yonsook Enloe.

M.Benno Blumenthal and John del Corral International Research Institute for Climate and Society OpenDAP 2007

U.S. Integrated Ocean Observing System (IOOS ® ) IOOS ® Biological Observations Data Project A Multi-Agency Effort to Enable Access to Biological Observations.

IOOS Data Services with the THREDDS Data Server Rich Signell USGS, Woods Hole IOOS DMAC Workshop Silver Spring Sep 10, 2013 Rich Signell USGS, Woods Hole.

THREDDS Catalogs Ethan Davis UCAR/Unidata NASA ESDSWG Standards Process Group meeting, 17 July 2007.

Unidata’s TDS Workshop TDS Overview – Part I July 2011.

Kevin O’Brien University of Washington/JISAO NOAA/PMEL The Observing System Monitoring Center Steve Hankin, PMEL Ted Habermann, NGDC David Neufeld, NGDC.

UAF/OSMC Presenters: Kevin O’Brien and Eugene Burger Abstract: Kevin O’Brien and Eugene Burger are from NOAA’s Pacific Marine Environmental Laboratory.

A Data Access Framework for ESMF Model Outputs Roland Schweitzer Steve Hankin Jonathan Callahan Kevin O’Brien Ansley Manke.

Documenting UAF Data Ted Habermann NOAA/NESDIS/National Geophysical Data Center.

The Unified Access Framework (UAF) Building NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) one step at a time Steve Hankin (PMEL),

1-2-3 February 2006 –Page 1 Mersea Integrated System How to improve Access/Downloading services ? How far do we go in terms of standardization ?

The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin,

Ed Armstrong – PI Luca Cinquini Chris Mattmann NASA Jet Propulsion Laboratory Frank O’Brien Zach Siegrist System Science Applications, Inc. 18 July 2012.

A Climate Data Portal Focused on realtime and retrospective in situ data Nancy Soreide, Don Denbo, Willa Zhu, NOAA/PMEL Charles Sun, NOAA/NODC Bernie Kilonsky,

Kevin Gomes, MBARI MBARI Data Architecture OOI Cyberinfrastructure: Data Product Generation Workshop San Diego May 20-21, 2008.

Weathertop Consulting, LLC Server-side OPeNDAP Analysis – Concrete steps toward a generalized framework via a reference implementation using F-TDS Roland.

LAS and THREDDS: Partners for Education Roland Schweitzer Steve Hankin Jonathan Callahan Joe Mclean Kevin O’Brien Ansley Manke Yonghua Wei.

The Research Data Archive at NCAR: A System Designed to Handle Diverse Datasets Bob Dattore and Steven Worley National Center for Atmospheric Research.

DOC / NOAA / NMFS / SWFSC / ERD

UAF-grid project status Steve Hankin 12 Jan., 2010.

Rich Signell Roland Viger Curtis Price USGS Community for Data Integration Feb 15, 2012.

The Unified Access Framework (UAF)The Unified Access Framework (UAF) A Global Earth Observation Integrated Data Environment (GEO-IDE) project to integrate.

1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)

In Situ Data Access Some reasons for success or failure Nancy N. Soreide, Donald W. Denbo NOAA Pacific Marine Environmental Laboratory IIPS Session 3B.

NQuery: A Network-enabled Data-based Query Tool for Multi-disciplinary Earth-science Datasets John R. Osborne 1, Kevin T. McHugh 2, and Donald W. Denbo.

Data Browsing/Mining/Metadata

Global Precipitation Data Access, Value-added Services and Scientific Exploration Tools at NASA GES DISC Zhong Liu1,4, D. Ostrenga1,2, G. Leptoukh4, S.

MERRA Data Access and Services

Integrating Data and Information Across Observing System

Live Access Server (LAS)

ExPLORE Complex Oceanographic Data

Presentation transcript:

Improving Data Catalogs with Free and Open Source Software Kevin O’Brien University of Washington Joint Institute for the Study of the Atmosphere and Ocean Steven C Hankin – NOAA/PMEL Roland Schweitzer – Weathertop Consulting AGU Fall Meeting 2013

The Unified Access Framework (UAF) A Global Earth Observation Integrated Data Environment (GEO-IDE) project An attempt to improve scientific data management and access Focus on successes

Lots of data already available

What “success” did UAF chose to copy? Year 1 focused on gridded datasets. Service stack: netCDF-CF-DAP-THREDDS-WMS Projects: (too many to name) Data formats: netCDFGRIBHDF Applications: MatlabArcGISFerret GrADS Google Earth IDV LAS ERDDAP … Users: (too many to name) …

Developing the UAF Catalog Cleaner (a ‘web crawler’) ‘RAW’ ‘CLEAN’

Tree Crawl Dataset Crawl Cleaner CatalogRef and Dataset URL’s Raw catalog XML

Tree Crawl Dataset Crawl Cleaner url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" url=" CatalogRef and Dataset URL’s

Tree Crawl Dataset Crawl Cleaner UAF Clean Catalog

How to provide feedback to data providers? Remember the “Building on Success” theme ncISO metadata assessment tool is very successful

How about a catalog quality assessment tool? How to provide feedback to data providers? Remember the “Building on Success” theme ncISO metadata assessment tool is very successful

Statistics for current catalog and all it’s children Links to rubric reports for child catalogs

Missing services Data issues

url

Data issues Original Catalog

Moving Forward…. Welcome feedback on rubric and Catalog Cleaner tool Change wording in rubric UAF master catalog to go beyond gridded files Use ERDDAP to including In Situ featureTypes Continue community outreach to improve catalogs

Thank you! UAF: geo-ide.noaa.gov Catalog Cleaner code and documentation: THREDDS: netCDF: OPeNDAP: CF: cf-pcmdi.llnl.gov AGU Fall Meeting 2013