DOC / NOAA / NMFS / SWFSC / ERD

Slides:

Advertisements

Similar presentations

Aggregation and Subsetting in ERDDAP (a middleman data server) Bob Simons NOAA NMFS SWFSC ERD.

Advertisements

Integrating NOAA’s Unified Access Framework in GEOSS: Making Earth Observation data easier to access and use Matt Austin NOAA Technology Planning and Integration.

Summary previous session 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data.

1 Lightning Products and Services at NOAA’s National Climatic Data Center Steve Ansari, Stephen Del Greco, Neal Lott (NOAA / NCDC)

GCI Research Activity Stefano Nativi, Mattia Santoro.

The NODC Glider Technical Specification Tom Ryan, Dan Seidov, John Relph (NODC) and James Bennett (University of Washington) U.S. IOOS National Glider.

Connecting HDF And ISO Metadata Ted Habermann, NASA/ESDIS Hook Hua, Barry Weiss, NASA/Jet Propulsion Lab Mike Folk, Gerd Heber, Elena Pourmal, The HDF.

Transitioning from FGDC CSDGM Metadata to ISO 191** Metadata National Coastal Data Development Center A division of the National Oceanographic Data Center.

National Coastal Data Development Center A division of the National Oceanographic Data Center Please a list of participants at each location to

GO! NC Presented by: NCDOT GIS Unit Overview and Demonstration DiscoverBuildConnect

Presented by Douglas Greer Creating Advanced WEBI Reports.

Steve Rutz NOAA/NESDIS National Oceanographic Data Center NODC Observing Systems Team Leader June 21, 2011.

Unidata TDS Workshop THREDDS Data Server Overview October 2014.

Data integration and dissemination in the MARsite project John Douglas (WP10 leader)

Overview of the ODP Data Provider Sergey Sukhonosov National Oceanographic Data Centre, Russia Expert training on the Ocean Data Portal technology, Buenos.

Bringing it All Together: NODC’s Geoportal Server as an Integration Tool for Interoperable Data Services Kenneth S. Casey, Ph.D. YuanJie Li NOAA National.

SITools Enhanced Use of Laboratory Services and Data Romain Conseil

Publish Calendars to the Web. CCUweb Presentation (10 Minutes) 1 Demonstration of published calendars (10 minutes) 2 Demonstration of importing calendar.

2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

IOOS Data Management Integration Standards Plans in the Northeast Eric Bridger (GMRI) Sep

THREDDS Data Server Ethan Davis GEOSS Climate Workshop 23 September 2011.

UWG 2013 Meeting PO.DAAC Web Services Demo. What are PO.DAAC Web Services?

Data Interoperability Basics Bruce Harold & Dale Lutz.

Glider data (*.*bd files) via Iridium or rsync from WHOI, Umaine or UMaryland Disk NetCDF/Acquire WMO ID IOOS Glider DAC ERDDAP OPeNDAP sequence or download.geoJson,.json.nc,.ncCF,.ncCFMA.csv,.tsv,

Enhancements to a Community Toolset for Ocean Model Data Interoperability: Unstructured grids, NCTOOLBOX, and Distributed Search Rich Signell (USGS), Woods.

Tools in Support of a National DMAC Derrick Snowden NERACOOS/ODP Annual Meeting 26 Sep 2012.

Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.

Architecture Renovation Yoshiyuki Kudo (JAXA) WGISS-37.

IOOS Model Data Interoperability Design ROMS POM WW3 WRF ECOM NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF)

Documentation from NcML to ISO Ted Habermann, NOAA NESDIS NGDC.

NODC ↔ Data Consumers Steve Rutz NOAA/NESDIS National Oceanographic Data Center NODC Observing Systems Team Leader June 21, 2011.

Opendap dev - meeting, Boulder, Feb 2007 OPeNDAP infrastructure in European Operational Oceanography T Loubrieu (IFREMER) T Jolibois (CLS)

IOOS Coastal Ocean Modeling Testbed (COMT) Cyberinfrastructure Oceans 12 Becky Baltes, IOOS Liz Smith, SURA Rich Signell, USGS Eoin Howlett, Kyle Wilcox,

Unidata TDS Workshop THREDDS Data Server Overview

Exploring Spatial Data Infrastructure in an Open Source World Jacqueline Lowe UNC-Asheville National Environmental Modeling and Analysis Center Jacqueline.

Tools for Creating and Editing Metadata. Desktop Applications Web Applications Graphical User Interface (GUI) XML editor Standalone Editor Part of Larger.

IODE Ocean Data Portal - ODP  The objective of the IODE Ocean Data Portal (ODP) is to facilitate and promote the exchange and dissemination of marine.

IOOS Data Services with the THREDDS Data Server Rich Signell USGS, Woods Hole IOOS DMAC Workshop Silver Spring Sep 10, 2013 Rich Signell USGS, Woods Hole.

The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg,

Unidata’s TDS Workshop TDS Overview – Part I July 2011.

User Working Group 2013 Data Access Mechanisms – Status 12 March 2013

UAF/OSMC Presenters: Kevin O’Brien and Eugene Burger Abstract: Kevin O’Brien and Eugene Burger are from NOAA’s Pacific Marine Environmental Laboratory.

Creating Good Documentation NOAA National Geophysical Data Center

Improving Data Catalogs with Free and Open Source Software Kevin O’Brien University of Washington Joint Institute for the Study of the Atmosphere and Ocean.

The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin,

Metadata Training for SEFSC Science Staff Part Two.

Writing Metadata Working Towards Best Practices for SEFSC.

Standard Metadata in Scientific Data Formats September 19, 2007 Flash at:

ERDDAP The Next Generation of Data Servers Bob Simons DOC / NOAA / NMFS / SWFSC / ERD Monterey, CA Disclaimer: The opinions expressed.

Convergence And Trust in Earth and Space Science Data Systems Ted Habermann, NOAA National Geophysical Data Center Documentation: It’s not just discovery...

ORNL DAAC SPATIAL DATA ACCESS TOOL Open Geospatial Consortium (OGC) Services Bruce E. Wilson Suresh K. Santhana Vannan Yaxing Wei Tammy W. Beaty National.

SDI for water resource management Theresia Freska & Wu Liqun.

Rich Signell Roland Viger Curtis Price USGS Community for Data Integration Feb 15, 2012.

1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)

IOOS Compliance Checker Luke Campbell, Software Engineer, RPS ASA.

5-7 May 2003 SCD Exec_Retr 1 Research Data, May Archive Content New Archive Developments Archive Access and Provision.

Update on Unidata Technologies for Data Access Russ Rew

ODP V2 Data Provider overview. 22 Scope Data Provider provides access to data and metadata of the local data systems. Data Provider is a wrapper, installed.

R2R ↔ NODC Steve Rutz NODC Observing Systems Team Leader May 12, 2011 Presented by L. Pikula, IODE OceanTeacher Course Data Management for Information.

2013 GSICS Joint Meeting, Williamsburg VA, USA, March GSICS Collaboration Servers Status: 2013 Peter Miu (EUMETSAT) CMA, CNES, EUMETSAT, ISRO,

Using Community-Driven Metadata Conventions with On-Line Data

Data Browsing/Mining/Metadata

Common Framework for Earth Observation Data

OneStop: Architecture Review

A platform for Linked Data publishing

Intermountain West Data Warehouse

OneStop Metadata Team Lead

Prepared by: Jennifer Saleem Arrigo, Program Manager

Compliance Checker Presented by Luke Campbell (RPS)

Presentation transcript:

DOC / NOAA / NMFS / SWFSC / ERD ERDDAP: One and Done Bob Simons DOC / NOAA / NMFS / SWFSC / ERD Monterey, CA bob.simons@noaa.gov Bob: Fast! One sentence per slide.

NOAA PARR Data Requirements Make a data management plan to... Archive the data (NCEI) Make the data accessible via a web service (NCEI?) Get the metadata into the data.noaa.gov catalog (NCEI) Do this by last week ASAP http://docs.lib.noaa.gov/noaa_documents/NOAA_Research_Council/NOAA_PARR_Plan_v5.04.pdf

NCEI Requirements Document your data (CF and ACDD, ISO 19115) Format your data (Gridded: .nc, Tabular: .ncCFMA) Ensure data integrity (.sha256) Organize your data (.tar.gz) Archive your data at NCEI (Send2NCEI or ATRAC) NCEI will make it accessible - Metadata via the data.noaa.gov and data.gov catalogs. Data via a web service (THREDDS and ?). https://sites.google.com/a/noaa.gov/ncei-ioos-archive/cookbook

NCEI Requirements - By Hand? Document your data (CF and ACDD, ISO 19115 !) Format your data (Gridded: .nc, Tabular: .ncCFMA !) Ensure data integrity (.sha256) Organize your data (.tar.gz) Archive your data at NCEI (Send2NCEI or ATRAC) NCEI will make it accessible - Metadata via the data.noaa.gov and data.gov catalogs. Data via a web service (THREDDS and ?). https://sites.google.com/a/noaa.gov/ncei-ioos-archive/cookbook

ERDDAP: One and Done ERDDAP is: A free, open-source, reusable, data server. Already installed and used at more than 50 organizations around the world. On NOAA's list of recommended data servers for use by groups within NOAA. The only DAP server that supports DAP sequences (for tabular data and in-situ data). https://geo-ide.noaa.gov/wiki/index.php?title=Data_Access_Technical_Recommendations http://coastwatch.pfeg.noaa.gov/erddap/index.html

Overview of ERDDAP

Getting Data Into ERDDAP The new Data Provider Form gathers metadata http://coastwatch.pfeg.noaa.gov/erddap/dataProviderForm.html ERDDAP can access data from many sources - various file types, relational databases, web services In ERDDAP, the dataset has CF, ACDD, and ISO 19115 metadata.  NCEI "Document your data" (CF, ACDD, ISO 19115)  NOAA "Data accessible via a web service" (DAP)  NOAA "Metadata in data.noaa.gov catalog"

Getting Data Out Of ERDDAP, To NCEI Do it by hand? Extract subsets in NCEI-friendly file formats, e.g., .nc and .ncCFMA. Add .sha256 digest files. Then compress it into .tar.gz. The new ArchiveADataset Tool simplifies this. You specify a dataset, a subset, a file type, and a digest type. It generates the .tar.gz file. http://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#ArchiveADataset  NCEI "Format your data" (.nc or .ncCFMA)  NCEI "Ensure data integrity" (.sha256)  NCEI "Organize your data" (.tar.gz)

Requirements Met With ERDDAP  NOAA: Make the data accessible via a web service (your ERDDAP and NCEI's ?)  NOAA: Get metadata into data.noaa.gov catalog  NOAA: Archive the data (NCEI)  NCEI: Document your data (CF and ACDD, ISO 19115)  NCEI: Format your data (Gridded: .nc, Tabular: .ncCFMA)  NCEI: Ensure data integrity (.sha256)  NCEI: Organize your data (.tar.gz)

Additional Benefits You can add datasets to ERDDAP quickly. ERDDAP follows Common Framework standards: DAP, WMS, SOS, WFS, OpenSearch, CF, ISO 19115, NetCDF https://www.whitehouse.gov/sites/default/files/microsites/ostp/common_framework_for_earth_observation_data_draft_120215.pdf Users can download subsets in different file types. Users can make custom graphs and maps. Other ERDDAP's can link to your datasets and always have the latest data (without transferring data). Data remains under data provider's control. NCEI? NOAA?

Please Give ERDDAP a try. http://coastwatch. pfeg. noaa Please Give ERDDAP a try! http://coastwatch.pfeg.noaa.gov/erddap/ bob.simons@noaa.gov Thank you!