Unidata’s TDS Workshop TDS Overview – Part I 21-22 July 2011.

Slides:



Advertisements
Similar presentations
1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.
Advertisements

TDS Archictecture Dec HTTP Tomcat Server TDS is a data server Datasets motherlode.ucar.edu THREDDS Server NetCDF-Java library Remote Access IDD.
Reading HDF family of formats via NetCDF-Java / CDM
Recent Work in Progress
The Model Output Interoperability Experiment in the Gulf of Maine: A Success Story Made Possible By CF, NcML, NetCDF-Java and THREDDS Rich Signell (USGS,
Integrating NOAA’s Unified Access Framework in GEOSS: Making Earth Observation data easier to access and use Matt Austin NOAA Technology Planning and Integration.
A Common Data Model In the Middle Tier Enabling Data Access in Workflows … HDF/HDF-EOS Workshop XIV September 29, 2010 Doug Lindholm Laboratory for Atmospheric.
Streaming NetCDF John Caron July What does NetCDF do for you? Data Storage: machine-, OS-, compiler-independent Standard API (Application Programming.
® OGC Web Services Initiative, Phase 9 (OWS-9): Innovations Thread - OPeNDAP James Gallagher and Nathan Potter, OPeNDAP © 2012 Open Geospatial Consortium.
THREDDS, CDM, OPeNDAP, netCDF and Related Conventions John Caron Unidata/UCAR Sep 2007.
The Future of NetCDF Russ Rew UCAR Unidata Program Center Acknowledgments: John Caron, Ed Hartnett, NASA’s Earth Science Technology Office, National Science.
Web based tools Ideas for presentation of operational meteorological data Ernst de Vreede KNMI EGOWS /6/2009 Ideas for presentation of operational.
A New Generation of Data Services for Earth System Science Education and Research: Unidata’s Plans and Directions AGU Fall Meeting San Francisco, CA 6.
Data Discovery 1.Data discovery systems 2.TDS and metadata 3.NcISO services.
Unidata TDS Workshop THREDDS Data Server Overview October 2014.
Introduction Downloading and sifting through large volumes of data stored in differing formats can be a time-consuming and sometimes frustrating process.
1 The NOAA Weather and Climate Toolkit Steve Ansari, Stephen Del Greco (NOAA / NCDC) Mark Phillips (UNC-Asheville / NEMAC) Bill Hankins (STG Inc.)
John Caron Unidata October 2012
1 The NOAA Weather and Climate Toolkit Steve Ansari, Stephen Del Greco, Neal Lott (NOAA / NCDC)
OPeNDAP and the Data Access Protocol (DAP) Original version by Dave Fulker.
Quick Unidata Overview NetCDF Workshop 25 October 2012 Russ Rew.
Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned Rich Signell USGS Woods Hole, MA / NOAA Silver Spring USA Model Data.
Unidata’s TDS Workshop TDS Overview – Part II October 2012.
2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.
Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.
Unidata’s Common Data Model John Caron Unidata/UCAR Nov 2006.
THREDDS Data Server Ethan Davis GEOSS Climate Workshop 23 September 2011.
Weathertop Consulting, LLC Wednesday, January 14, 2009 IIPS 11A.2 1 A General Purpose System for Server-side Analysis of Earth Science Data Roland Schweitzer.
NetCDF-Java Overview John Caron Oct 29, Contents Data Models / Shared Dimensions Coordinate Systems Feature Types NetCDF Markup Language (NcML)
NcML Aggregation vs Feature Collections. NcML functionality 1.Modify the objects found in CDM files – Especially Attributes – Don’t have to rewrite the.
Unidata’s TDS Workshop TDS Overview – Part II Unidata July 2011.
N-Wave Stakeholder Users Conference Wednesday, May 11, Marine St, Rm 123 Boulder, CO Linda Miller and Mike Schmidt Unidata Program Center (UPC)-Boulder,
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
Tools in Support of a National DMAC Derrick Snowden NERACOOS/ODP Annual Meeting 26 Sep 2012.
Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.
IOOS Model Data Interoperability Design ROMS POM WW3 WRF ECOM NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF)
THREDDS Data Server Unidata’s Common Data Model Background / Summary John Caron Unidata/UCAR Mar 2007.
1 International Standards for Data Interoperability GALEON Geo-interface for Air, Environment, Land, Ocean NetCDF Ben Domenico Unidata Program Center*
DAP4 James Gallagher & Ethan Davis OPeNDAP and Unidata.
Opendap dev - meeting, Boulder, Feb 2007 OPeNDAP infrastructure in European Operational Oceanography T Loubrieu (IFREMER) T Jolibois (CLS)
IOOS Modeling Testbed Cyberinfrastructure Rich Signell, USGS, Woods Hole, MA IOOS-RA-Briefing, Feb 14, 2012.
Unidata TDS Workshop THREDDS Data Server Overview
1 NASA CEOP Status & Demo CEOS WGISS-24 Oberpfaffenhofen, Germany October 15, 2007 Yonsook Enloe.
Recent developments with the THREDDS Data Server (TDS) and related Tools: covering TDS, NCML, WCS, forecast aggregation and not including stuff covered.
Unidata’s Common Data Model and the THREDDS Data Server John Caron Unidata/UCAR, Boulder CO Jan 6, 2006 ESIP Winter 2006.
IOOS Data Services with the THREDDS Data Server Rich Signell USGS, Woods Hole IOOS DMAC Workshop Silver Spring Sep 10, 2013 Rich Signell USGS, Woods Hole.
THREDDS Catalogs Ethan Davis UCAR/Unidata NASA ESDSWG Standards Process Group meeting, 17 July 2007.
DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan.
1 NASA CEOP Final Summary CEOS WGISS-26 Boulder, Colorado September 23, 2008 Yonsook Enloe
The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin,
Unidata’s Common Data Model and NetCDF Java Library API Overview John Caron Unidata/UCAR Nov 2008.
Unidata's Involvement in Developing and Supporting Climate Science Infrastructure Russ Rew UCAR Unidata April 2010.
GIS for Atmospheric Sciences and Hydrology By David R. Maidment University of Texas at Austin National Center for Atmospheric Research, 6 July 2005.
Weathertop Consulting, LLC Server-side OPeNDAP Analysis – Concrete steps toward a generalized framework via a reference implementation using F-TDS Roland.
LAS and THREDDS: Partners for Education Roland Schweitzer Steve Hankin Jonathan Callahan Joe Mclean Kevin O’Brien Ansley Manke Yonghua Wei.
Data Stewardship at the NOAA Data Centers Sub Topic - Value Added Products ESIP Federation Meeting, Washington, DC January 6-8, 2009.
ORNL DAAC SPATIAL DATA ACCESS TOOL Open Geospatial Consortium (OGC) Services Bruce E. Wilson Suresh K. Santhana Vannan Yaxing Wei Tammy W. Beaty National.
1 Key Results from GALEON John Caron Ben Dominico UCAR/Unidata.
Unidata Technologies Relevant to GO-ESSP: An Update Russ Rew
Rich Signell Roland Viger Curtis Price USGS Community for Data Integration Feb 15, 2012.
1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)
NetCDF: Data Model, Programming Interfaces, Conventions and Format Adapted from Presentations by Russ Rew Unidata Program Center University Corporation.
Interoperability Day Introduction Standards-based Web Services Interfaces to Existing Atmospheric/Oceanographic Data Systems Ben Domenico Unidata Program.
Update on Unidata Technologies for Data Access Russ Rew
THREDDS Data Server (TDS) and Data Discovery John Caron Unidata/UCAR May 15, 2006.
Unidata Infrastructure for Data Services Russ Rew GO-ESSP Workshop, LLNL
NetCDF-Java version 2.2 Common Data Model John Caron Unidata/UCAR Dec 10, 2004.
Data Browsing/Mining/Metadata
Remote Data Access Update
What is NetCDF ? And what are its plans for world domination?
Presentation transcript:

Unidata’s TDS Workshop TDS Overview – Part I July 2011

THREDDS Data Server (TDS) Web server for scientific data (written in 100% Java) Can serve any dataset the netCDF-Java library can read – E.g., netCDF-3, netCDF-4, HDF-4, HDF-5, HDF-EOS, GRIB-1, GRIB-2 Catalogs advertise available datasets and services Data access (subset) services: – OPeNDAP – OGC WMS and WCS – NCSS Data collection services – Aggregation – Point/station collection Metadata services – THREDDS – ncISO: ISO, UDDC, NcML

Servlet Container THREDDS Data Server Datasets catalog.xml motherlode.ucar.edu THREDDS Server NetCDF-Java library Remote Access Client IDD Data HTTPServer WMS WCS OPeNDAP configCatalog.xml

Common Data Model (CDM) Implemented in the netCDF-Java library (3 layers) NetCDF Data Access: – Access to array-based scientific data (netCDF data model) – Read various file formats, map into data model (IOSP) NetcdfDataset: – Geospatial coordinate systems (conventions) – NcML: Modify existing dataset – Aggregation of datasets Scientific Feature Types, e.g.: – Gridded Data; Radial Data; Swath Data – Discrete Sampling Features: Point Data; Station Data; Profile Data Trajectory (e.g., aircraft track) Data

NetcdfDataset Application Scientific Feature Types NetCDF-Java/ CDM architecture OPeNDAP THREDDS Catalog.xml NetCDF-3 HDF5 I/O service provider GRIB GINI NIDS NetcdfFile NetCDF-4 … Nexrad DMSP CoordSystem Builder Datatype Adapter NcML

Servlet Container THREDDS Data Server Datasets catalog.xml motherlode.ucar.edu THREDDS Server NetCDF-Java library Remote Access Client IDD Data HTTPServer WMS WCS OPeNDAP configCatalog.xml

THREDDS Catalogs Catalogs collect, organize, and describe datasets – Hierarchical collections of datasets (virtual directories) – Each dataset may have multiple access methods – Documentation (metadata) may be added To collections of datasets To individual datasets (inheritance optional) Servers can use catalogs to advertise the datasets they serve

THREDDS Catalogs <catalog xmlns=“ xmlns:xlink=" version="1.0.2"> odap <catalogRef xlink:title="My Other Catalog“ xlink:href="myOtherCatalog.xml" /> <catalogRef xlink:title="Far Away Univ catalog“ xlink:href=" />

THREDDS Catalogs

Servlet Container THREDDS Data Server Datasets catalog.xml motherlode.ucar.edu THREDDS Server NetCDF-Java library Remote Access Client IDD Data HTTPServer WMS WCS OPeNDAP configCatalog.xml

Data Access Services Bulk File Transfer – HTTP Server (any file) Remote access, subsetting CDM files – OPeNDAP (any* CDM file) – Web Coverage Service (grids) – Web Map Service (grids) – Others NetCDF Subset Service (grids) Radar Collection Subset Service

Data Access Services Clients OPeNDAP (lots, including IDL, MatLAB)lots WCS (still minimal, but expect to grow) – GDAL, CADCorp, George Mason, OWSLib WMS (speak WMS; consume GIF, JPEG, and PNG) – Browsers, Portals, ARC GIS, Google Earth, etcetc NetCDF Subsetter – HTML forms interface – Web service (CUAHSI) – ESRI ArcGIS (manual download of netcdf-CF)

Servlet Container THREDDS Data Server Datasets catalog.xml motherlode.ucar.edu THREDDS Server NetCDF-Java library Remote Access Client IDD Data HTTPServer WMS WCS OPeNDAP configCatalog.xml

Configuration Catalogs Administrator tells the TDS which / how datasets are to be served Identify the files on disk Name, organize, sort, filter into virtual collections Add discovery metadata Integrated with NcML – Create virtual datasets

TDS Configuration Catalogs … <datasetScan name="Ocean Satellite Data“ path="data/ocean/sat" location= "/data/tds/netcdf/"> odap Grid NetCDF <documentation xlink:href=" xlink:title="NCEP Model documentation" /> Transmitted through Unidata Internet Data Distribution. …

TDS and NcML NcML: an XML representation of netCDF. – Can “fix” metadata problems – Can augment metadata – Can aggregate datasets General Aggregations: joinNew, joinExisting, Union Specialized aggregations: – Forecast Model Run Collection (FMRC) – Point Feature Collections Embed NcML into the TDS configuration catalog Server serves a virtual dataset defined by NcML – NcML hidden from the client

TDS / NcML Modify and serve through TDS

Servlet Container THREDDS Data Server Datasets catalog.xml motherlode.ucar.edu THREDDS Server NetCDF-Java library Remote Access Client IDD Data HTTPServer WMS WCS OPeNDAP configCatalog.xml

THREDDS Data Server Getting Started TDS is written in 100% Java – Installing Java 1.6 TDS uses the Java Servlet framework – Installing Tomcat 6 Deploying the TDS …