MATLAB, netCDF, and OPeNDAP

Slides:



Advertisements
Similar presentations
Reading HDF family of formats via NetCDF-Java / CDM
Advertisements

The HDF Group Support for NPP/NPOESS by The HDF Group Mike Folk, Elena Pourmal The HDF Group HDF/HDF-EOS Workshop XIV September 30, 2010.
Recent Work in Progress
ROMS/TOMS Web Sites & ROMS: NetCDF and Matlab David Robertson and Hernan Arango IMCS, Rutgers University New Brunswick, NJ, USA.
A Common Data Model In the Middle Tier Enabling Data Access in Workflows … HDF/HDF-EOS Workshop XIV September 29, 2010 Doug Lindholm Laboratory for Atmospheric.
Streaming NetCDF John Caron July What does NetCDF do for you? Data Storage: machine-, OS-, compiler-independent Standard API (Application Programming.
The Future of NetCDF Russ Rew UCAR Unidata Program Center Acknowledgments: John Caron, Ed Hartnett, NASA’s Earth Science Technology Office, National Science.
Активное распределенное хранилище для многомерных массивов Дмитрий Медведев ИКИ РАН.
NetCDF Ed Hartnett Unidata/UCAR
© 2008The MathWorks, Inc. ® ® The MATLAB Low-Level HDF5 Interface John Evans.
1 NetCDF and Self-Describing Data Kate Hedstrom January 2010
OPeNDAP and the Data Access Protocol (DAP) Original version by Dave Fulker.
Netcdf course Intro CF convention Netcdf excercises.
NetCDF-4 The Marriage of Two Data Formats Ed Hartnett, Unidata June, 2004.
MapServer-OGR-OPeNDAP: An Integrated System for Uniform Access to Land and Oceanographic Datasets Frank Warmerdam Consultant Thomas E. Burk University.
NetCDF for Developers and Data Providers Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 14 April 2011.
OOI CyberInfrastructure: Technology Overview - Hyrax January 2009 Claudiu Farcas OOI CI Architecture & Design Team UCSD/Calit2.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
Unidata’s Common Data Model John Caron Unidata/UCAR Nov 2006.
Designing and Developing WS B. Ramamurthy. Plans We will examine the resources available for development of JAX-WS based web services. We need an IDE,
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept
DAP4 James Gallagher & Ethan Davis OPeNDAP and Unidata.
Accessing Remote Datasets using the DAP protocol through the netCDF interface. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
McIDAS achievements Compiled & Installed McIDAS-X v2008 Tested BOWTIE correction Enhanced MOD4 and MOD6 server code to reflect changes in v005 algorithms.
Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
Documentation javadoc. Documentation not a programmer's first love lives in a separate file somewhere usually a deliverable on the schedule often not.
1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA.
NetCDF file generated from ASDC CERES SSF Subsetter ATMOSPHERIC SCIENCE DATA CENTER Conversion of Archived HDF Satellite Level 2 Swath Data Products to.
Remote Data Access with OPeNDAP Dr. Dennis Heimbigner Unidata netCDF Workshop October 25, 2012.
The HDF Group Data Interoperability The HDF Group Staff Sep , 2010HDF/HDF-EOS Workshop XIV1.
The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.
OPeNDAP Hyrax Harnessing the power of the BES OPeNDAP Hyrax Back-End Server Patrick West
NetCDF-4: Software Implementing an Enhanced Data Model for the Geosciences Russ Rew, Ed Hartnett, and John Caron UCAR Unidata Program, Boulder
NetCDF and Scientific Data Durability Russ Rew, UCAR Unidata ESIP Federation Summer Meeting
Data File Formats: netCDF by Tom Whittaker University of Wisconsin-Madison SSEC/CIMSS 2009 MUG Meeting June, 2009.
MATLAB's HDF5 Updates John Evans Image and Scientific Data Formats.
CF 2.0 Coming Soon? (Climate and Forecast Conventions for netCDF) Ethan Davis ESO Developing Standards - ESIP Summer Mtg 14 July 2015.
Rich Signell Roland Viger Curtis Price USGS Community for Data Integration Feb 15, 2012.
Update on Unidata Technologies for Data Access Russ Rew
1 PSI/PhUSE Single Day Event – SAS Applications – June 11, 2009 SAS Drug Development from the Inside Magnus Mengelbier Director.
NcBrowse: A Graphical netCDF File Browser Donald Denbo NOAA-PMEL/UW-JISAO
Utilities for netCDF-4 Dr. Dennis Heimbigner Unidata Advanced netCDF Workshop July 25, 2011.
TSDS (HPDE DAP). Objectives (1) develop a standard API for time series-like data, (2) develop a software package, TSDS (Time Series Data Server), that.
Unidata Infrastructure for Data Services Russ Rew GO-ESSP Workshop, LLNL
NetCDF Data Model Details Russ Rew, UCAR Unidata NetCDF 2009 Workshop
MATLAB, Big Data, and HDF Server
Adding CF Attributes to an HDF5 File
DAP+NETCDF Using the netCDF-4 Data Model
(Network Common Data Form)
Data Analytics using MATLAB and HDF5
Moving from HDF4 to HDF5/netCDF-4
Intro to CMIP, the WHOI CMIP5 community server, and planning for CMIP6
NetCDF 3.6: What’s New Russ Rew
Plans for an Enhanced NetCDF-4 Interface to HDF5 Data
Efficiently serving HDF5 via OPeNDAP
Extending the NetCDF Supported Data Formats using a Dispatch Layer
Tom Rink Tom Whittaker Paolo Antonelli Kevin Baggett.
Unidata Advanced netCDF Workshop
McIDAS-V: Why it’s Based on VisAD and IDV
NetCDF and Scientific Data Standard
Remote Data Access Update
ROMS/TOMS Web Sites & ROMS: NetCDF and Matlab
Accessing Remote Datasets through the netCDF interface.
Dispatch Layer and the NetCDF Architecture
HDF-EOS Workshop XXI / The 2018 ESIP Summer Meeting
NCL variable based on a netCDF variable model
Future Development Plans
OPeNDAP/Hyrax Interfaces
Presentation transcript:

MATLAB, netCDF, and OPeNDAP John Evans john.evans@mathworks.com

New In R2012a: NetCDF Library version 4.1.3 and OPeNDAP >> url = ['http://eosdap.hdfgroup.uiuc.edu:8080/opendap/data/NASAFILES' … '/hdf4/AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf']; >> ncdisp(url, 'TopographyU274'); >> help ncread; >> data = ncread(url,'TopographyU274’);'); >> lon = ncread(url,'LongitudeU272'); >> lat = ncread(url,‘LatitudeU271'); >> data(data==0) = NaN; >> pcolor(lon,lat,data’); >> shading flat; colorbar

Informal Interface (high level) High level interfaces aimed at convenience, basic command line work. ncdisp - Display contents of NetCDF file in command window. ncread - Read data from a variable in a NetCDF file. ncreadatt - Read an attribute value from a NetCDF file. ncwrite - Write data to a NetCDF file. ncwriteatt - Write an attribute to a NetCDF file. ncinfo - Return information about a NetCDF file. nccreate - Create a variable in a NetCDF file. ncwriteschema - Add NetCDF schema definitions to a NetCDF file.

Formal Interface (low-level) Low-level follow the library API, aimed at developers. >> help netcdf >> help netcdf.open High level interface built on top of low-level interface. Low level interface sits on top of netCDF library.

OPeNDAP with either interface NetCDF library compiled with OPeNDAP support Any customer code using either netCDF interface is now OPeNDAP-enabled by default. >> ncid = netcdf.open(url); >> [numdims, numvars] = netcdf.inq(ncid); >> info = ncinfo(url)

OPeNDAP in earlier versions of MATLAB… Possible to do with Unidata’s netcdf-java. Illustrates how to integrate MATLAB with 3rd party jar files. >> javaaddpath('netcdfAll-4.2.jar'); >> import ucar.nc2.dods.* >> jncid = NetcdfFile.open(url); Possibly soon in Debian? Otherwise get from Unidata or go get SNCTOOLS from http://mexcdf.sourceforge.net. Other OPeNDAP implementations, i.e. “LOADDAP”

NetCDF Support netCDF-3 and netCDF-4 classic model  netCDF-4 groups, unsigned integer datatypes can read some HDF5 files >> h5create(‘myfile.h5’,’/mydataset’,[100 200]); >> h5disp(‘myfile.h5’); >> ncdisp(‘myfile.h5’);

NetCDF Support (h5disp output) HDF5 myfile.h5 Group '/' Dataset 'myDataset1' Size: 100x200 MaxSize: 100x200 Datatype: H5T_IEEE_F64LE (double) ChunkSize: [] Filters: none FillValue: 0.000000

NetCDF Support (ncdisp output) Source: myfile.h5 Format: netcdf4 Dimensions: phony_dim_0 = 200 phony_dim_1 = 100 Variables: myDataset1 Size: 100x200 Dimensions: phony_dim_1,phony_dim_0 Datatype: double

NPP Access Example via HDF5 High Level >> h5file = [‘VISTO_npp_d20100906_t2105098_’, … ‘e2106344_b00012_c20110707163027621025_noaa_ops.h5’] >> rdataset = ‘/Data_Products/VIIRS-IST-EDR/VIIRS-IST-EDR_Gran_0’ >> h5disp(h5file,rdataset); HDF5 VISTO_npp_d20100906_t2105098_e2106344_ … noaa_ops.h5 Dataset 'VIIRS-IST-EDR_Gran_0' Size: 5 MaxSize: Inf Datatype: H5T_REFERENCE ChunkSize: 5 Filters: none FillValue: H5T_REFERENCE Attributes: 'Ascending/Descending_Indicator': 0 'Beginning_Date': '20100906' 'Beginning_Time': '210509.899480Z' 'East_Bounding_Coordinate': -97.946198 .

NPP Access Example (continued) >> dreferenced_data = h5read(h5file,rdataset); dereferenced_data = [2457600x1 uint16] [2457600x1 uint8 ] [ 2x1 single]

References http://www.mathworks.com http://www.unidata.ucar.edu/software/netcdf-java http://www.opendap.org http://mexcdf.sourceforge.net (SNCTOOLS) http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/api18-m.html (low level HDF5 API examples)