Introduction to NetCDF Ernesto Munoz. Outline Overview of NetCDF Overview of NetCDF NetCDF file information NetCDF file information CDL utilities: ncdump,

Slides:



Advertisements
Similar presentations
Data Formats: Using self-describing data formats Curt Tilmes NASA Version 1.0 Review Date.
Advertisements

WOCE Global Data V3 WOCE-DPC Report Nathan Bindoff and David M. Legler Co-Chairs, WOCE DPC WOCE Conference November 2002 All of it.
The NCAR Command Language (NCL) and the NetCDF Data Format Research Tools Presentation Matthew Janiga 10/30/2012.
Preparing CMOR for CMIP6 and other WCRP Projects
Streaming NetCDF John Caron July What does NetCDF do for you? Data Storage: machine-, OS-, compiler-independent Standard API (Application Programming.
Reading NetCDF Files in Matlab and analyzing the data.
The Future of NetCDF Russ Rew UCAR Unidata Program Center Acknowledgments: John Caron, Ed Hartnett, NASA’s Earth Science Technology Office, National Science.
NetCDF An Effective Way to Store and Retrieve Scientific Datasets Jianwei Li 02/11/2002.
NetCDF 3.6: What’s New Russ Rew Unidata Program Center University Corporation for Atmospheric Research
Активное распределенное хранилище для многомерных массивов Дмитрий Медведев ИКИ РАН.
Marine GIS Applications using ArcGIS Global Classroom training course Marine GIS Applications using ArcGIS Global Classroom training course By T.Hemasundar.
NetCDF Ed Hartnett Unidata/UCAR
Introduction to NetCDF Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 13 April 2011.
1 CF Unleashed: Introduction to Cf/Radial Joe VanAndel National Center for Atmospheric Research 2013/1/8 The National Center for Atmospheric.
Status of netCDF-3, netCDF-4, and CF Conventions Russ Rew Community Standards for Unstructured Grids Workshop, Boulder
Session 2: Using OPeNDAP-enabled Applications to Access Australian Data Services and Repositories eResearch Australasia 2011, ½ Day Morning Workshop, Thursday.
1 NetCDF and Self-Describing Data Kate Hedstrom January 2010
OPeNDAP and the Data Access Protocol (DAP) Original version by Dave Fulker.
ElVis Developments for Simulation and Analysis Programs Tarun Pondicherry Summer 2006 Science Ed High School Intern Eliot Feibush, Mentor 8/16/2006.
Data Formats: Using Self-describing Data Formats Curt Tilmes NASA Version 1.0 February 2013 Section: Local Data Management Copyright 2013 Curt Tilmes.
Netcdf course Intro CF convention Netcdf excercises.
AON Data Questionnaire Results 21 Respondents Last Updated 27 March 2007 First AON PI Meeting Scot Loehrer, Jim Moore.
February 1 & 31 Csci 2111: Data and File Structures Week4, Lectures 1 & 2 Fundamental File Structure Concepts & Managing Files of Records.
Unidata’s TDS Workshop TDS Overview – Part II October 2012.
Feature Collections Subsetting 1. Overview 2. NCSS 2.1. Dataset description 2.2. Grid requests 2.3. Grid as point requests 3. CdmrFeature.
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
Animation of DSM2 Outputs in ArcMap Siqing Liu Bay Delta Office Department of Water Resources 2/17/2015.
CCGrid 2014 Improving I/O Throughput of Scientific Applications using Transparent Parallel Compression Tekin Bicer, Jian Yin and Gagan Agrawal Ohio State.
Scientific Computing Division A tutorial Introduction to Fortran Siddhartha Ghosh Consulting Services Group.
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
Deutscher Wetterdienst
Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept
Project 4 : SciDAC All Hands Meeting, September 11-13, 2002 A. Choudhary, W. LiaoW. Gropp, R. Ross, R. Thakur Northwestern UniversityArgonne National Lab.
Accessing Remote Datasets using the DAP protocol through the netCDF interface. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
_______________________________________________________________CMAQ Libraries and Utilities ___________________________________________________Community.
Climate Data Formats Deniz Bozkurt
NetCDF Data Model Issues Russ Rew, UCAR Unidata NetCDF 2010 Workshop
1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA.
Sciamachy features and usage with respect to end-users The typical fate of retrieval people dealing with large datasets… C. Frankenberg, SRON team, IUP.
CKD Workshop  30 March 2011  Jim Kinter  Data Lessons from Project Athena GrADS Station Data Model Used for in situ observational data Individual reports.
CCGrid, 2012 Supporting User Defined Subsetting and Aggregation over Parallel NetCDF Datasets Yu Su and Gagan Agrawal Department of Computer Science and.
The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.
Information Technology: GrADS INTEGRATED USER INTERFACE Maps, Charts, Animations Expressions, Functions of Original Variables General slices of { 4D Grids.
NetCDF-4: Software Implementing an Enhanced Data Model for the Geosciences Russ Rew, Ed Hartnett, and John Caron UCAR Unidata Program, Boulder
NetCDF and Scientific Data Durability Russ Rew, UCAR Unidata ESIP Federation Summer Meeting
Data File Formats: netCDF by Tom Whittaker University of Wisconsin-Madison SSEC/CIMSS 2009 MUG Meeting June, 2009.
Advances in the NetCDF Data Model, Format, and Software Russ Rew Coauthors: John Caron, Ed Hartnett, Dennis Heimbigner UCAR Unidata December 2010.
Postgraduate Computing Lectures PAW 1 PAW: Physicist Analysis Workstation What is PAW? –A tool to display and manipulate data. Learning PAW –See ref. in.
UC 2006 Tech Session 1 NetCDF in ArcGIS 9.2. UC 2006 Tech Session2 Overview Introduction to Multidimensional DataIntroduction to Multidimensional Data.
NetCDF and binary read in MATLAB Pierre Chien 2009/03/19.
NetCDF: Data Model, Programming Interfaces, Conventions and Format Adapted from Presentations by Russ Rew Unidata Program Center University Corporation.
Update on Unidata Technologies for Data Access Russ Rew
NcBrowse: A Graphical netCDF File Browser Donald Denbo NOAA-PMEL/UW-JISAO
Utilities for netCDF-4 Dr. Dennis Heimbigner Unidata Advanced netCDF Workshop July 25, 2011.
TSDS (HPDE DAP). Objectives (1) develop a standard API for time series-like data, (2) develop a software package, TSDS (Time Series Data Server), that.
Unidata Infrastructure for Data Services Russ Rew GO-ESSP Workshop, LLNL
NetCDF Data Model Details Russ Rew, UCAR Unidata NetCDF 2009 Workshop
® Sponsored by Improving Access to Point Cloud Data 98th OGC Technical Committee Washington DC, USA 8 March 2016 Keith Ryden Esri Software Development.
Other Projects Relevant (and Not So Relevant) to the SODA Ideal: NetCDF, HDF, OLE/COM/DCOM, OpenDoc, Zope Sheila Denn INLS April 16, 2001.
(Network Common Data Form)
SRNWP Interoperability Workshop
NetCDF 3.6: What’s New Russ Rew
What is FITS? FITS = Flexible Image Transport System
Common Wind Tunnel Data Format
Unidata Advanced netCDF Workshop
CdmCL, a Specific Textual Constraint Language for Common Data Model
NetCDF and Scientific Data Standard
Accessing Remote Datasets through the netCDF interface.
NCL variable based on a netCDF variable model
Presentation transcript:

Introduction to NetCDF Ernesto Munoz

Outline Overview of NetCDF Overview of NetCDF NetCDF file information NetCDF file information CDL utilities: ncdump, ncgen CDL utilities: ncdump, ncgen Reading NetCDF file Reading NetCDF file with Fortranwith Fortran with GrADSwith GrADS with Matlabwith Matlab Writing NetCDF file Writing NetCDF file

What is NetCDF? Network Common Data Form (NetCDF) Network Common Data Form (NetCDF) An interface for (array-oriented) data access An interface for (array-oriented) data access A collection of libraries of data-access routines (for Fortran, C++, etc.) A collection of libraries of data-access routines (for Fortran, C++, etc.) A machine-independent format for scientific data A machine-independent format for scientific data Developed at Unidata UCAR Developed at Unidata UCAR More information at: More information at:

Positive Attributes of NetCDF Self-describing – includes information about the data it contains Self-describing – includes information about the data it contains Portable - A machine-independent binary data format Portable - A machine-independent binary data format Direct-access – can access efficiently a subset of the dataset Direct-access – can access efficiently a subset of the dataset

NetCDF file contents Dimensions Dimensions Information about its name and length At most one unlimited dimension (i.e., a dimension with no maximum size) Variables Variables Information about its name, type, shape, attributes & values Coordinate variable: 1-D array with the same name as its dimension (e.g., latitude)Coordinate variable: 1-D array with the same name as its dimension (e.g., latitude) Primary variable: the dataPrimary variable: the data Fixed-length variable: primary variable with fixed dimensionsFixed-length variable: primary variable with fixed dimensions Record variable: primary variable with an unlimited dimensionRecord variable: primary variable with an unlimited dimension Attributes – “ metadata ” Attributes – “ metadata ” Information about its name, type, values & length Variable attributesVariable attributes Global attributesGlobal attributes

Example of NetCDF file contents Data values on a 2-D latitude/longitude grid Data values on a 2-D latitude/longitude grid E.g., zonal and meridional wind dataE.g., zonal and meridional wind data Information about the dimensions Information about the dimensions Latitudes, longitudes, vertical levels and timesLatitudes, longitudes, vertical levels and times The units of the data values The units of the data values Scaling factor and offset to be applied to data Scaling factor and offset to be applied to data

CDL - Common Data Language NetCDF ↔ ASCII translator NetCDF ↔ ASCII translator Its two main utilities: Its two main utilities: 1. ncdump (from netcdf to ascii) 1. ncdump (from netcdf to ascii) to get overview of NetCDF file contentsto get overview of NetCDF file contents 2. ncgen (from ascii to netcdf) 2. ncgen (from ascii to netcdf) to create NetCDF filesto create NetCDF files to create Fortran (or C) programs to write NetCDF filesto create Fortran (or C) programs to write NetCDF files

ncdump For quick glimpse at file’s “metadata” To view header info: To view header info: ncdump -h myfile.nc To view header info plus values of coordinate variables: To view header info plus values of coordinate variables: ncdump -c myfile.nc To output data values in ascii form: To output data values in ascii form: ncdump –v varname myfile.nc

Exercise Look at description of file with ncdump Compare -c and -h options Compare -c and -h options ncdump -h myfile.nc > filehdr.txt ncdump -c myfile.nc > filehdrdim.txt

Reading NetCDF with Fortran Typical order of calls to read a NetCDF file NF_OPEN ! open NetCDF dataset NF_INQ_DIMID ! get dimensions IDs NF_INQ_DIMID ! get dimensions IDs NF_INQ_DIMLEN ! get dimensions lengths NF_INQ_DIMLEN ! get dimensions lengths NF_INQ_VARID ! get variables IDs NF_INQ_VARID ! get variables IDs NF_GET_ATT ! get attribute values NF_GET_ATT ! get attribute values NF_GET_VAR ! get values of variables NF_GET_VAR ! get values of variables NF_CLOSE ! close NetCDF dataset

Reading NetCDF With Matlab With Matlab Read Takemasa’s memo at:Read Takemasa’s memo at: mo_netcdf.htmlhttp:// mo_netcdf.html With GrADS With GrADS Open GrADS with gradsnc command (instead of grads)Open GrADS with gradsnc command (instead of grads) Open NetCDF file with sdfopen command (instead of open)Open NetCDF file with sdfopen command (instead of open) For more information refer to: more information refer to:

Writing a NetCDF file with ncgen ncgen can be used to: ncgen can be used to: Generate code that will produce a NetCDF fileGenerate code that will produce a NetCDF file Fortran: ncgen -f mydata.cdl > mydata.f Fortran: ncgen -f mydata.cdl > mydata.f C: ncgen -c mydata.cdl > mydata.c C: ncgen -c mydata.cdl > mydata.c Generate a NetCDF fileGenerate a NetCDF file ncgen -o mydata.nc mydata.cdl ncgen -o mydata.nc mydata.cdl Best practices to writing NetCDF files: Best practices to writing NetCDF files: