Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.

Slides:



Advertisements
Similar presentations
Introduction to C Programming
Advertisements

Symbol Table.
Core Java Lecture 4-5. What We Will Cover Today What Are Methods Scope and Life Time of Variables Command Line Arguments Use of static keyword in Java.
The Web Warrior Guide to Web Design Technologies
Chapter 5: Elementary Data Types Properties of types and objects –Data objects, variables and constants –Data types –Declarations –Type checking –Assignment.
The Future of NetCDF Russ Rew UCAR Unidata Program Center Acknowledgments: John Caron, Ed Hartnett, NASA’s Earth Science Technology Office, National Science.
Homework Any Questions?. Statements / Blocks, Section 3.1 An expression becomes a statement when it is followed by a semicolon x = 0; Braces are used.
©2004 Brooks/Cole Chapter 2 Variables, Values and Operations.
Introduction to NetCDF Ernesto Munoz. Outline Overview of NetCDF Overview of NetCDF NetCDF file information NetCDF file information CDL utilities: ncdump,
NetCDF 3.6: What’s New Russ Rew Unidata Program Center University Corporation for Atmospheric Research
1 Key Concepts:  Why C?  Life Cycle Of a C program,  What is a computer program?  A program statement?  Basic parts of a C program,  Printf() function?
Guide To UNIX Using Linux Third Edition
Cs164 Prof. Bodik, Fall Symbol Tables and Static Checks Lecture 14.
Chapter 3: Introduction to C Programming Language C development environment A simple program example Characters and tokens Structure of a C program –comment.
NetCDF Ed Hartnett Unidata/UCAR
Introduction to NetCDF Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 13 April 2011.
Status of netCDF-3, netCDF-4, and CF Conventions Russ Rew Community Standards for Unstructured Grids Workshop, Boulder
Show of Hands... How many traveled to be here? University/Gov't/Industry How many use netCDF? Primary programming language for netCDF? Other data formats.
Comp 248 Introduction to Programming Chapter 4 - Defining Classes Part A Dr. Aiman Hanna Department of Computer Science & Software Engineering Concordia.
Windows Programming Lecture 05. Preprocessor Preprocessor Directives Preprocessor directives are instructions for compiler.
INTRODUCTION TO JAVA CHAPTER 1 1. WHAT IS JAVA ? Java is a programming language and computing platform first released by Sun Microsystems in The.
NetCDF-4 and Parallel I/O GSFC, Nov 20,2008 Ed Hartnett.
NetCDF for High Performance Computing Introduction to NetCDF What is netCDF? NetCDF Data Models How we think of data. NetCDF Software Libraries Using.
CH2 – Using Data. Constant Something which cannot be changed Data Type Format and size of a data item Intrinsic Data Types Pg. 47 – Table 2-1 Basic ones.
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
Chapter 8: Arrays.
Arrays An array is a data structure that consists of an ordered collection of similar items (where “similar items” means items of the same type.) An array.
Deutscher Wetterdienst
Constants Numeric Constants Integer Constants Floating Point Constants Character Constants Expressions Arithmetic Operators Assignment Operators Relational.
Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept
Project 4 : SciDAC All Hands Meeting, September 11-13, 2002 A. Choudhary, W. LiaoW. Gropp, R. Ross, R. Thakur Northwestern UniversityArgonne National Lab.
Accessing Remote Datasets using the DAP protocol through the netCDF interface. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
NetCDF Data Model Issues Russ Rew, UCAR Unidata NetCDF 2010 Workshop
BY A Mikati & M Shaito Awk Utility n Introduction n Some basics n Some samples n Patterns & Actions Regular Expressions n Boolean n start /end n.
Introduction to Programming
ISBN Chapter 6 Data Types Introduction Primitive Data Types User-Defined Ordinal Types.
Chapter 3 Syntax, Errors, and Debugging Fundamentals of Java.
Liang, Introduction to Programming with C++, Second Edition, (c) 2010 Pearson Education, Inc. All rights reserved Chapter 13 File Input and.
Copyright Curt Hill Variables What are they? Why do we need them?
NetCDF and Scientific Data Durability Russ Rew, UCAR Unidata ESIP Federation Summer Meeting
 In the java programming language, a keyword is one of 50 reserved words which have a predefined meaning in the language; because of this,
Advances in the NetCDF Data Model, Format, and Software Russ Rew Coauthors: John Caron, Ed Hartnett, Dennis Heimbigner UCAR Unidata December 2010.
CSI 3125, Preliminaries, page 1 Data Type, Variables.
Aside: Running Supplied *.java Programs Just double clicking on a *.java file may not be too useful! 1.In Eclipse, create a project for this program or.
NTFS Filing System CHAPTER 9. New Technology File System (NTFS) Started with Window NT in 1993, Windows XP, 2000, Server 2003, 2008, and Window 7 also.
Unidata Technologies Relevant to GO-ESSP: An Update Russ Rew
Developing Conventions for netCDF-4 Russ Rew, UCAR Unidata June 11, 2007 GO-ESSP.
Utilities for netCDF-4 Dr. Dennis Heimbigner Unidata Advanced netCDF Workshop July 25, 2011.
NetCDF Data Model Details Russ Rew, UCAR Unidata NetCDF 2009 Workshop
Other Projects Relevant (and Not So Relevant) to the SODA Ideal: NetCDF, HDF, OLE/COM/DCOM, OpenDoc, Zope Sheila Denn INLS April 16, 2001.
The purpose of a CPU is to process data Custom written software is created for a user to meet exact purpose Off the shelf software is developed by a software.
Information and Computer Sciences University of Hawaii, Manoa
DAP+NETCDF Using the netCDF-4 Data Model
JAVA MULTIPLE CHOICE QUESTION.
APPENDIX a WRITING SUBROUTINES IN C
NetCDF 3.6: What’s New Russ Rew
Other Kinds of Arrays Chapter 11
Other Kinds of Arrays Chapter 11
Programming Language Concepts (CIS 635)
Computer Science 210 Computer Organization
Unidata Advanced netCDF Workshop
Introduction to Primitive Data types
Chapter 2: Java Fundamentals
Variables Title slide variables.
Compound Statements A Quick Overview
Homework Any Questions?.
Java Programming Language
NCL variable based on a netCDF variable model
SPL – PS1 Introduction to C++.
Introduction to Primitive Data types
Presentation transcript:

Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009

Overview The NCGEN4 Utility NCGEN4 Command Synopsis netCDF4 in CDL: Types netCDF4 in CDL: Typed Attributes netCDF4 in CDL: Groups Scope Rules Specifying Data Constants Rules for Using Braces Special Attributes Debugging Note: The Cycle Building and Installing NCGEN4 Extended Example

The NCGEN4 Utility NCGEN4 extends the CDL language Includes all of the netCDF-4 data model True inverse of ncdump Includes the special attributes (chunking, etc) Supports binary, C, NOT FORTRAN (yet)  Experimental: Java and NcML Can produce 4 kinds of binary netCDF files:  netcdf-3 (classic) 32 bit  netcdf-3 (classic) 64 bit  netcdf-3 (classic, but stored in netcdf-4 file format)  netcdf-4 (supports full netcdf-4 model)

NCGEN4 Command Synopsis Primary Command Line Options [-b] Create a (binary) netCDF file (default) [-o ] Name for the binary netCDF file created [-k ] Format of the file to be created 1 => classic 32 bit 2 => classic 64 bit 3 => netcdf-4/CDM 4 => classic, but stored in an enhanced file format [-x] Don't initialize data with fill values [-l ] Specify output language to use when generating source code to create or define a netCDF file matching the CDL specification. Input CDL file

netCDF4 in CDL: Types New section called “types:” Consistent with the output of ncdump Supports the new primitive data types: ubyte, ushort, uint, string, int64 (LL), uint64 (ULL) Supports the new user-defined types: int enum enum_t {off=0,on=1,unknown=2}; opaque(11) opaque_t; compound cmpd_t { vlen_t f1; enum_t f2;}; int(*) vlen_t;

netCDF4 in CDL: Typed Attributes vlen_t v:attr = {17, 18, 19}; Attribute typing is optional (=> type inferred) Warning! x:attr = “abc”; is inferred to be type char, not string Instead say string x:attr = “abc”; Why? for backward compatibility with ncgen Good practice to add “_t” to the end of all type names Why? Because X :attr = … might be interpreted incorrectly; is X a type or variable?

netCDF4 in CDL: Groups group: g {…} A group can itself contain dimensions, types, variables, and groups Name prefixing allows references to types and dimensions that reside in other groups Example: /g/cmpd_t => Do not use ‘/’ in your names Pretty much like the Unix file system Or Windows, but using forward slashes

Scope Rules Scope rules determine how references to a dimension or type without a prefix are interpreted General rule: 1.Look in immediately enclosing group 2.Look in the parent of the immediately enclosing group and so on up the enclosing groups For dimensions, if not found => error For types, continue to search the whole group tree to find a unique match, then error if not found

Specifying Data Constants Constants for user defined types require the use of braces {…} in certain places. dimensions: d=2; types: int(*) vlen_t; compound cmpd_t { int64 f1; string f2;}; variables: vlen_t v1(d); cmpd_t v2(d); data: v1 = {7, 8, 9}, {17,18,19}; v2 = {107LL, “abc”}, { LL, “xyz”};

Rules for Using Braces The top level is automatically assumed to be a list of items, so it should not be inside {...}  Different than C constants lists Instances of UNLIMITED dimensions (other than as the first dimension) must be surrounded by {...} in order to specify the size. Instances of VLENs must be surrounded by {...} in order to specify the size. Compound instances must be embedded in {...} Compound fields may optionally be embedded in {...}. No other use of braces is allowed.

Special Attributes Special attributes specified in an ncgen4 CDL file will be properly handled Consistent with ncdump -s Global special attributes  “_Format” – specify the netCDF file format “classic” “64-bit offset” “netCDF-4” “netCDF-4 classic model”  Overridden by the -k flag

Special Attributes (cont.) Per-variable special attributes  “_ChunkSizes” – list of chunk sizes 1 per dimension  “_DeflateLevel” – compression level: integer (0-9)  “_Endianness” – “big” or “little”  “_Fletcher32” – “true” or “false” to set check summing  “_NoFill” – “true” or “false” to set persistent NoFill property  “_Shuffle” – “true” or “false” to set shuffle filter  “_Storage” – “contiguous” or “chunked” to set storage mode

Debugging Note Use the “Cycle”, Luke Use ncgen/ncgen4 to convert your.cdl to.nc Then use ncdump to convert your.nc to.cdl Compare.cdl to.cdl Watch out for UNLIMITED! dimensions: u = unlimited; variables: v1(u); v2(u); data: v1 = {1,2,3,4}; v2 = {7,8}; Ncdump produces v2 = {7,8,_,_};

Building and Installing NCGEN4 Easy: add --enable-ncgen4 to your list of./configure flags Ncgen4 will be installed along with ncdump and the original ncgen

Extended Example netcdf foo { types: ubyte enum enum_t {Clear = 0, Cumulonimbus = 1, Stratus = 2}; opaque(11) opaque_t; int(*) vlen_t; dimensions: lat = 10; lon = 5; time = unlimited ; variables: long lat(lat), lon(lon), time(time); float Z(time,lat,lon), t(time,lat,lon); double p(time,lat,lon); long rh(time,lat,lon); string country(time,lat,lon); ubyte tag;

Extended Example (cont.) // variable attributes lat:long_name = "latitude"; lat:units = "degrees_north"; lon:long_name = "longitude"; lon:units = "degrees_east"; time:units = "seconds since :00:00"; // typed variable attributes string Z:units = "geopotential meters"; float Z:valid_range = 0., 5000.; double p:_FillValue = ; long rh:_FillValue = -1; vlen_t :globalatt = {17, 18, 19};

Extended Example (cont.) data: lat = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90; lon = -140, -118, -96, -84, -52; group g { types: compound cmpd_t { vlen_t f1; enum_t f2;}; } // group g group h { variables: /g/cmpd_t compoundvar; data: compoundvar = { {3,4,5}, Stratus } ; } // group h }

Questions?