Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown

Visualization and Enabling Technologies Section National Center for Atmospheric Research Photos: Lynda Lester Mesa Lab · Boulder, Colorado

HDF/HDEOS Workshop XI, November 2007 Topics Two interfaces: the same capabilities (mostly) –NCL - a self-contained scripting language –PyNGL and PyNIO - Python modules Quick overview: visualization and analysis Data model State of our I/O component development Enabling access to HDF 5, HDFEOS 5, and NetCDF 4

Brief History NCAR Graphics HLUsNCLNIOPyNGLPyNIO PyNIO and PyNGL July 2006 (Numeric 24.x/NumPy 1.0) PyNIO 2005 PyHLU 2003 PyNGL 2004 Original Fortran Late 60s GKS/CGM/Fortran 77/UNIX conversion 1980s NCAR Graphics 1992: NCARG Conference C interface Mid 90s HLUs Mid 90s NCL Mid 90s GSUN - Late 90s 1995: NCL Conference 2000-present: 30+ NCL Workshops NIO 2005

HDF/HDEOS Workshop XI, November 2007 NCAR Command Language (NCL) A scripting language designed for scientific data analysis and visualization 1.Simple, robust file input and output 2.Hundreds of analysis functions 3.Visualizations (2D) are world class and highly customizable

HDF/HDEOS Workshop XI, November 2007 NCL: Data analysis Array-based syntax and operators Hundreds of functions - Spherical harmonics - Scalar and vector regridding - Vertical interpolation - EOFs Many tailored to geosciences Most handle missing data Can call C and Fortran routines

HDF/HDEOS Workshop XI, November 2007 NCL: Visualization High-quality and customizable visualizations Contours, XY, vectors, streamlines Maps with most common map projections Handles data on regular and irregular grids, triangular meshes Specialized scripts for skew-T, wind roses, histograms, Taylor diagrams, panels, bar charts GSUN interface: simplifies visualization Over 1,400 visualization options available

HDF/HDEOS Workshop XI, November 2007

PyNGL - Python module Python NCL Graphics Library Python version of popular GSUN interface Same publication-quality graphics as NCL Utilizes existing Python modules and development tools (swig, Numeric, NumPy) Contains some climate-specific data analysis functions Extensive and updated documentation

HDF/HDEOS Workshop XI, November 2007 NCL Data Model Based on netCDF 3 Language variables can have: Named dimensions Attributes Coordinates variables A convention in NetCDF but a language feature in NCL 1d array with the same name as a dimension Contains coordinate values for dimension elements

HDF/HDEOS Workshop XI, November 2007 Supported file format I/O One function reads all supported data formats: NetCDF (now including NetCDF4 classic), HDF4, HDF-EOS 2, GRIB 1 and 2 Writes NetCDF and HDF4 All file formats massaged into same model: file variables have basically the same features as internal variables Flat name space: HDF group names are appended to variable names

HDF/HDEOS Workshop XI, November 2007 PyNIO - Python module Same I/O library as NCL (libnio) Reads and writes same formats as NCL Same NetCDF-like view of all formats But interfaces with NumPy for data access Modeled on the Scientific NetCDF module –NioVariable type is a reference to the variable in the file (metadata is attached to variable) –Dereferencing it results in a NumPy variable with no metadata *Numpy is a Python array processing module

HDF/HDEOS Workshop XI, November D Coordinates Traditional NetCDF coordinate variables are 1D vectors However, satellite data and model data on modern grids require 2D coordinate grids 2D coordinate variables created on the fly for GRIB grids and HDFEOS Grid-type data Added value variables that allow data to be plotted on any map projection

HDF/HDEOS Workshop XI, November 2007 Enabling NetCDF 4 ‘classic’ Recompile with NetCDF 4 and HDF beta (we also included szip) Add options for specifying format and compression level Tests using ncl_convert2nc on GRIB files show reduction in file size by ~1/2 over GRIB. Caveats: –Beta version NetCDF 4 not yet supported on some architectures (64 bit mainly) –Not supported for OPeNDAP NetCDF client library

HDF/HDEOS Workshop XI, November 2007 Path to HDF, HDFEOS 5 and full NetCDF 4 support size_t for dimension sizes to enable large variables on 64-bit hosts Add support for more atomic types (int64) Support for groups in the file context User control of chunking Compound and variable length data types Components to read HDF 5 and HDFEOS 5 Current NetCDF module extended to handle v4 More support for aggregation

HDF/HDEOS Workshop XI, November 2007 Current status NCL released this week –NCSA-style Open Source license (finally) –Binaries available under slightly more restrictive license (because of included 3rd party software) PyNGL/PyNIO available –Supports NumPy 1.0.x and Python 2.5 –Binaries (easy installation) available for: Various flavors of Linux, Mac OSX, Cygwin Other Unix systems, 32 and 64 bit

Distribution of PyNGL/PyNIO sites Distribution of NCL sites

HDF/HDEOS Workshop XI, November 2007 Support Websites with extensive documentation Tutorials Hundreds of examples with downloadable scripts Active lists Hands-on training workshops

HDF/HDEOS Workshop XI, November 2007 Download locations NCL PyNGL and PyNIO

HDF/HDEOS Workshop XI, November 2007 Questions? me:

HDF/HDEOS Workshop XI, November 2007 What kind of research (from our ESG download website) astronomy climate change atmospheric researchrainfall oceanography cloud modeling air quality marine studies aerosol light absorption utility forecasting data conversion planetary atmospheres dust physics fire modeling hydrology electric power microbiology pollution paleoclimatology medical flood risk mgmt glaciers lightning salmon abundance acoustics storms agriculture weather forecasting education monsoons

filename: ruc2.T00Z.bgrbanl path: /Users/dbrown/grib/ruc2.T00Z.bgrbanl.grb file global attributes: dimensions: gridx_236 = 113 gridy_236 = 151 lv_HYBL2 = 40 lv_DBLL3 = 5 variables: float PRES_236_TRO ( gridx_236, gridy_236 ) center : US National Weather Service - NCEP (WMC) long_name : Pressure units : Pa _FillValue : -999 coordinates : gridlat_236 gridlon_236 level_indicator : 7 grid_number : 236 parameter_number : 1 model : RUC Model from FSL (isentropic; scale: 20km at 40N) forecast_time : 0 initial_time : 05/11/2000 (00:00) … float gridlat_236 ( gridx_236, gridy_236 ) corners : long_name : latitude GridType : Lambert Conformal Secant or Tangent, Conical or bipolar units : degrees_north … Sample GRIB output from PyNIO

HDF/HDEOS Workshop XI, November 2007 Resources (a.k.a. attributes) Same control mechanism for NCL and PyNGL Detailed control of the appearance of a visualization Example using POP grid ocean current data

HDF/HDEOS Workshop XI, November 2007

NCL and PyNGL visualization examples scatter and line plots, contours, vectors, skew-t, meteograms, maps, wind barbs, bar charts, streamlines, trajectories, filled polygons, paneled visualizations

Skew-T graphic Courtesy of Dennis Shea A Skew-T plot is used by meteorologists to analyze data from a balloon sounding.

Based on a visualization of Adam Phillips

Based on a visualization of Joel Norris (Scripps) using dummy data

Data from Climate Analysis Section Christian Guillemot

THORPEX - an international research and development program responding to the weather related challenges of the 21st century to accelerate improvements in the accuracy of 1-day to 2- week high impact weather forecasts for the benefit of society, the economy and the environment. GRIB 2 TIGGE (THORPEX Interactive Grand Global Ensemble) data Japan Meteorological Agency

Courtesy Jeff Yin NCAR CGD

Image courtesy of Nan Rosenbloom, CGD

First two map databases built-in; high-resolution available as simple download

Christophe Cassou (CNRS/CERFACS) This grid could be described as a tripole grid that is further modified by the arbitrary displacement of some portions of the grid to achieve finer resolution over areas of interest (typically, ocean areas).

Courtesy Mark Stevens, NCAR CGD

Brett D. Estrada Naval Research Laboratory

Grid from Tom Gross NOAA/NOS/CSDL/MMAP

HDF/HDEOS Workshop XI, November 2007 Special grids you can contour Soon to include vectors and streamlines Grids already on a triangular mesh Grids with “cell boundary” information Grids represented by one-dimensional arrays of x, y, and z values

HDF/HDEOS Workshop XI, November

HDF/HDEOS Workshop XI, November

HDF/HDEOS Workshop XI, November 2007 ARPEGE GRID (used by Meteo-France) Christophe Cassou (CNRS/CERFACS) Similar to ISCCP grid, but with somewhat finer resolution. Grid is rotated with respect to the globe so as to put its poles somewhere other than at the North and South Pole.

Data from Dave Randall, Todd Ringler, Ross Heikes of CSU Most geodesic grids appear to be formed by elaborating an icosahedron; each of the 20 faces of the icosahedron is subdivided into smaller triangles in a more or less obvious way.

Data from Dave Randall, Todd Ringler, Ross Heikes of CSU

Station data: contouring on a triangular mesh

HDF/HDEOS Workshop XI, November 2007 Grid from Brett D. Estrada, Naval Research Laboratory Triangular mesh from Brett D. Estrada, Naval Research Laboratory

Triangular mesh from Tom Gross NOAA/NOS/CSDL/MMAP

HDF/HDEOS Workshop XI, November 2007 Samples from VASPACKT Vectors and streamlines on a triangular mesh Developed by Dave Kennison Simple vectors Curly vectors Streamlines Examples use dummy data

Simple vectors

Curly vectors

Streamlines