Sponsored by the National Science Foundation GENI I&M Workshop NetCDF and Local Data Manager (LDM) Mike Zink November 4,

Sponsored by the National Science Foundation 2 November 3, 2010 Overview

Sponsored by the National Science Foundation 3 November 3, 2010 What is NetCDF NetCDF is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array- oriented scientific data. Uses HDF5 as data storage layer. Also provides read-only access to some HDF4, HDF5 archives. Parallel I/O for high performance computing. Developed and maintained by Unidata at UCAR

Sponsored by the National Science Foundation 4 November 3, 2010 Who uses NetCDF? NetCDF is widely used in University Earth Science community. Used for Intergovernmental Panel on Climate Change (IPCC) data sets. Used by NASA and other large data producers. Used for NEXRAD and CASA radar data (moments)

Sponsored by the National Science Foundation 5 November 3, 2010 NetCDF Example 5 float Reflectivity(Radial, Gate) ; Reflectivity:Units = "dBz" ; float Velocity(Radial, Gate) ; Velocity:Units = "MetersPerSecond" ; float SpectralWidth(Radial, Gate) ; SpectralWidth:Units = "MetersPerSecond" ; float DifferentialReflectivity(Radial, Gate) ; DifferentialReflectivity:Units = "dB" ; float DifferentialPhase(Radial, Gate) ; DifferentialPhase:Units = "Degrees" ; float CrossPolCorrelation(Radial, Gate) ; CrossPolCorrelation:Units = "Unitless" ; float NormalizedCoherentPower(Radial, Gate) ; NormalizedCoherentPower:Units = "Unitless" ; float SpecificPhase(Radial, Gate) ; SpecificPhase:Units = "DegreePerKm" ; float HPropagationPhase(Radial, Gate) ; HPropagationPhase:Units = "Radians" ; float VPropagationPhase(Radial, Gate) ; VPropagationPhase:Units = "Radians" ; int GateFlags(Radial, Gate) ; GateFlags:Units = "BitField" ; float CorrectedReflectivity(Radial, Gate) ; CorrectedReflectivity:Units = "dBZ" ; float CorrectedDifferentialReflectivity(Radial, Gate) ; CorrectedDifferentialReflectivity:Units = "dB" ;

Sponsored by the National Science Foundation 6 November 3, 2010 OpeNDAP Client for NetCDF OPenDAP ( is a widely supported protocol for access to remote data Defined and maintained by the OPenDAP organization Designed to serve as intermediate format for accessing a wide variety of data sources. Client is now built into netCDF C library.

Sponsored by the National Science Foundation 7 November 3, 2010 Local Data Manager (LDM) LDM software acquires data and shares them with other nodes Data product is treated as a opaque unit, thus nearly any data can be relayed LDM can handle data from: –National Weather Service "NOAAport channel 3" data streams –NEXRAD radar data –lightning data from the National Lightning Detection Network –GOES satellite imagery

Sponsored by the National Science Foundation 8 November 3, 2010 LDM Features User configurable –execute an arbitrary program with the data product as input Supports distributed processing –Data ingest functions can be separated from storage and use functions –LDM servers may be configured to relay subsets of data among themselves, distributing the storage and processing around the network. Extensible –New decoders can be added easily Event-driven –Handles data as it arrives

Sponsored by the National Science Foundation 9 November 3, 2010 Demo Data Flow Dynamic end-to-end Nowcasting –Mapping Nowcast Workflows onto GENI Archival Storage Radar Nodes raw live data Upstream LDM feed archived netcdf data archived netcdf data Nowcast Processing aggregated multi-radar data aggregated multi-radar data Post to Web Nowcast images for display NetCDF over LDM NetCDF over LDM LDM server Storage NetCDF over LDM NetCDF over LDM LDM server Merge NetCDF over LDM NetCDF over LDM LDM server Nowcasting

Sponsored by the National Science Foundation 10 November 3, 2010 Multi-radar NetCDF Data Nowcast Processing 1.DiCloud Archival Service (S3) 2.LDM Data Feed (EC2) 1.DiCloud Archival Service (S3) 2.LDM Data Feed (EC2) raw live data Generate raw live data ViSE/CASA radar nodes Generate raw live data ViSE/CASA radar nodes Use proxy to track usage-based spending on Amazon and enforce quotas and limits Use proxy to track usage-based spending on Amazon and enforce quotas and limits 1.Ingest mulit-radar data feeds 2.Merge and grid multi-radar data 2.Generate 1min, 5min, and 10min Nowcasts 3.Send results over NLR to Umass 4.Repeat 1.Ingest mulit-radar data feeds 2.Merge and grid multi-radar data 2.Generate 1min, 5min, and 10min Nowcasts 3.Send results over NLR to Umass 4.Repeat ViSE views steerable radars as shared, virtualized resources ViSE views steerable radars as shared, virtualized resources Nowcast images for display Nowcast images for display