Download presentation
Presentation is loading. Please wait.
1
Deutscher Wetterdienst
Data Formats and Tools Deutscher Wetterdienst R.W. Mueller, R.Hollmann, C.Träger-Chatterjee
2
Content Overview HDF5 netCDF Binary ASCII Conclusion
3
Overview New data formats have been developed
better handling of manifold information provided by satellite data, reanalysis or model data optimise computing performance (IO-process) reduce disk space needed Requirements to the data format: storage of … … the data itself with high resolution in space-time Different data layers possible … the meta-information, e.g. Calibration coefficients Geolocation and projection Statistical error information Gain and offset Whatever the operator would like to add as meta information
4
Overview – the favourites
For satellite data two formats are important. Different but related – both with associated data model HDF: Hierarchical data format netCDF: network Common Data Form Further formats for satellite data: HRIT raw data format not discussed, focus on products Specific Binary Format always possible, no common data model ASCII no data model, quite seldom
5
HDF – Hierarchical Data Format
HDF5 - general purpose library and file format for storing scientific data Create and store almost any kind of scientific data structure e.g. images, arrays of vectors, structured and unstructured grids, … one can also mix and match different data formats in HDF5 files Efficient storage and I/O created to address the data management needs of high performance, data intensive computing environments As a result, library and format emphasize storage and I/O efficiency (especially on parallel machines), including file compression
6
HDF – Hierarchical Data Format
The most recent version is HDF5, but a lot of data are still in HDF4 format. Both are machine independent (no big / little endian problem) Information, tools, examples and the HDF software (library) available at and Widely used, e.g.: MODIS (HDF4) Eumetsat, e.g. all SAFs (HDF5)
7
HDF command line tools No downward compatibility
many hdf5 command line tools and interfaces (e.g. implemented in f90,c programs) can not be used for HDF4 files. h5dump - dumps displays the input of the hdf file in ASCII h5ls - lists the contents of a file, enables fast checks if the needed data is in there h5import - imports ASCII to hdf5 configuration file is needed, hence some basic knowledge about HDF data model and structure required
8
HDF5 as ASCII using h5dump
Common data model but in detail it can look quite different, comments in red !! HDF5 "TRS_SR_ _1200_V000.hdf" { filename GROUP "/" { definition of a group GROUP "Data" { DATASET "TRS" { definition of the dataset DATATYPE H5T_STD_I16BE def. of the data type DATASPACE SIMPLE { ( 3712, 3712 ) / ( 3712, 3712 ) } the dimension DATA { the data (0,0): , , , , , , , (0,7): , , , , , , ,…. (883,707): 495, 455, 436, 436, 378, 323, 378, 416, 342, 277, 296, ……} ATTRIBUTE "Gain" { … definition of attributes continued on the next slide
9
HDF5 as ASCII using h5dump
ATTRIBUTE "Gain" { … Gain and offset DATATYPE H5T_IEEE_F32BE used to reduce needed DATASPACE SCALAR disk space (possible to DATA { save data as integer) (0): 0.25 } ATTRIBUTE "Offset" {….. DATATYPE and DATASPACE …. DATA { (0): 0 ATTRIBUTE "nodatavalue" {…. DATATYPE and DATASPACE …… DATA { Attribute is also used (0): for unit, title,… Gain: e.g. divied by 100 Offset: e.g. subtract 5 (???)
10
HDF5 as ASCII using h5dump
GROUP "Geolocation" { definition of a new group, and the DATASET "projection" { dataset needed to define the projection DATATYPE H5T_COMPOUND { H5T_STRING { STRSIZE 128; STRPAD H5T_STR_NULLTERM; CSET H5T_CSET_ASCII; CTYPE H5T_C_S1; } "reference ellipsoid"; H5T_ARRAY { [10] H5T_IEEE_F32LE } "parameter"; } DATASPACE SIMPLE { ( 1 ) / ( 1 ) } DATA { (0): { "geostationary view", "WGS-84", [ 1856, 1856, , , -1, -1, -1, -1, -1, -1 ] } }} DATASET "region" { a group usually consists of different datasets }}}
11
HDF GUI Tools -HDFView-
The complex data model might act as a deterrend for beginners Graphical User Interface HDFView overcomes this handicap. It is a tool for browsing and editing HDF4 and HDF5 files using a GUI Relatively easy to install and available for many platforms, e.g Windows, Solaris, AIX, Linux Everything can be managed with buttons and mouse clicks Data can be saved as ASCII table Images can be generated and saved.
12
HDF Tools – CMSAF GUI Features:
Software available for CM-SAF customers via Features: visualisation of CM-SAF products (in HDF5 format) simple data analysis Export (ASCII, lat/lon grid) Uses free IDL Virtual Machine
13
HDF Tools CM-SAF GUI Example for Metadata
14
More on this topic in the exercise session
HDF Tools CM-SAF GUI More on this topic in the exercise session
15
netCDF Information, tools, examples and the netCDF library are available at: Widely used, e.g.: Reanalysis data of National Centers for Environmental prediction (NCEP) and European Centre for Medium Weather forecast (ERA40) HOAPS, Hamburg Ocean Atmosphere Parameters and Fluxes from Satellite Data CM-SAF selected monthly means
16
netCDF command line tools
ncdump - file shows the input of the netCDF file ncgen - converts ascii to netcdf and vica versa sounds easy but a configuration file (CDL file) is needed some basic knowledge about the net CDF data model and structure however, easier to handle for beginners as HDF5 example of ASCII CDL configuration file:
17
netCDF as ASCII netcdf SRBmm200604 { dimensions: lat = 501 ;
lon = 741 ; time = UNLIMITED ; // (0 currently) variables: float lat(lat) ; lat:long_name = "latitude" ; lat:units = "degree" ; float lon(lon) ; lon:long_name = "longitude" ; lon:units = "degrees" ; float Z(lat, lon) ; Z:units = "Watt" ; Z:valid_range = 0., ; data: lat = 35, 35.05, 35.1, 35.15, 35.2, 35.25, 35.3, 35.35, 35.4, … ; lon= 44,45,….; Z=300,340,…; }
18
netCDF Tools – Integrated Data Viewer (IDV)
Free GIS tool Display data / generate maps Imports netCDF
19
netCDF GUI Tool CDAT Open source integrated environment for data analysis and visualisation. Mainly netCDF, but can also deal with GRIB and HDF. Import of binary and ASCII data possible. Available for different platforms but not for Windows!
20
Binary Data Usually used instead of ASCII
to reduce disk space and to increase the computing performance. Machine readable format not readable by humans Usually files with / without header and data as defined data type e.g. float (2.44) or integer (4) Reading and writing with e.g. C, C++, Fortran Formats are not common indivdual read / write routines needed some tools can read and visualise binary data. e.g. CDAT, GRADS, idl data is not self-explanatory The length of the header and the data type has to be known
21
Binary Data and ASCIII Examples for binary data:
International Satellite Cloud Climatology Project (ISCCP). AVHRR based USGS land use maps.
22
ASCII readable with a text editor a quite unusual format
sometimes provided by the data centre for subsets of the data on request, e.g. CM-SAF I do not know any example
23
Conclusion HDF5 netCDF Binary ASCII
Header, describing the data. Data in binary format HDF-View, CM-SAF GUI Official format of CM-SAF Daten netCDF Header, describing the data. Less cryptic than HDF5. Data in binary format Diverse GIS, e.g. ArcView, Integrated Data Viewer, CDAT On demand some CM-SAF data can be provided in netCDF. Binary Instead of ASCII, to reduce disk space ASCII Readable with a text editor.
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.