1. Use of OpenEarthTools to make netCDF
Extract
Transform
Load
Provide
Subversion repository
add meta information
data
netCDF on OPeNDAP server
models
transform to netCDF
netCDF on web server
tools 2 1
data providers = data users
3 D:\... 3

2. netCDF on OPeNDAP server
Contents
Extract
Transform
Load
Provide 2a
use tools > 2b
read data 2c
> save as netCDF
add meta information
data
netCDF on OPeNDAP server
models
transform to netCDF
netCDF on web server
tools 1
data providers = data users
3 D:\... 3

3. OpenEarthTools Download and install TortoiseSVN Subversion client
Make a checkout of OpenEarthTools (full or just Matlab subset)
Start Matlab and go to OpenEarthTools
>> cd F:\checkouts\OpenEarthTools\matlab\
Add OpenEarth paths to Matlab:
>> oetsettings
Inspect OpenEarthTools
When many people work on oen thing, we need a few basic rules
Document your functions and toolboxes
Exploit the clickable links to files + directories in new Matlab version
>> help functionname % in path
>> help directoryname % in path
Displays:
all H1 lines (first comment line in file)
or displays help in contents.m
Add crosslinks: with 3 crosslinks, 6 clicks connect 729 files ≈ size OpenEarthTools
See also

4. OpenEarthTools documentation
www links
other m function
other directories in path

5. OpenEarthTools documentation
>> help applications
OpenEarth applications
rijkswaterstaat - data: rijkswaterstaat data types (donar)
delft3d delftd file formats
knmi read knmi wind and meteo timeseries
swan swan toolbox
>> help rijkswaterstaat
donar_read read ASCII text file from
getwaterbase get data from waterbase.nl
getwaterbasestation - get stations names for parameter
getwaterbaseparameter - get parameter from waterbase.nl
hmcz_read read meteo file from rws
>> help donar_read
DONAR_READ read ASCII text file from
DAT = donar_read(fname,<keyword,value>)
See also: getwaterbase,
contents.m
H1 line
donar_read.m
% DONAR_READ read ASCII text file from %
% DAT = donar_read(fname,<keyword,value>)
% See also: getwaterbase,

6. netCDF on OPeNDAP server
Contents
Extract
Transform
Load
Provide 2a
use tools > 2b
read data 2c
> save as netCDF
add meta information
data
netCDF on OPeNDAP server
models
transform to netCDF
netCDF on web server
tools 1
data providers = data users
3 D:\... 3

7. Read a raw dataset Always read data into one struct
Separate columns (Excel) into fieldnames
Include as much meta-info as possible in the struct
Use English names
Use SI units
Use documentation: cells are block starting with %% (~ chapters)
Align

8. Read a raw dataset Dedicated matlab tools >> xlsread
>> textscan
>> csvread
Dedicated OpenEarthTools
>> donar_read
>> KNMI knmi_potwind
>> knmi_etmgeg
/ SeaDataNet profile data (CTD):
>> odvread
Dedicate yourselves by peeling the file & add to OpenEarthTools:
>> oetnewfun, load_template
>> fgetl, fscanf % ascii
>> fread % binary

9. Read a raw data waterbase

10. Read a raw data KNMI wind

11. Read a raw data KNMI meteo

12. Read a raw data NODC ODV

13. Test with some raw data
Test for yourselve some raw data with dedicated scripts, choose one.
Find the associated matlab function using the help documentation.
Rijkswaterstaat
KNMI
NODC
/ SeaDataNet profile data (CTD):

14. Inspect a raw dataset raw data file Different types of raw data
Different structs for time series. Hmm.
Waterbase
KNMI wind
KNMI meteo
How to unify?
No guidelines for structs
International guidelines for netCDF!
Idea: collaborate!
Each BwN participant transforms
raw data they use anyway
into netCDF
Everyone can readily access
previously used data in a
uniform manner
No standards available
raw data struct
International standards available
netCDF

15. netCDF on OPeNDAP server
Contents
Extract
Transform
Load
Provide 2a
use tools > 2b
read data 2c
> save as netCDF
add meta information
data
netCDF on OPeNDAP server
models
transform to netCDF
netCDF on web server
tools 1
data providers = data users
3 D:\... 3

16. netCDF What is netCDF How to make netCDF Matlab netCDF interface
Conventions (vocabularies):
Standard meta-info: CF convention + GNU/CC/?
standard quantities: CF convention
Standard units: Unidata UD units
Standard coordinate codes: EPSG
Save script in repository
OpenEarthRawData: for dedicated script
OpenEarthTools: for more general script
Inspect and plot netCDF

17. netCDF = linked dimensions, variables, attributes
Self-describing file:
meta-info
never lost
no manual
needed
open file format
guarantees
future access
units standard_name comment fillvalue
z = f(z)
t =f(t)
standard_name
standard_name
long_name
long_name
units
units
# = f(#)
long_name t
units z
chlorophyll = f(t,z,#)
standard_name
long_name
units
title
conventions #
website
institute
disclaimer
copyright

18. Let’s look at a basic netCDF file
nc_cf_example2nc.nc (in OpenEarthTools checkout)
>> nc_dump(‘nc_cf_example2nc.nc’)
ncBrowse
HDFview
Hierarchical structure as file system
Attributes
Dimensions
Variables

19. netCDF (1) Any amount of meta-data (Attributes) can be appended.
Handles large datasets (file > 4GB allowed with in 64-bit offset mode).
Scaleable for future applications (divide data over multiple files).
Open source specifications and interfaces.
Fast data storage and retrieval (direct access binary).
Small data storage (binary is small + internal zip option).
Simple yet robust.
“Following examination by 20 reviewers, NASA's Earth Science Data Systems Standards Process Group has concluded that netCDF classic should be adopted as a recommended standard.
A major strength of netCDF classic, according to the reviewers, was that it has fostered data interoperability and exchange through its self-describing file format, platform independent architecture, and robust access methods. Additionally, its overall file format and metadata attributes were simple enough to be easily understood and applied yet robust enough to describe and store multidimensional data of different types in the same file” (source: Unidata news)

20. netCDF (2)
2 independent implementations of NetCDF interface (C & Java):
reliable
redundant
Generally used, so there’s always someone who can offer help available.
netCDF is de facto standard for coastal ocean circulation models (GOTM, ROMS, ECOM-SED and in the near future also Delft3D-FLOW).
Lots of simple and free tools on the web that operate on NetCDF files:
ncBrowse (NOAA)
NCO
Panoply (NASA)
ncview (linux)
Implemented in numerous commercial tools too
Quickplot (Delft3D)

21. netCDF (3)
Interfaces to sophisticated scientific data manipulation languages:
C-interface
Java-interface
Fortran
Python
IDL
Matlab
Through java-interface (in OET)
mexcdf for C-interface (in OET)
Native matlab interface (since release 2008b)
Merged with OPeNDAP-interface
Handle netCDF file on web as if it on your own PC.
Additional, widely accepted conventions for NetCDF
implemented in main NetCDF viewers
CF meta data convention

22. Creating a netCDF file See : nc_cf_example2nc.m
Now step by step walking through
Use nc_cf_example2nc.nc as template when doing it yourselves

23. >> nc_create_empty % 1
units standard_name comment fillvalue
z = f(z)
t =f(t)
standard_name
standard_name
long_name
long_name
units
units
# = f(#)
long_name t
units z
chlorophyll = f(t,z,#)
standard_name
long_name
units
title
conventions #
website
institute
disclaimer
copyright

24. >> nc_create_empty % 0
Web-sources (waterbase, knmi)
Read directly from web
Or download to local cache
Your own data:
Save raw data in repository
OpenEarthRawData with
INSPIRE XML meta-data
Then read data in Matlab
and create empty netCDF file

25. >> nc_attput % 1 z = f(z) t =f(t) # = f(#) t z
units standard_name comment fillvalue
z = f(z)
t =f(t)
standard_name
standard_name
long_name
long_name
units
units
# = f(#)
long_name t
units z
chlorophyll = f(t,z,#)
standard_name
long_name
units
title
conventions #
website
institute
disclaimer
copyright

26. >> nc_attput % 1
See CF meta-data conventions:
OpenEarth conventions:
disclaimer + terms_for_use
$Headurl$

27. >> nc_add_dimension % 2
units standard_name comment fillvalue
z = f(z)
t =f(t)
standard_name
standard_name
long_name
long_name
units
units
# = f(#)
long_name t
units z
chlorophyll = f(t,z,#)
standard_name
long_name
units
title
conventions #
website
institute
disclaimer
copyright

28. >> nc_add_dimension % 2

29. >> nc_addvar % 3 z = f(z) t =f(t) # = f(#) t z
units standard_name comment fillvalue
z = f(z)
t =f(t)
standard_name
standard_name
long_name
long_name
units
units
# = f(#)
long_name t
units z
chlorophyll = f(t,z,#)
standard_name
long_name
units
title
conventions #
website
institute
disclaimer
copyright

30. >> nc_addvar % 3

31. >> nc_varput % 4
CF convention
vocabulary of
quantities

32. >> nc_varput % 4
UNIDATA UD units package
vocabulary of
units

33. >> nc_varput % 4 z = f(z) t =f(t) # = f(#) t z
units standard_name comment fillvalue
z = f(z)
t =f(t)
standard_name
standard_name
long_name
long_name
units
units
# = f(#)
long_name t
units z
chlorophyll = f(t,z,#)
standard_name
long_name
units
title
conventions #
website
institute
disclaimer
copyright

34. >> nc_varput % 4

35. >> nc_dump % 5 z = f(z) t =f(t) # = f(#) t z
units standard_name comment fillvalue
z = f(z)
t =f(t)
standard_name
standard_name
long_name
long_name
units
units
# = f(#)
long_name t
units z
chlorophyll = f(t,z,#)
standard_name
long_name
units
title
conventions #
website
institute
disclaimer
copyright

36. >> nc_dump % 5

37. >> nc_dump % 5

38. Plot netCDF in free ncBrowse

39. Plot netCDF in free ncBrowse

40. Plot netCDF in Matlab

41. Plot netCDF in ArcGis ArcGIS
ArcGis also reads and writes netcdf files.
ArcGIS

42. Check netCDF in your own text editor (xml)
xml representation of a netcdf file
XML

43. Check netCDF in … IDV NCO JAVA netCDF tools Not so stable. Very useful
#diff
ncdiff -v time file1.nc file2.nc
#compression & packingncpdq -4 -L 9 in.nc out.nc # Deflated packing (~80% lossy compression)
#selecting variables by regex
ncks -v '^Q..' in.nc # Q01--Q99, QAA--QZZ, etc.
Useful
JAVA netCDF tools
Web hyperslabs, cool!

44. Summary current session
Extract
Transform
Load
Provide 2a
use tools > 2b
read data 2c
> save as netCDF
add meta information
data
netCDF on OPeNDAP server
models
transform to netCDF
netCDF on web server
tools 1
data providers = data users
3 D:\... 3

45. Next session: using netCDF from others
Extract
Transform
Load
Provide
add meta information
data
netCDF on OPeNDAP server
models
transform to netCDF
netCDF on web server
tools 2 1
data providers = data users
3 D:\... 3

46. Read a raw data very large example
WATERBASE returns 400 Mb ASCII files for water levels
Has incomplete meta-info (no location code)
Mixture of Amersfoort, (lat,lon), UTM31 coordinates

47. Read a raw data example Get ASCII wind file from KNMI website
Read ASCII wind files as if they were one
Transform to Delft3D standards (time in minutes): W2 = f(W)
Write as Delft3D *.wnd file