Use of OpenEarthTools to make netCDF Extract Transform Load Provide Subversion repository add meta information data netCDF on OPeNDAP server models transform to netCDF netCDF on web server tools 2 1 data providers = data users 3 D:\... 3 http://…

netCDF on OPeNDAP server Contents Extract Transform Load Provide 2a use tools > 2b read data 2c > save as netCDF add meta information data netCDF on OPeNDAP server models transform to netCDF netCDF on web server tools 1 data providers = data users 3 D:\... 3 http://…

OpenEarthTools Download and install TortoiseSVN Subversion client http://subversion.tigris.org/ Make a checkout of OpenEarthTools (full or just Matlab subset) Start Matlab and go to OpenEarthTools >> cd F:\checkouts\OpenEarthTools\matlab\ Add OpenEarth paths to Matlab: >> oetsettings Inspect OpenEarthTools When many people work on oen thing, we need a few basic rules Document your functions and toolboxes Exploit the clickable links to files + directories in new Matlab version >> help functionname % in path >> help directoryname % in path Displays: all H1 lines (first comment line in file) or displays help in contents.m Add crosslinks: with 3 crosslinks, 6 clicks connect 729 files ≈ size OpenEarthTools See also

OpenEarthTools documentation www links other m function other directories in path

OpenEarthTools documentation >> help applications OpenEarth applications rijkswaterstaat - data: rijkswaterstaat data types (donar) delft3d - delftd file formats knmi - read knmi wind and meteo timeseries swan - swan toolbox >> help rijkswaterstaat donar_read - read ASCII text file from www.waterbase.nl getwaterbase - get data from waterbase.nl getwaterbasestation - get stations names for parameter getwaterbaseparameter - get parameter from waterbase.nl hmcz_read - read meteo file from hmcz@ rws >> help donar_read DONAR_READ read ASCII text file from www.waterbase.nl DAT = donar_read(fname,<keyword,value>) See also: getwaterbase, www.waterbase.nl contents.m H1 line donar_read.m % DONAR_READ read ASCII text file from www.waterbase.nl % % DAT = donar_read(fname,<keyword,value>) % See also: getwaterbase, www.waterbase.nl

netCDF on OPeNDAP server Contents Extract Transform Load Provide 2a use tools > 2b read data 2c > save as netCDF add meta information data netCDF on OPeNDAP server models transform to netCDF netCDF on web server tools 1 data providers = data users 3 D:\... 3 http://…

Read a raw dataset Always read data into one struct Separate columns (Excel) into fieldnames Include as much meta-info as possible in the struct Use English names Use SI units Use documentation: cells are block starting with %% (~ chapters) Align

Read a raw dataset Dedicated matlab tools >> xlsread >> textscan >> csvread Dedicated OpenEarthTools www.waterbase.nl >> donar_read http://www.knmi.nl/klimatologie/onderzoeksgegevens/potentiele_wind/ >> KNMI knmi_potwind http://www.knmi.nl/klimatologie/daggegevens/download.html >> knmi_etmgeg www.NODC.nl / SeaDataNet profile data (CTD): >> odvread Dedicate yourselves by peeling the file & add to OpenEarthTools: >> oetnewfun, load_template >> fgetl, fscanf % ascii >> fread % binary

Read a raw data waterbase

Read a raw data KNMI wind

Read a raw data KNMI meteo

Read a raw data NODC ODV

Test with some raw data Test for yourselve some raw data with dedicated scripts, choose one. Find the associated matlab function using the help documentation. Rijkswaterstaat www.waterbase.nl KNMI http://www.knmi.nl/klimatologie/onderzoeksgegevens/potentiele_wind/ http://www.knmi.nl/klimatologie/daggegevens/download.html NODC www.NODC.nl / SeaDataNet profile data (CTD):

Inspect a raw dataset raw data file Different types of raw data Different structs for time series. Hmm. Waterbase KNMI wind KNMI meteo How to unify? No guidelines for structs International guidelines for netCDF! Idea: collaborate! Each BwN participant transforms raw data they use anyway into netCDF Everyone can readily access previously used data in a uniform manner No standards available raw data struct International standards available netCDF

netCDF on OPeNDAP server Contents Extract Transform Load Provide 2a use tools > 2b read data 2c > save as netCDF add meta information data netCDF on OPeNDAP server models transform to netCDF netCDF on web server tools 1 data providers = data users 3 D:\... 3 http://…

netCDF What is netCDF How to make netCDF Matlab netCDF interface Conventions (vocabularies): Standard meta-info: CF convention + GNU/CC/? standard quantities: CF convention Standard units: Unidata UD units Standard coordinate codes: EPSG Save script in repository OpenEarthRawData: for dedicated script OpenEarthTools: for more general script Inspect and plot netCDF

netCDF = linked dimensions, variables, attributes Self-describing file: meta-info never lost no manual needed open file format guarantees future access units standard_name comment fillvalue z = f(z) t =f(t) standard_name standard_name long_name long_name units units # = f(#) long_name t units z chlorophyll = f(t,z,#) standard_name long_name units title conventions # website institute disclaimer copyright

Let’s look at a basic netCDF file nc_cf_example2nc.nc (in OpenEarthTools checkout) >> nc_dump(‘nc_cf_example2nc.nc’) ncBrowse HDFview Hierarchical structure as file system Attributes Dimensions Variables

netCDF (1) Any amount of meta-data (Attributes) can be appended. Handles large datasets (file > 4GB allowed with in 64-bit offset mode). Scaleable for future applications (divide data over multiple files). Open source specifications and interfaces. Fast data storage and retrieval (direct access binary). Small data storage (binary is small + internal zip option). Simple yet robust. “Following examination by 20 reviewers, NASA's Earth Science Data Systems Standards Process Group has concluded that netCDF classic should be adopted as a recommended standard. A major strength of netCDF classic, according to the reviewers, was that it has fostered data interoperability and exchange through its self-describing file format, platform independent architecture, and robust access methods. Additionally, its overall file format and metadata attributes were simple enough to be easily understood and applied yet robust enough to describe and store multidimensional data of different types in the same file” (source: Unidata news)

netCDF (2) 2 independent implementations of NetCDF interface (C & Java): reliable redundant Generally used, so there’s always someone who can offer help available. netCDF is de facto standard for coastal ocean circulation models (GOTM, ROMS, ECOM-SED and in the near future also Delft3D-FLOW). Lots of simple and free tools on the web that operate on NetCDF files: ncBrowse (NOAA) NCO Panoply (NASA) ncview (linux) Implemented in numerous commercial tools too Quickplot (Delft3D)

netCDF (3) Interfaces to sophisticated scientific data manipulation languages: C-interface Java-interface Fortran Python IDL Matlab Through java-interface (in OET) mexcdf for C-interface (in OET) Native matlab interface (since release 2008b) Merged with OPeNDAP-interface Handle netCDF file on web as if it on your own PC. Additional, widely accepted conventions for NetCDF implemented in main NetCDF viewers CF meta data convention

Creating a netCDF file See : nc_cf_example2nc.m Now step by step walking through Use nc_cf_example2nc.nc as template when doing it yourselves

>> nc_create_empty % 1 units standard_name comment fillvalue z = f(z) t =f(t) standard_name standard_name long_name long_name units units # = f(#) long_name t units z chlorophyll = f(t,z,#) standard_name long_name units title conventions # website institute disclaimer copyright

>> nc_create_empty % 0 Web-sources (waterbase, knmi) Read directly from web Or download to local cache Your own data: Save raw data in repository OpenEarthRawData with INSPIRE XML meta-data Then read data in Matlab and create empty netCDF file

>> nc_attput % 1 z = f(z) t =f(t) # = f(#) t z units standard_name comment fillvalue z = f(z) t =f(t) standard_name standard_name long_name long_name units units # = f(#) long_name t units z chlorophyll = f(t,z,#) standard_name long_name units title conventions # website institute disclaimer copyright

>> nc_attput % 1 See CF meta-data conventions: http://cf-pcmdi.llnl.gov/ OpenEarth conventions: disclaimer + terms_for_use $Headurl$

>> nc_add_dimension % 2 units standard_name comment fillvalue z = f(z) t =f(t) standard_name standard_name long_name long_name units units # = f(#) long_name t units z chlorophyll = f(t,z,#) standard_name long_name units title conventions # website institute disclaimer copyright

>> nc_add_dimension % 2

>> nc_addvar % 3 z = f(z) t =f(t) # = f(#) t z units standard_name comment fillvalue z = f(z) t =f(t) standard_name standard_name long_name long_name units units # = f(#) long_name t units z chlorophyll = f(t,z,#) standard_name long_name units title conventions # website institute disclaimer copyright

>> nc_addvar % 3

>> nc_varput % 4 CF convention vocabulary of quantities

>> nc_varput % 4 UNIDATA UD units package vocabulary of units

>> nc_varput % 4 z = f(z) t =f(t) # = f(#) t z units standard_name comment fillvalue z = f(z) t =f(t) standard_name standard_name long_name long_name units units # = f(#) long_name t units z chlorophyll = f(t,z,#) standard_name long_name units title conventions # website institute disclaimer copyright

>> nc_varput % 4

>> nc_dump % 5 z = f(z) t =f(t) # = f(#) t z units standard_name comment fillvalue z = f(z) t =f(t) standard_name standard_name long_name long_name units units # = f(#) long_name t units z chlorophyll = f(t,z,#) standard_name long_name units title conventions # website institute disclaimer copyright

>> nc_dump % 5

>> nc_dump % 5

Plot netCDF in free ncBrowse

Plot netCDF in free ncBrowse

Plot netCDF in Matlab

Plot netCDF in ArcGis ArcGIS ArcGis also reads and writes netcdf files. ArcGIS

Check netCDF in your own text editor (xml) xml representation of a netcdf file XML

Check netCDF in … IDV NCO JAVA netCDF tools Not so stable. Very useful #diff ncdiff -v time file1.nc file2.nc #compression & packingncpdq -4 -L 9 in.nc out.nc # Deflated packing (~80% lossy compression) #selecting variables by regex ncks -v '^Q..' in.nc # Q01--Q99, QAA--QZZ, etc. Useful JAVA netCDF tools Web hyperslabs, cool!

Summary current session Extract Transform Load Provide 2a use tools > 2b read data 2c > save as netCDF add meta information data netCDF on OPeNDAP server models transform to netCDF netCDF on web server tools 1 data providers = data users 3 D:\... 3 http://…

Next session: using netCDF from others Extract Transform Load Provide add meta information data netCDF on OPeNDAP server models transform to netCDF netCDF on web server tools 2 1 data providers = data users 3 D:\... 3 http://…

Read a raw data very large example WATERBASE returns 400 Mb ASCII files for water levels Has incomplete meta-info (no location code) Mixture of Amersfoort, (lat,lon), UTM31 coordinates

Read a raw data example Get ASCII wind file from KNMI website Read ASCII wind files as if they were one Transform to Delft3D standards (time in minutes): W2 = f(W) Write as Delft3D *.wnd file