1. Data Standards Workflow Raw dataScriptsDatabase Store raw data in subversion to keep track of history Stored files (netcdf) accessible through the web Extract Transform Load Charts & Maps Tools and websites Provide Add meta information Script to convert raw data into netcdf OpenEarth RawData OpenEarth OPeNDAP OpenEarth Tools

2. Data Standards Workflow Raw dataScriptsDatabase Store raw data in subversion to keep track of history Stored files (netcdf) accessible through the web Extract Transform Load Charts & Maps Tools and websites Provide Add meta information Script to convert raw data into netcdf OpenEarth RawData OpenEarth OPeNDAP OpenEarth Tools

3. Transform Add metadata Store in netcdf Save script in subversion

4. Add metadata Use the inspire meta data form to store information about the dataset. http://www.inspire-geoportal.eu/inspireEditor.htm Click launch editor Transform

5. Turn validation on Transform – add metadata validation

6. Location in subversion micore File identification Transform – add metadata

7. History of your data. Transform – add metadata quality

8. Please fill in limitations of use. Transform – add metadata constraints

9. Store in course/Pcnumber/inspire_description.xml Transform – add metadata Save metadata file 1.Save metadata file (local) 2.Add to subversion (local) 3.Commit => metadata into subversion (remote)

10. Transform Add metadata Store in netcdf Save script in subversion

11. Store in netcdf What’s netcdf? Write a script to transform data into netcdf Using CF convention Transform

12. What is netcdf Data format defined by unidata Data store used for coverage data and multidimensional data CF Metadata convention Transform – store in netcdf - netcdf

13. What is netcdf X Z T Y An array based data structure for storing multidimensional data N-dimensional coordinates systems X coordinate (e.g. longitude) Y coordinate (e.g. latitude) Z coordinate (e.g. altitude) Time dimension … other dimensions Variables – support for multiple variables Temperature, humidity, pressure, salinity, etc Geometry – implicit or explicit Regular grid (implicit) Irregular grid Points TransformTransform – store in netcdf - netcdf

14. Storing Multidimensional Data XYZQ 1110.5 1120.3 1210.6 1220.1 2110.4 2120.2 2210.9 2220.3 0.50.4 0.60.9 0.30.2 0.10.3 12 1 2 1 2 X YZ 32 numbers 14 numbers Transform – store in netcdf - netcdf

15. Data Model Data model for netcdf and others. Also usable for hdf, opendap, grib, etc. See the java library for details Data model for netcdf and others. Also usable for hdf, opendap, grib, etc. See the java library for details Transform – store in netcdf - netcdf

16. ArcGis ArcGis also reads and writes netcdf files. Transform – store in netcdf – netcdf - applications

17. Your favorite text editor xml representation of a netcdf file Transform – store in netcdf - netcdf

18. Other Tools NCO #diff ncdiff -v time file1.nc file2.nc #compression & packingncpdq -4 -L 9 in.nc out.nc # Deflated packing (~80% lossy compression) #selecting variables by regex ncks -v '^Q..' in.nc # Q01--Q99, QAA--QZZ, etc. IDV Very useful Web hyperslabs, cool! Not so stable. Transform – store in netcdf - netcdf

19. Data Standards Workflow Raw dataScriptsDatabase Store raw data in subversion to keep track of history Stored files (netcdf) accessible through the web Extract Transform Load Charts & Maps Tools and websites Provide Add meta information Script to convert raw data into netcdf OpenEarth RawData OpenEarth OPeNDAP OpenEarth Tools

20. Store in netcdf What’s netcdf? Write a script to transform data into netcdf Using CF convention Transform – store in netcdf - script

21. Write script Read raw data Read header line Read data Read all data Create function to read all data Use function in Matlab Raw data into empty netcdf file Create empty netcdf file Add dimensions and variables Store variables Read values Transform – store in netcdf - script

22. Reading raw data into memory Use one of the following matlab functions to read the file data into an array fscanf Transform – store in netcdf - script

23. Example: Transect.txt file 1999 58 -135 3531 -130 3541 -125 3631 -120 4171 -115 6221 -110 8231 -105 9841 -100 10971 -95 12171 -90 12951 … 200 -2415 210 -2995 220 -3595 99999999999 99999999999 2000 58 -135 3531 -130 3541 -125 3631 -120 4171 -115 6221 -110 8231 -105 9841 -100 10971 -95 12171 -90 12951 Header line Year number of points Points X Z X Z …. 9999999 Location: OpenEarthRawData\course\example\raw Transform – store in netcdf - script

24. Read header line >> fid = fopen('..\raw\transect.txt') fid = 15 >> header = fscanf(fid, '%d', 2) header = 2000 58 >> year = header(1) year = 2000 >> npoint = header(2) npoint = 58 Transform – store in netcdf - script

25. % read header header = fscanf(fid, '%d', 2); year = header(1); % store year in time time(i) = year; npoint = header(2); % read data data = fscanf(fid, '%d', npoint*2); data = reshape(data, [2, npoint]); % use column vectors data = data'; Read data >> % read data data = fscanf(fid, '%d', npoint*2) data = -150 3741 -140 3581 -135 >> data = reshape(data, [2, npoint]) data = Columns 1 through 7 -150 -140 -135 -130 3741 3581 3531 3541 1 2 >> % use column vectors data = data' data = -150 3741 -140 3581 -135 3531 3 Transform – store in netcdf - script

26. Read all data % preallocate all data % (time, coastward) transectseries = NaN(3, 58); coastward_distance = NaN(58, 1); time = NaN(3, 1); % open file and get file id fid = fopen('..\raw\transect.txt'); i = 1; while (~feof(fid)) % read header header = fscanf(fid, '%d', 2); year = header(1); % store year in time time(i) = year; npoint = header(2); % read data data = fscanf(fid, '%d', npoint*2); data = reshape(data, [2, npoint]); % use column vectors data = data' % store data in transect series transectseries(i,:) = data(:,2); coastward_distance(:) = data(:,1); fgetl(fid); i = i + 1; end Transform – store in netcdf - script

27. Create a function function transect = readtransect(filename) % preallocate all data % (time, coastward) transectseries = NaN(3, 58); coastward_distance = NaN(58, 1); time = NaN(3, 1); % open file and get file id fid = fopen(filename); i = 1; while (~feof(fid)) % read header header = fscanf(fid, '%d', 2); year = header(1); % store year in time time(i) = year; npoint = header(2); % read data data = fscanf(fid, '%d', npoint*2); data = reshape(data, [2, npoint]); % use column vectors data = data'; % store data in transect series transectseries(i,:) = data(:,2); coastward_distance(:) = data(:,1); fgetl(fid); i = i + 1; end transect = struct('series', transectseries, … 'distance', coastward_distance, 'time', time); end Transform – store in netcdf - script

28. Use the new function >> data = readtransect('..\raw\transect.txt') data = series: [3x58 double] distance: [58x1 double] time: [3x1 double] Transform – store in netcdf - script

29. Loading data into netcdf What does a netcdf file look like Required meta information Transform – store in netcdf - script

30. Netcdf file transect.nc netcdf transect { dimensions: coastward = 58 ; time = 3 ; variables: float coastward_distance(coastward) ; coastward_distance:unit = "metre" ; float year(time) ; year:unit = "year" ; float height(time, coastward) ; height:unit = "metre" ; data: coastward_distance = -135, -130,…, 150, 160, 170, 180, 190, 200, 210, 220 ; year = 1999, 2000, 2001 ; height = 353, 354, … -142, -146, -170, -206, -232, -273, -309, -346, -375, -388, … -32, … -92, -110, -127, -143, -156, -177, -211, -259, -303, -334 ; } Transform – store in netcdf - script

31. Create an empty netcdf file >> nc_create_empty(outputfile) >> nc_dump(outputfile) netcdf transect.nc { dimensions: variables: } Transform – store in netcdf - script

32. Add dimensions nc_add_dimension(outputfile, 'crossshore', 58) nc_add_dimension(outputfile, 'time', 3) nc_dump(outputfile) >> netcdf transect.nc { dimensions: coastward = 58 ; time = 3 ; variables: } help nc_add_dimension Transform – store in netcdf - script

33. Add variables crossshoreVariable = struct(... 'Name', 'crossshore_distance',... 'Nctype', 'float',... 'Dimension', {{‘crossshore'}},... 'Attribute', struct('Name', 'unit', 'Value', 'metre')... ); nc_addvar(outputfile, crossshoreVariable); timeVariable = struct(... 'Name', 'year',... 'Nctype', 'float',... 'Dimension', {{'time'}},... 'Attribute', struct('Name', 'unit', 'Value', 'year')... ); nc_addvar(outputfile, timeVariable); heightVariable = struct(... 'Name', 'height',... 'Nctype', 'float',... 'Dimension', {{'time', ‘crossshore'}},... 'Attribute', struct('Name', 'unit', 'Value', 'metre')... ); nc_addvar(outputfile, heightVariable); nc_dump(outputfile) help nc_addvar Transform – store in netcdf - script

34. Result netcdf transect.nc { dimensions: coastward = 58 ; time = 3 ; variables: float coastward_distance(coastward), shape = [58] coastward_distance:unit = "metre" float year(time), shape = [3] year:unit = "year" float height(time,coastward), shape = [3 58] height:unit = "metre" } Transform – store in netcdf - script

35. Store variables nc_varput(outputfile, 'height', data.series) nc_varput(outputfile, 'year', data.time) nc_varput(outputfile, 'coastward_distance', data.distance) help nc_varput Transform – store in netcdf - script

36. Result: Netcdf file transect.nc netcdf transect { dimensions: coastward = 58 ; time = 3 ; variables: float coastward_distance(coastward) ; coastward_distance:unit = "metre" ; float year(time) ; year:unit = "year" ; float height(time, coastward) ; height:unit = "metre" ; data: coastward_distance = -135, -130,…, 150, 160, 170, 180, 190, 200, 210, 220 ; year = 1999, 2000, 2001 ; height = 353, 354, … -142, -146, -170, -206, -232, -273, -309, -346, -375, -388, … -32, … -92, -110, -127, -143, -156, -177, -211, -259, -303, -334 ; } Transform – store in netcdf - script

37. Read values surface(nc_varget(outputfile, 'height')') Transform – store in netcdf - script

38. Store in netcdf What’s netcdf? Write a script to transform data into netcdf Using CF convention Transform – store in netcdf - convention

39. CF convention Standard used by USGS, NOAA, Arcgis, GDAL Climate and Forecast (CF) Convention http://www.unidata.ucar.edu/software/netcdf/docs/conventions.html Initially developed for Climate and forecast data Atmosphere, surface and ocean model-generated data Also used for observational datasets CF is the most widely used convention for geospatial netCDF data. Transform – store in netcdf - convention

40. Improve output Store extra attributes Title Author Standard_name Transform – store in netcdf - convention

41. Transform Add metadata Store in netcdf Save script in subversion

42. Transform – save script Save script 1.Save script (local, using matlab https://repos.deltares.nl/repos/OpenEarthRawData/course/PCnr/scipts/) 2.Add to subversion (local) 3.Commit => script into subversion (remote)