1. November 2010Deltares Configuration Course1 Day 2

2. November 2010Deltares Configuration Course2 TimeSeriesSets

3. November 2010Deltares Configuration Course3 FEWS Time Series Time series are available from two source:  external  simulated Time series are in two categories in relation to time:  historical (continuous in time)  forecasting (characterised by its start time) Time series can be in four formats:  0D – scalar  1D – vector or longitudinalprofile  2D – grid  2D – polygon Time Series are handled in the form of Time Series Sets

4. November 2010Deltares Configuration Course4 FEWS Time Series – external historical 00:00 04:00 07:00 10:00 Complete series. Added incrementally. Can be edited by the user. Only new and changed values are stored Complete series Manual edit Time of import

5. November 2010Deltares Configuration Course5 FEWS Time Series – external forecasting Time of Forecast. Added and stored individually. Can be edited by the user. Usage of series depends on T0 00:00 04:00 08:00 Series at 08:00 Series at 06:00 00:00 04:00 08:00 06:00

6. November 2010Deltares Configuration Course6 FEWS Time Series – simulated historical System Time. Continuous in time. Referenced by the forecast model that produced them. Approved series shown automatically (including history). Most often linked by model states 04:00 08:00 12:00 Series at 16:00 Series at 08:00 00:00 04:00 08:00 12:00 16:00

7. November 2010Deltares Configuration Course7 FEWS Time Series – simulated forecasting System Time. Continuous in time (in combination with simulated historical). Referenced by the forecast model that produced them. Approved and selected series can be viewed 04:00 08:00 12:00 Series at 12:00 00:00 04:00 08:00 12:00 16:00

8. November 2010Deltares Configuration Course8 FEWS Time Series – temporary temporary time series  time series used in pre- / post-processing  not stored in database

9. November 2010Deltares Configuration Course9 Time Series Sets Properties of time series: location Id – or – locationSetId parameter Id timestep relativeViewPeriod valueType (scalar, grid, profile,..) timeSeriesType (external historical, external forecast,..) moduleInstanceId (import, PDM, KW, ISIS,..) readWriteMode expiryTime (when to delete from database) synchLevel simple transformations (delay, multiplier, divider, incrementer) ensembleId

10. November 2010Deltares Configuration Course10 Time Series Sets All data handling is done through Time Series Sets Some elements are only used when retreiving time series, other elements only when storing data

11. November 2010Deltares Configuration Course11 Time Series Sets - Examples Import Run General Adapter Run

12. November 2010Deltares Configuration Course12 FEWS Workflows and Modules

13. November 2010Deltares Configuration Course13 FEWS Workflows Workflows are logical sequences of running forecast modules Workflows must be registered in the workflow descriptors – Module Instance ID – Sub-workflows – Run independent – Fallback activity – Ensemble

14. November 2010Deltares Configuration Course14 FEWS Modules and Module Instances. Modules  are FEWS plug-ins that can be used in a workflow  are registered in moduleDescriptor file as Java Classes  are part of the system configuration. Module Instances  are configured modules of a particular module  are registered in moduleInstanceDescriptors file  are part of the regional configuration

15. November 2010Deltares Configuration Course15 Sub workflows Workflows can be nested: Workflow “DO_ALL” – (sub)Workflow “DO_Step1” – (sub)Workflow “DO_Step2” Where: (sub)Workflow “DO_Step1” – ModuleInstance “DO_Step1a” – ModuleInstance “DO_Step1b” – ModuleInstance “DO_Step1c” (sub)Workflow “DO_Step2” – ModuleInstance “DO_Step2a” – ModuleInstance “DO_Step2b” – ModuleInstance “DO_Step2c” FEWS Explorer:. Manual Forecast Display  Workflow “DO_ALL”  (sub)Workflow “DO_Step1”  (sub)Workflow “DO_Step2”

16. November 2010Deltares Configuration Course16 Introduction to some available modules. Import. Export. Transformation. General Adapter. Lookup. ErrorModel. Correlation. PerformanceIndicator. ThresholdEvent. Report

17. November 2010Deltares Configuration Course17 FEWS Workflows FEWS Workflows can be run (and debugged) on module basis (via F12)

18. November 2010Deltares Configuration Course18 Importing data

19. November 2010Deltares Configuration Course19 Flood Forecasting processing steps Create evaporation profiles for hydrological model Calculate catchment average precipitation based on rain gauge data Import data from external sources Combine precipitation sources (e.g. Rain gauge, Radar data, Meteorological Forecast models) Run hydrological model to calculate discharge for catchments Run hydrodynamic model to calculate water level for forecast points Prepare input for hydrodynamic model based on hydrological models, gauging stations, tidal boundaries Run error correction model to update output form hydrodynamic model

20. November 2010Deltares Configuration Course20 Importing data Import Modules allows data from external sources to be imported. Various Scalars – e.g. PI XML and Wiski. ASCII grid time series. Binary GRIB (grid) and BIL time series. Currently around 50 data import types FEWS imports files from a directory specified in the global.properties

21. November 2010Deltares Configuration Course21 Import Module Key Features:. ID mapping. Flag mapping. Units mapping. Time zone mapping. Tolerance. Validate

22. November 2010Deltares Configuration Course22 EA Import Module

23. November 2010Deltares Configuration Course23 Import Module – ID mapping Matching ‘outside world’ to the ‘FEWS world’ – locations (ids) – parameter (ids) Three levels – direct mapping  location names AND parameter names differ – indirect  either location names or parameter names differ – 1:1  location and parameter names do NOT differ Used when importing and exporting time series

24. November 2010Deltares Configuration Course24 Id Mapping Direct Indirect 1:1

25. November 2010Deltares Configuration Course25 Import Module – Unit conversion Map internal units to external units Used when importing and exporting time series Extra option to convert datum of level recordings (external unit mAOD)

26. November 2010Deltares Configuration Course26 Import Module – Flag conversion Map internal quality flags to external quality flags Used when importing and exporting time series

27. November 2010Deltares Configuration Course27 Import Module – other elements Time zone mapping  database stores all data in GMT, specify timezone when importing Tolerance  when imported data is not exactly observed at the cardinal time step Validate  Imported files can be validated against their own schema

28. November 2010Deltares Configuration Course28 Import Module - Exercise Import Time series from EA-XML format (specific Environment Agency format) Analyse the 5 time series – 2 * Observed_waterlevel for Hydro1 and Hydro2 – 2 * Rainfall for Raingauge and Meteostation – 1 * Temperature for Meteostation Add elements to ID mapping table Run import workflow Analyse data with the Time Series Display

29. November 2010Deltares Configuration Course29 Validating Data Allow quality checking of all scalar time series data Difference in hard limit and soft limit checks – Hard limit checks mark data as unreliable – Soft limit checks mark data as doubtful

30. November 2010Deltares Configuration Course30 Validating Data Option to enter monthly limits Rule can be specified for one time series or for a complete set

31. November 2010Deltares Configuration Course31 Validating Data - Exercise Check the data just imported Define validation rules Enter the rules in the ValidationRuleSets XML file Delete the data store Run import workflow View data Check the difference!

32. November 2010Deltares Configuration Course32 Thresholds

33. November 2010Deltares Configuration Course33 Example (Map) Willington Cableway (Midlands, UK) ! T0

34. November 2010Deltares Configuration Course34 Example (Graph) Willington Cableway (Midlands, UK)

35. November 2010Deltares Configuration Course35 Thresholds Utility The thresholds utility checks when time series cross thresholds  Observed and simulated time series  Icons are shown on the map  Graphs/lines cross threshold lines  Events can be triggered (MC)  Up events and down events Three configuration files  thresholdWarningLevels.xml  definition of (unique) thresholds + details  thresholds.xml  the grouping of the (selected) thresholds  thresholdValueSets.xml  timeseries (loc/par.) and actual levels info

36. November 2010Deltares Configuration Course36 Thresholds - thresholdWarningLevels Unique threshold information – colors: displays, reports – icons normal/warning historic colour forecast colour !

37. November 2010Deltares Configuration Course37 Thresholds Utility - Thresholds Thresholds usage – groups – selection(s)

38. November 2010Deltares Configuration Course38 Thresholds Utility – Thresholds Values Sets Detailed threshold information – per location/parameter – actual values to be crossed for the different thresholds – ‘triggers’ (action events for up- or downcrossings)

39. November 2010Deltares Configuration Course39 Thresholds – Exercise Add thresholds for the hydro locations

40. November 2010Deltares Configuration Course40 Presentation of data in FEWS

41. November 2010Deltares Configuration Course41 Display of data - Filters

42. November 2010Deltares Configuration Course42 Display of data - Filters Filters are used to: – define the locations that are displayed on the map – define the locations that are displayed in the list box – define the parameters that are displayed in the list box

43. November 2010Deltares Configuration Course43 Display of data - Filters Filters organised in folders with child filters arrange time series sets in logical groups import elements in the time series set – extra options view permissions validation icons visible link to map extent

44. November 2010Deltares Configuration Course44 Display of data - Filters Allow/Disallow editing Relative View Period for Icon on main map synchLevel – for live system (5=edited data)

45. November 2010Deltares Configuration Course45 Display of data - Filters child filter organised in sub-folders (childs)

46. November 2010Deltares Configuration Course46 Time Series Display Time Series Display is used to show data in a graph and in a table Editing and viewing data Pre-configured displays Thresholds Historic Events

47. November 2010Deltares Configuration Course47 Time Series Display Time Series Display uses 4 configuration files – TimeSeriesDisplayConfig – Thresholds and ThresholdValueSets – DisplayGroups – HistoricalEvents

48. November 2010Deltares Configuration Course48 Time Series Display - TimeSeriesDisplayConfig default view period time markers parameters module instance mappings

49. November 2010Deltares Configuration Course49 Time Series Display - DisplayGroups Display groups are pre-defined displays are used to create folder structure of displays can be nested

50. November 2010Deltares Configuration Course50 Time Series Display - DisplayGroups display element contains time series to display multiple sub plots for each subplot – time series Options – minimum – maximum – convertDatum – relativeViewPeriod – nrOfRecentForecasts

51. November 2010Deltares Configuration Course51 Time Series Display - Exercise Exercise: Add pre-configured displays Add a new display group for the new locations Add a new display for the hydro locations Add a new display for the meteo locations

52. November 2010Deltares Configuration Course52 FEWS Data manipulation modules

53. November 2010Deltares Configuration Course53 Flood Forecasting processing steps Create evaporation profiles for hydrological model Calculate catchment average precipitation based on rain gauge data Import data from external sources Combine precipitation sources (e.g. Rain gauge, Radar data, Meteorological Forecast models) Run hydrological model to calculate discharge for catchments Run hydrodynamic model to calculate water level for forecast points Prepare input for hydrodynamic model based on hydrological models, gauging stations, tidal boundaries Run error correction model to update output form hydrodynamic model

54. November 2010Deltares Configuration Course54 Interpolation/Transformation Module Interpolation Module was separate, but now integrated with Transformation Module The Interpolation module generates data at desired locations or at desired points in time by means of Serial Interpolation methods  linear interpolation  block interpolation  default value  extrapolation Spatial Interpolation methods  Kriging  Inverse Distance  bi-linear interpolation  grid cell averaging over catchments

55. November 2010Deltares Configuration Course55 Interpolation Module – serial interpolation

56. November 2010Deltares Configuration Course56 Interpolation Module – Exercise Analyse module that fills gaps in the water level time series Add the new module instance to the module instance descriptors Configure a workflow that executes the interpolation module instance Check results!

57. November 2010Deltares Configuration Course57 Interpolation Possible problems / issues of note Time Series Set to Interpolate  ModuleInstanceId of series!!! (ImportTelemetry)  Relative View Period When interpolating to fill gaps  Input time series same as output time series Using LocationSets and/or Locations  Linear interpolation – can be done using a locationSet  Default – take care as default value may vary per location

58. November 2010Deltares Configuration Course58 Transformation Module. Concepts of transformations function. User defined function. Rating curve. Catchment averaging. Data Hierarchy. Typical Profiles. Aggregation / dis-aggregation. Equidistant to non-equidistant. Exercise: Adding rating curves

59. November 2010Deltares Configuration Course59 Transformation - Configuration

60. November 2010Deltares Configuration Course60 Transformation – Input/Output Inputs and Outputs are defined as TimeSeriesSet4

61. November 2010Deltares Configuration Course61 Transformation User defined functions and rating curves Different function / parameters per segment / range Define ranges in rating curves file / coefficientsets file

62. November 2010Deltares Configuration Course62 Transformation – User defined function Schema

63. November 2010Deltares Configuration Course63 Transformation – User defined function Example

64. November 2010Deltares Configuration Course64 Transformation – User defined function Functions Generic function parser used “Any” function can be defined

65. November 2010Deltares Configuration Course65 Transformation – Rating curve. Same approach as other transformations using stageDischarge class. Convention on inputId / ouputId & parameter names. Rating curves are contained in the CoefficientSetFiles (RatingCurves.xml)

66. November 2010Deltares Configuration Course66 Transformation – stageDischarge (Rating curve) Rating curves are linked to locations One location or locationSet Use coefficients from CoefficientSetFile

67. November 2010Deltares Configuration Course67 Transformation – stageDischarge (Rating curve) StageDischargeCoefficientSets.xml (name not fixed) In CoefficientSetsFiles folder Referenced in Transformation ModuleInstance Rating curve defined only once Referenced by locationId

68. November 2010Deltares Configuration Course68 Transformation – Weighted averaging  Convention on inputId / ouputId & parameter names  formula: (X1 * a + X2 * b + X3 * c) / (a + b + c)  number of locations is flexible, missing locations will be left out

69. November 2010Deltares Configuration Course69 Data Hierarchy Fallback options to complete a series used first used second used third

70. November 2010Deltares Configuration Course70 Transformation – Data Hierarchy / Merge used first used second result

71. November 2010Deltares Configuration Course71 Transformation – Typical profile Examples of typical profile use:  Evaporation series (one value per calendar day)  Temperature variation across day

72. November 2010Deltares Configuration Course72 Transformation – Typical profile Fixed data/time

73. November 2010Deltares Configuration Course73 Transformation Module - Exercise Add rating curve for location Hydro1 Analyse transformation file  register module instance in descriptors file  add module instance to the exercise workflow

74. November 2010Deltares Configuration Course74 FEWS External Modules/Models

75. November 2010Deltares Configuration Course75 Flood Forecasting processing steps Create evaporation profiles for hydrological model Calculate catchment average precipitation based on rain gauge data Import data from external sources Combine precipitation sources (e.g. Rain gauge, Radar data, Meteorological Forecast models) Run hydrological model to calculate discharge for catchments Run hydrodynamic model to calculate water level for forecast points Prepare input for hydrodynamic model based on hydrological models, gauging stations, tidal boundaries Run error correction model to update output form hydrodynamic model

76. November 2010Deltares Configuration Course76 FEWS External Modules Available Model Adapters  CEH Adapters (SNOWP, SNOW, PDM, KW, ARMA, TCM)  HR (ISIS)  PlanB Adapters (TRITON & PRTF)  DHI Adapters (Mike11, NAM)  Midlands Region (DODO, MCRM)  Southern Region (STF)  Northwest Region (NW TF – Common Adapter)  Wales (SW Overtopping module - Common Adapter)  SouthWest (Bruton/Holbeam Dam module – Common Adapter)  Delft (SOBEK, RIBASIM, HYMOS, Sacramento)  SMHI (HBV)  University of Karlsruhe (PRMS)  JRC (Lisflood - PCRaster) http://public.wldelft.nl/display/FEWSDOC/Models+linked+to+Delft-Fews

77. November 2010Deltares Configuration Course77 General Adapter  General adapter concepts  State handling  ID mapping

78. November 2010Deltares Configuration Course78 General Adapter -Concepts Delft-FEWS General Adapter Module Adapter Module module input data module output data published interface input data (pi xml) Pre Post output data (pi xml)

79. November 2010Deltares Configuration Course79 General Adapter – Configuration From global.properties General Adapter keywords DELFT-FEWS reserved keyword

80. November 2010Deltares Configuration Course80 General Adapter – Export Activities Export Activities Time Series – scalar Time Series – grids (map stacks) Time Series – profile States ModuleDataSet ModuleParameters

81. November 2010Deltares Configuration Course81 Export Activities – Scalar/longitudinal time Series. Exports time series from database to a PI-XML file. Assign File name to export data to. Assign Time Series Sets to export. Written to exportDir. LocationID’s & ParameterID’s will be translated on export (as indicated in IdMapping)

82. November 2010Deltares Configuration Course82 Export activities - grid time series. Same principle as exporting scalar time series. Define name of export file. Export file contains reference to grid. Data exported as a “mapstack” Files: pcr00000.000 pcr00000.001.. pcr00000.023 pcr00000.024

83. November 2010Deltares Configuration Course83 Export activities - grid time series Supported grid file formats ASC: ArcView/ARC-INFO grid interchange file  Time Series as map stack PCR: PCRaster file format  Time Series as map stack BIL: USGS standard grid format  Time series in one file

84. November 2010Deltares Configuration Course84 Export Activities – Module parameters. DELFT-FEWS administers module parameters. Exports to external module as PI-XML file. Held in Config\ModuleParFiles

85. November 2010Deltares Configuration Course85 Export Activities – Module dataset DELFT-FEWS administers module dataset  Dataset: native module file  Held in Config\ModuleDataSets by DELFT-FEWS as a ZIP file Exports to external module by extracting data in ZIP file Exported to a “root”directory

86. November 2010Deltares Configuration Course86 Exporting States DELFT-FEWS can manage module states What is a state?  Module initial conditions  Handled in “native”module format Type of state;  Cold State – default initial conditions  Warm State – initial conditions from a previous run For each module requiring management of states;  ZIP file of cold state MUST be available in ColdStates dir.

87. November 2010Deltares Configuration Course87 Exporting States Defining “how” to export state moduleInstanceId stateExportDir stateConfigFile stateLocations  read  write From the perspective of the model!!

88. November 2010Deltares Configuration Course88 Exporting States stateConfigFile For export: Written by DELFT-FEWS ; read by adapter For import:Written by adapter ; read by DELFT-FEWS

89. November 2010Deltares Configuration Course89 Exporting States. Defining “what” to state export; stateSelection. State selection determines length of module run!!!!!. Cold State: Always use a cold state to initiate module run. Warm State: Use most suitable/recent state in search period

90. November 2010Deltares Configuration Course90 04:0016:0004:0016:0004:00 Start 02:00 Jan 05. T0 16:00 04 Jan Start 7:30 and T0 07:00 Jan 05 Start 19:30 and T0 19:00 Jan 05 Start 07:30 and T0 07:00 Jan 06 Start 02:00 Jan 06. T0 16:00 05 Jan States 02:00 07:0019:0007:00 State Search period

91. November 2010Deltares Configuration Course91 General Adapter – Execute Activities. Running modules. Executables or Java classes. Arguments can be passed. Timout is set in case module “hangs”. If Diagnostic file indicates error – or non zero return code  Executable considered to have failed

92. November 2010Deltares Configuration Course92 General Adapter – Import Activities Import Time Series  Scalar  Longitudinal  Grid Import States

93. November 2010Deltares Configuration Course93 General Adapter – Import Activities. Importing time series. Same principle as exporting. File name & Time Series Set to save data to. IdMapping used to translate locationId’s and parameterId’s. Grids imported as mapstack + XML file

94. November 2010Deltares Configuration Course94 General Adapter – Import Activities. Importing states. Simple instruction where to find stateConfigFile

95. November 2010Deltares Configuration Course95 Burn-in profiles. Avoid “abrupt” shock to model on startup. Mainly relevant to HD modules (stability) Only applied when starting from a cold state  Identify start value in cold state  Gradual “climb” to actual value

96. November 2010Deltares Configuration Course96 Burn-in profiles Burn-in at Temple Sowerby

97. November 2010Deltares Configuration Course97 Startup/Shutdown activities Clean up files from module  Startup – before running module  Shutdown – after completing module

98. November 2010Deltares Configuration Course98 Module diagnostics/debugging PI-XML includes module diagnostics exchange  To be written by module/adapter  General Adapter Defines where diagnostics file is expected  Not all logging passed – use native module log files External module log inserted into NFFS log  Most messages DEBUG level  Error/Fatal: System messages Detecting module failure. Error in diagnostics log. non-zero return code

99. November 2010Deltares Configuration Course99 General Adaptor - Exercise Add unit hydrograph model

100. November 2010Deltares Configuration Course100 FEWS Gridded Data

101. November 2010Deltares Configuration Course101 FEWS Gridded Data Fews supports the display of both irregular and regular grids Available (common) Import types  ArcInfoAscii  BIL  BUFR  CSV  PI  HDF5  NetCDF

102. November 2010Deltares Configuration Course102 FEWS Gridded Data Commonly used data types  Precipitation  Soil Moisture  Pressure  Wind  Temperature  Wave height  Surge

103. November 2010Deltares Configuration Course103 FEWS Gridded Data Data can be manipulated using transformation and interpolation modules  Cookie cutter  Interpolation  Transformation  Data hierachy

104. November 2010Deltares Configuration Course104 FEWS Gridded Data

105. November 2010Deltares Configuration Course105 Gridded Data - Exercise Import Gridded Data