Model performance & Evaluation Calibration and Validation

Calibration and validation Prepared by: Ilyas Masih and Marloes Mul UNESCO-IHE, The Netherlands

Evaluation of model performance The process of assessing the performance of a model requires the subjective and/or objective estimates of the “closeness” of the simulated behaviour of the model to observations. Qualitative or subjective assessment: to assess the systematic (e.g., over- or under prediction) and dynamic (e.g., timing, rising limb, falling limb, and base flow) behaviour of the model. Objective assessment: requires the use of a mathematical estimate of the error between the simulated and observed hydrologic variable(s) – i.e. objective or efficiency criteria.

Evaluation of model performance The confidence in use of a model (for reasoning and the formulation of testable predictions) will depend on how ‘close’ the model is to reality. degree of accuracy (bias), degree of precision (uncertainty) and degree of correspondence (‘sameness’ of the quantities in question) in the comparative evaluation of both fields (observation vs simulation). Calibration is a process of changing the model input values (usually parameters) to match as closely as possible the simulated behaviour of the model to the observation.

Schematic of model evaluation and Calibration process Vrugt (2004)

Calibration Improving the model results by Changing parameters For when the understanding of the processes is insufficient For when the information is insufficient to determine the parameters

Parameter Obtained by measuring Derived from observations (such as the K-value) Derived by calibration

Commonly used variables against which water systems models are evaluated and calibrated Water balance components: e.g. streamflows, groundwater levels, actual evapotranspiration, soil moisture Water quality: e.g. sediment, Nitrogen, Phosphorous Water allocation: demand and supply Other? (e.g. reservoir volume/level and releases)

Commonly used Objective functions (criteria) in water systems models’ evaluation and calibration Bias Mean square Error; root mean square error, relative root mean square error Coefficient of determination Nash-Sutcliffe efficiency Other? (e.g. comparison of the slopes of the observed and simulated flow duration curves)

Objective functions: example Nash-Stucliffe efficiency (NSE), coefficient of determination (R2) and Bias(error) (%)

Calibration approaches: Manual calibration Most widely used calibration method Visual comparison of measured and simulated data Semi-intuìtive trial and error process for parameter adjustment Excellent model calibrations, but manual calibration... is highly labor-intensive (human resources) is difficult to learn procedures are model-dependent results are user-dependent

Calibration approaches: Automatic calibration The procedure that optimizes an objective function by systematically searching the parameter space according to a fixed set of rules Example Monte Carlo method Genetic Algorithms Shuffled Complex Evolution

Calibration approaches: Manual VS Automatic calibration “The debate on whether manual or automatic calibration of hydrological models is superior is likely to remain inconclusive. However, better understanding of the catchment is the key for the success of both approaches.” Masih, I., 2011.

A note of caution on the use of objective functions (from Krause et al “In general, many efficiency criteria contain a summation of the error term (difference between the simulated and the observed variable at each time step) normalized by a measure of the variability in the observations. To avoid the canceling of errors of opposite sign, the summation of the absolute or squared errors is often used for many efficiency criteria. As a result, an emphasis is placed on larger errors while smaller errors tend to be neglected. Since errors associated with high streamflow values tend to be larger than those associated with errors for lower values, calibration (both manual and automatic) attempts aimed at minimizing these types of criteria often lead to fitting the higher portions of the hydrograph (e.g., peak flows) at the expense of the lower portions (e.g., baseflow). Further, different efficiency criterion may place emphasis on different systematic and/or dynamic behavioural errors making it difficult for a hydrologist to clearly assess model performance.”

A note of caution on the use of objective functions Every objective functions has strengths and weakness What we can do to overcome these limitations Select appropriate objective function according to the study objectives Use more than one objective function (Multi-objective calibration) Use more than one variable in calibration (e.g. ET and Q) Calibrate at more than one observation point (e.g. more flow gauges within a basin than only at one station) What else?

Examples of model calibration and evaluation: Masih et al Examples of model calibration and evaluation: Masih et al., 2010-HBV model application, Iran

Examples of model calibration and evaluation: Masih et al Examples of model calibration and evaluation: Masih et al., 2010-HBV model application, Iran Calibration (Oct. 1988-Sep. 1994) R2= 0.91; NSE = 0.91; Bias= -5% Observed (blue) Simulate (red)

Some examples: Model calibration and evaluation Example from Masih et al., 2011-SWAT model application: What do you observe by visually inspecting these graphs Which one is good or bad? What could be the reason? Can we improve the fit through calibration process or this is what we can achieve at best?

HYPE model application in Swedon, Lindsrom et al., 2010 Figure 5 | Examples of local model evaluations from different river basins in Sweden (gray: simulation and black: observations). The model was optimized to the observations in each particular diagram. S: snow depth at Ljusnedal (R 2 = 0.92); E: evaporation at Norunda (R 2 = 0.59; Grelle 2008); G: groundwater level at Svartberget (R 2 = 0.90); F: soil frost depth at Svartberget (R 2 = 0.84, Lindstro¨m et al. 2002); W: lake water level in Lake Mo¨ ckeln (R 2 = 0.95); Q: discharge at Torsebro (R 2 = 0.94); O18: 18O content at Stubbetorp (R 2 = 0.54, Andersson & Lepisto¨ 1998); TN: total nitrogen at JRK research basin O19 (R 2 = 0.68) and TP: total phosphorous at JRK research basin AB5 (R 2 = 0.41).

Validation Model should be able to reproduce field observations not used in calibration Can be carried out as long as an independent data set has been kept aside

Validation Ensuring theories and assumptions are correct Ensuring computer programming code is correct Water balance calculation Parameters are valid values Compare to second data series of observed data

Examples model validation: Masih et al Examples model validation: Masih et al., 2010-HBV model application, Iran. Same parameter set used for the validation as of calibration Validation (Oct.1994-Sep. 2001) R2= 0.81; NSE = 0.67; Bias= 20% Observed (blue) Simulate (red)

Examples model validation: Merz and Bloschl, 2004-HBV model application to 308 Austrian catchments

Model Evaluation: Calibration and validation If simulations are good in both calibration and validation period, it gives an indication of the possibility of a model to be confidently used for the intended purpose Most often the model performance during validation period is lower than calibration, but could be other way around as well. Modeler should try to find reasons for high/low performance Sensitivity and uncertainty analysis are also integral part of calibration and validation process, and should be properly included in a modelling study.

Discussion and Questions