1. Eawag: Swiss Federal Institute of Aquatic Science and Technology Analyzing input and structural uncertainty of a hydrological model with stochastic, time-dependent parameters Peter Reichert Eawag Dübendorf and ETH Zürich, Switzerland

2. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Contents Motivation Approach Implementation Application Discussion  Motivation  Approach  Implementation  Application  Discussion

3. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Motivation Approach Implementation Application Discussion

4. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Motivation Approach Implementation Application Discussion  Environmental modelling is often based on deterministic models that describe substance and organism mass balances in environmental compartments.  Statistical inference with such models is often based on the assumption that the data is independently and identically distributed around the predictions of the deterministic model at „true“ parameter values.  The concept underlying this approach is that the deterministic model describes the „true“ system behaviour and the probability distributions centered at the model predictions the measurement process.

5. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Motivation Approach Implementation Application Discussion  Empirical evidence often demonstrates the invalidity of these statistical assumptions:  Residuals are often heteroscedastic and autocorrelated.  The residual error is usually (much) larger than the measurement error.  This leads to incorrect results of statistical inference. In particular, parameter and model output uncertainty are usually underestimated.  These obviously wrong results lead to abandoning of the statistical approach and to the development of conceptually poorer techniques in applied sciences.  We are interested in a statistically satisfying approach to this problem.

6. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Motivation Suggested solution (Kennedy and O‘Hagan, 2001, and many earlier, more case-specific approaches): Extend the model by a discrepancy or bias term. Replace: by: where y M = deterministic model, x = model inputs,  = model parameters,  y = observation error, B = bias or model discrepancy, Y M = random variable representing model results. Motivation Approach Implementation Application Discussion The bias term is usually formulated as a non-parametric statistical description of the model deficits (typically as a Gaussian stochastic process).

7. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Motivation Advantage of this approach: Statistical description of model discrepancy improves uncertainty analysis. Disadvantage: Lack of understanding of the cause of the discrepancy makes it still difficult to extrapolate. Motivation Approach Implementation Application Discussion We are interested in a technique that supports identification of the causes and reduction of these discrepancies.

8. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Motivation 1.Errors in deterministic model structure. 2.Errors in model input. 3.Inadequateness of a deterministic description of systems that contain intrinsic non-deterministic behaviour due to  influence factors not considered in the model,  model simplifications (e.g. aggregation, adaptation, etc.),  chaotic behaviour not represented by the model. Motivation Approach Implementation Application Discussion There are three generic causes of failure of the description of nature with a deterministic model plus measurement error:

9. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Motivation Pathway for improving models: 1.Reduce errors in deterministic model structure to improve average behaviour. 2.Add adequate stochasticity to the model structure to account for random influences. Motivation Approach Implementation Application Discussion This requires the combination of statistical analyses with scientific judgment. This talk is about support of this process by statistical techniques. Because of these deficits we cannot expect a deterministic model to describe nature appropriately.

10. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Approach Motivation Approach Implementation Application Discussion

11. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Approach Questions: 1.How to make a deterministic, continuous-time model stochastic? 2.How to distinguish between deterministic and stochastic model deficits? Motivation Approach Implementation Application Discussion  Replacement of differential equations (representing conservation laws) by stochastic differential equations can violate conservation laws and does not address the cause of stochasticity directly.  It seems to be conceptually more satisfying to replace model parameters (such as rate coefficients, etc.) by sto- chastic processes, as stochastic external influence factors usually affect rates and fluxes rather than states directly. The model consists then of an extended set of stochastic differential equations of which some have zero noise.

12. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Approach Motivation Approach Implementation Application Discussion

13. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Approach Note that the basic idea of this approach is very old. The original formulation was, however, limited to linear or weakly nonlinear, discrete-time systems with slowly varying driving forces (e.g. Beck 1987). The bias term approach is a special case of our approach that consists of an additive output parameter. Motivation Approach Implementation Application Discussion Our suggestion is to  extend this original approach to continuous-time and nonlinear models;  allow for rapidly varying external forces;  embed the procedure into an extended concept of statistical „bias-modelling“ techniques.

14. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Implementation Motivation Approach Implementation Application Discussion

15. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Model Deterministc model: Consideration of observation error: Motivation Approach Implementation Application Discussion

16. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Model Model with parameter i time-dependent: Motivation Approach Implementation Application Discussion

17. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Time Dependent Parameter This has the advantage that we can use the analytical solution: The time dependent parameter is modelled by a mean-reverting Ornstein Uhlenbeck process: or, after reparameterization: Motivation Approach Implementation Application Discussion

18. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Inference We combine the estimation of  constant model parameters,, with  state estimation of the time-dependent parameter(s),, and with  the estimation of (some of the) (constant) parameters of the Ornstein-Uhlenbeck process of the time dependent parameter(s),. Motivation Approach Implementation Application Discussion

19. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Inference Gibbs sampling for the three different types of parameters. Conditional distributions: Ornstein-Uhlenbeck process (cheap) simulation model (expensive) Ornstein-Uhlenbeck process (cheap) Motivation Approach Implementation Application Discussion Tomassini et al. 2007

20. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Inference Metropolis-Hastings sampling for each type of parameter: Multivariate normal jump distributions for the parameters  M and  P. This requires one simulation to be performed per suggested new value of  M. The discretized Ornstein-Uhlenbeck parameter,, is split into subintervals for which OU-process realizations conditional on initial and end points are sampled. This requires the number of subintervals simulations per complete new time series of . Motivation Approach Implementation Application Discussion Tomassini et al. 2007

21. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Application Motivation Approach Implementation Application Discussion Application

22. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Hydrological Model Simple Hydrological Watershed Model (1): Kuczera et al. 2006 Motivation Approach Implementation Application Discussion

23. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Hydrological Model Simple Hydrological Watershed Model (2): Kuczera et al. 2006 1 2 3 4 5 7 8 A B 8 model parameters 3 initial conditions 1 standard dev. of obs. err. 3 „modification parameters“ C Motivation Approach Implementation Application Discussion 6

24. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Hydrological Model Simple Hydrological Watershed Model (3): Motivation Approach Implementation Application Discussion

25. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Model Application  Data set of Abercrombie watershed, New South Wales, Australia (2770 km 2 ), kindly provided by George Kuczera (Kuczera et al. 2006).  Box-Cox transformation applied to model and data to decrease heteroscedasticity of residuals.  Step function input to account for input data in the form of daily sums of precipitation and potential evapotranspiration.  Daily averaged output to account for output data in the form of daily averaged discharge. Motivation Approach Implementation Application Discussion

26. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Analysis with Constant Parameters  Estimation of 11 model parameters: 8 rate parameters 3 initial conditions 1 measurement standard deviation  Priors: Independent lognormal distributions for all parameters with the exception of the measurement standard deviation (1/  ).  Modification factors (f rain, f pet, f Q ) kept equal to unity. Motivation Approach Implementation Application Discussion

27. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Results for Constant Parameters Motivation Approach Implementation Application Discussion

28. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Results for Constant Parameters Motivation Approach Implementation Application Discussion

29. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Results for Constant Parameters Motivation Approach Implementation Application Discussion

30. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Results for Constant Parameters Motivation Approach Implementation Application Discussion  Residuals are heteroscedastic and autocorrelated.  The standard deviation of the residuals is larger than the measurement error (increasing from 0.24 m 3 /s at a discharge of zero to 30 m 3 /s at 100 m 3 /s).  Model predictions are overconfident. In addition: ground water level trend seems unrealistic. The results show the typical deficiencies of deterministic models:

31. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis Motivation Approach Implementation Application Discussion

32. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 1 Step 1: Estimation of time-dependent parameters  Estimation of 11 time-dependent parameters: 8 rate parameters 3 modification factors (f rain, f pet, f Q )  Ornstein-Uhlenbeck process applied to the log of each parameter sequentially. Hyperparameters:  =1d,  =0.2 (22%) fixed, only estimation of initial value and mean (0 for log f rain, f pet, f Q ).  Constant parameters as before. Motivation Approach Implementation Application Discussion

33. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 2 Step 2: Analyzing Degree of Bias Reduction  As quality of fit is insufficient (residual standard deviation larger than measurement error), quality of fit is a primary indicator of bias (when being careful with regard to overfitting).  Reduction of autocorrelation can be checked as a secondary criterion (it is likely to be accompanied by reduction of residual standard deviation). Motivation Approach Implementation Application Discussion

34. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 2 Improvement of fit: Motivation Approach Implementation Application Discussion Nash-Sutcliffe indices: f rain 0.90 k s 0.84 f Q 0.67 s F 0.63 f pet 0.60 k r 0.57 k et 0.54 q lat,max 0.54 k dp 0.53 k gw,max 0.52 k bf 0.52 base0.51 Assessment:  Input (f rain ) and output (f Q ) modifications.  Potential for soil / runoff model (k s, S F ) improvements.  Some potential for river and evaporation improvements. Random or deterministic?

35. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 3 Motivation Approach Implementation Application Discussion Step 3: Identification of Potential Dependences  Despite doing an exploratory analysis of the values of time dependent parameters on all model states and inputs, no significant dependences could be found.  This is an indication that it may be difficult to improve the deterministic model, or that the improvement will be restricted to a small number of data points.

36. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 4 Step 4: Improvement of Deterministic Model : Motivation Approach Implementation Application Discussion Extension 1: Modification of runoff flux: Extension 2: Modification of sat. area funct.: Extentsion 1 has two, extension 2 three additional model parameters.

37. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 4 Model Extensions: Motivation Approach Implementation Application Discussion

38. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 4 Previous results: Motivation Approach Implementation Application Discussion Extended models: Nash-Sutcliffe indices: ext. 10.73 ext. 20.51 Nash-Sutcliffe indices: f rain 0.90 k s 0.84 f Q 0.67 s F 0.63 f pet 0.60 k r 0.57 k et 0.54 q lat,max 0.54 k dp 0.53 k gw,max 0.52 k bf 0.52 base0.51

39. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 4 Motivation Approach Implementation Application Discussion Original Model: Modified Model:

40. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 4 Motivation Approach Implementation Application Discussion Original Model: Modified Model:

41. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 4 Motivation Approach Implementation Application Discussion Original Model: Modified Model:

42. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 4 Motivation Approach Implementation Application Discussion Conclusions of Step 4  The significant increase in the Nash-Sutcliffe index is caused by the elimination of a small number of outliers.  All other deficiencies remain.  This is the reason why the improvement could not have been detected in the exploratory analysis.  It seems questionable that the remaining deficiencies could be significantly reduced by improvements of the deterministic model.

43. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 5 Motivation Approach Implementation Application Discussion Step 5: Addition of Stochasticity to the Model Major sources of indeterminism:  Spatial aggregation: Aggregation of distributed reservoirs in a much smaller number of reservoirs in the model leads to the same model results for different „states of nature“ (that lead to different results in nature).  Rainfall uncertainty: Spatial heterogeneity of rainfall intensity is not well captured by point rainfall measurements.

44. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 5 Motivation Approach Implementation Application Discussion It seems reasonable to summarize these sources of indeterminism by a stochastic rain modification factor f rain. To quantify input uncertainty (combined with aggregation error) we need an informative prior for the measurement error. We choose  Q,trans ~ N(0.5,0.05). 0.5 corresponds to a standard deviation in original units increasing from 0.1 m 3 /s at a discharge of zero to 12.6 m 3 /s at a discharge of 100 m 3 /s. The standard deviation of the Ornstein-Uhlenbeck process for log f rain is now estimated from the data.

45. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 5 Motivation Approach Implementation Application Discussion Time-dependent parameter f rain :

46. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 5 Motivation Approach Implementation Application Discussion

47. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 5 Motivation Approach Implementation Application Discussion Original Model: Modified Model with Time-Dependent Parameter f rain :

48. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 5 Motivation Approach Implementation Application Discussion Original Model: Modified Model with Time-Dependent Parameter f rain :

49. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Deficiency Analysis / Step 5 Motivation Approach Implementation Application Discussion Original Model: Modified Model with Time-Dependent Parameter f rain :

50. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Discussion Motivation Approach Implementation Application Discussion

51. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Discussion The suggested procedure seems to fulfil the expectations of supporting the identification of model deficits and of introducing stochasticity into a deterministic model. It is related to and can be viewed as a generalization of previous work on Time-dependent parameters using Kalman filtering (e.g. Beck and Young 1976, etc.) Modelling of bias of deterministic models (Craig et al. 1996, Kennedy and O‘Hagan 2001, Bayarri et al. 2005, etc.) Rainfall multipliers (Kuczera 1990, Kavetski et al. 2001, etc.) Motivation Approach Implementation Application Discussion

52. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Discussion There is need for future research in the following areas: Explore alternative ways of learning from the identified parameter time series. Different formulation of time-dependent parameters (for some applications smoother behaviour). Include multiple time-dependent parameters into the analysis. Use a more specific model to represent input uncertainty. Improve efficiency (linearization, emulation). Learn from more applications. Motivation Approach Implementation Application Discussion

53. Data-driven and physically- based models, IMS, Singapore, Jan. 2008 Acknowledgements  Collaboration for this paper: Johanna Mieleitner  Development of the technique: Hans-Rudolf Künsch, Roland Brun, Christoph Buser, Lorenzo Tomassini, Mark Borsuk. Hydrological example and data: George Kuczera. Interactions at SAMSI: Susie Bayarri, Tom Santner, Gentry White, Ariel Cintron, Fei Liu, Rui Paulo, Robert Wolpert, John Paul Gosling, Tony O‘Hagan, Bruce Pitman, Jim Berger, and many more. Motivation Approach Implementation Application Discussion