Complex Models in the Water Industry Water Statisticians User Group 14 April 2010 Edward Glennie

Outline of Presentation Replies to questionnaire Modelling generally: Sources of error Methods of assessing the size of any errors Two specific water industry applications Optimising interventions to maintain a set of assets Use of time series rainfall Pointers for the future

Replies to questionnaire – Thank you! Model description Count Asset failure probability, deterioration & influence of weather 5 Optimise investment in a set of assets 3 Per capita water consumption micro-component model 1 Economic level of leakage Water resources with rainfall time series DG5 flood predictor Sewer network hydraulic modelling Sewage treatment process models

Models and modelling issues Model description Count Asset failure probability, deterioration & influence of weather 5 Optimise investment in a set of assets 3 Per capita water consumption micro-component model 1 Economic level of leakage Water resources with rainfall time series DG5 flood predictor Sewer network hydraulic modelling Time series rainfall and climate change projections - Sewage treatment process models

Use of words A model has outcomes (predictions, results), which are used to help make decisions The outcomes are subject to errors (uncertainty) The results are robust if the errors are small enough for us to take the decision with confidence

Walker et al (2003) quoted in Refsgaard (2007) Sources of error Source of error Example Context Modelling asset failure rates – assume current operating conditions will apply in future Input data Modelling asset failure rates – early reporting of failures may be incomplete Parameter uncertainty Pipe roughness values in water distribution modelling Model structure Asset failure rates – use of linear model (or Weibull etc) to predict future failure rates Model technical uncertainty Numerical approximations, software bugs Walker et al (2003) quoted in Refsgaard (2007)

Adapted from Brown (2004) quoted in Refsgaard (2007) Types of error Statistical Size of potential errors in outcomes can be estimated Qualitative Potential errors in outcomes can be identified and described Recognised ignorance Aware of poorly understood potential errors Unrecognised ignorance ! Adapted from Brown (2004) quoted in Refsgaard (2007)

Types of statistical error If it is due to imperfect knowledge and could be reduced by more information, it is epistemic If it is due to inherent variability and could not be reduced by more information, it is stochastic

Optimise interventions in a set of assets Asset data Renewal & planned maintenance data Current and future failure rates Failure rate model Failure data Weather data System data, GIS data, judgement… Actual failure impacts Failure impacts model Optimal interventions Optimiser Set of possible interventions

Failure rate modelling - methods Divide assets into cohorts, regression / fit (e.g.) Weibull function Bayesian approach: probability, severity & consequence of failure, partition assets, tune model using observed failures Definitions of failure

Failure rate modelling - issues Source of error Example Type Context e.g. Assume current ‘environment’ and operating conditions will apply in future Qualitative or Recognised ignorance Input data e.g. Complete failure data may exist for only a few years Civil assets are not operated to end of life failure Qualitative? Parameter uncertainty   Statistical Model structure Assumed form of model to predict future failure rates (linear, Weibull etc) Model technical uncertainty None?

Failure impact modelling - methods Impacts: direct costs & customer service Infrastructure, civil assets, M&E, ICA Methods GIS based e.g. for flooding using contours Population potentially affected e.g. water supply low pressure Use of actual data? Large element of judgement

Failure impact modelling - issues Source of error Example Type Context Input data Parameter uncertainty Model structure Model technical uncertainty

Failure impact modelling issues (second try) Is there a hierarchy of possible consequences for each failure?

Impact of failure – probability distribution

Failure impact modelling issues (second try) Is there a hierarchy of possible consequences for each failure? Major impacts result from a combination of several failures Are non-failure impacts of deterioration taken into account? e.g. leakage, pump efficiency Qualitative errors / recognised ignorance?

Optimisation Monetarise all impacts? Use LoS targets as constraints in optimisation ‘It’s difficult to prove that you’ve reached the optimum’ Bring modeller’s and engineer’s perspectives together

Optimising investment – for discussion and development Failure rate modelling Cohort vs Bayesian approach Failure impact modelling Use of judgement: benchmark against other industries (nuclear? Little 1998) Use of data to verify models (but gathering data costs money!) Sensitivity analyses

Time Series Rainfall Used because rainfall history affects system behaviour, as well as current rainfall, e.g. runoff depends on wetness of ground Rainfall patterns are local Actual time series are best, if they exist Often time series have to be generated Much data collection & research effort, but issues remain (short duration storms)

Climate change and time series rainfall How should a ‘current’ time series be modified to take account of climate change? Focus on 1 in ~1 year storms, with sewer system storage volume in mind

Empirical relationships: UKCP09 rainfall model Parameters Mean daily rainfall Proportion of dry days Variance Skewness Lag -1 autocorrelation Time series generator Daily rainfall time series Empirical relationships: 1 hour to 24 hours Hourly rainfall time series Source: UKCP09 Guidance - Annex

Empirical relationship – 1 hour vs 1 day

What we did Specific location Historical 30 year hourly TSR Current UKCP09 30 year hourly TSR (100 times) Scenario UKCP 30 year hourly TSR (100 times) Summary HIST: storm nrs. by depth & duration Summary CNTR: storm nrs. by depth & duration Summary SCEN: storm nrs. by depth & duration Comparison: CNTR ~= HIST? Comparison: Derive factors SCEN/CNTR Adjusted historical TSR

A better way? Specific location Historical 30 year hourly TSR Current UKCP09 5 parameters Scenario UKCP 5 parameters Comparison: CNTR ~= HIST? Comparison: Change factors SCEN/CNTR 5 parameters & uncertainties 5 parameters - changed Adjusted historical TSR

Methods of assessing errors & their significance Statistical estimates Sensitivity analysis Scenario analysis Uncertainty matrix

Uncertainty matrix Source Type Importance Weight Uncertainty * Weight Problem context ……………………………. Input data Parameter error Model structure Model technical uncertainty

Methods of assessing errors & their significance Statistical estimates Sensitivity analysis Scenario analysis Uncertainty matrix Involve stakeholders There are others – see Refsgaard (2007). Some also control / reduce errors.

Ideas to take forward Optimising investment Modelling of impacts TSR & Climate change Use basic parameters underlying the time series generator Framework for understanding modelling errors Consistent use: sources, types, how to assess (& minimise)

References Brown JD, Knowledge, uncertainty and physical geography: towards the development of methodologies for questioning belief, 2004, Trans Inst. British Geographers vol.52 (6) pp.367-381 Little MP, Invited editorial in J Radio Prot, 1998, vol.18 p.239 Refsgaard JC et al, Uncertainty in the environmental modelling process – a framework and guidance, Environmental Modelling and Software, 2007, vol.22 pp.1543-1556 UKCP, 2009, UK Climate Projections, http://ukcp09.defra.gov.uk/ Walker WE et al, Defining uncertainty – a conceptual basis for uncertainty management in model-based decision support, Integrated Assessment, 2003, vol.4 (1) pp.5-17