February 2012

Offerred by: C.E. Brockway Brockway Engineering Jim Brannon Leonard Rice Engineers, Inc John Koreny HDR Inc Willem Schreuder Principia Mathematica Dave Colvin Leonard Rice Engineers, Inc Reviewed by:Dave Blew Idaho Power Company Jon Bowline Idaho Power Company January 20, 2012

Need for Uncertainty Analysis Administrators and users need to analyze potential risk in relying on output Most likely predicted outcome needed Range or probability of occurrence of specific magnitudes or outcomes Administrative water rights decisions (discretionary or not) require technically sound and scientifically supported knowledge of model capabilities

Model Uncertainty Sources Conceptual Uncertainty Mathematical Uncertainty Parameter Uncertainty Internal Calibration Uncertainty(model overspecification) Calibration target uncertainty PREDICTIVE UNCERTAINTY-(Integrated effect or accuracy with which the model can simulate response of an output parameter representing all sources of uncertainty) Outlined by C. Brendeke

Current ESPAM2 Uncertainty Analysis Provides an estimate of the range of values for a specific output due to parameter adjustments within which the model will remain calibrated May be called a duel model approach “Remain calibrated”-modeller defined Assumes that the both the conceptural model and input data are correct (without error) when determining the range of predictions Separate analyses required to determine the impact of unadjustable input data or conceptual model uncertainty

Monte Carlo Uncertainty Analysis More rigorous, time consuming, more computing power Unconstrained-random range of input parameters Model may not remain calibrated Null Space Analysis-many calibrated models Monte Carlo analysis provides a probability distribution for any chosen output parameter Modeler may define range of input parameters as a measure of input parameter uncertainty

Probability and Risk Analysis Quantifiable range of simulated values with associated probability is helpful Sometimes the user is concerned about the range and probability of output if a single parameter is varied (personal knowledge of the uncertainty in a specific parameter) Current procedure provides no probability distribution or confidence limits on output Utilization of sensitivity analysis during calibration can provide insight into improvement in calibration

Recommendations Complete the ESPAM2 calibration as soon as possible Complete the uncertainty analysis as outlined as soon as possible Fully document to procedure and results Complete the proposed verification as soon as possible, first priority verification, lower priority for pre calibration period verification Compare similar output from ESPAM1.1 and ESPAM2 Officially adopt ESPAM2, modify and improve transfer tool and guidelines

USE of ESPAM2 MODEL ESHMC has a responsibility to advise the Director and staff to: 1. Develop the best available scientific tool for evaluating ESPA hydrologic relationships and 2.Providing hydrologic guidelines in the use of the model for administrative decisions 3. Provide guidance on technical deficiencies and the real meaning of simulation results 4. Provide adequate information to the Director and model users to understand and defend the model

USE OF ESPAM2-Uncertainty ESPAM2 is a tool for making decisions and developing policy. Neither ESPAM2 or ESHMC makes policy Uncertainty analysis can: 1. Provide insight into sensitive parameters and need for additional data 2. Provide guidance in improving calibration 3. Provide insight into output parameter variability and bias and assist in evaluating risk in use of model output

USE OF MODEL FOR WATER RIGHTS ADMINISTRATION ESHMC or IDWR staff do not necessarily tell the Director how to use the output of the model ESHMC and IDWR staff can tell the Director: 1. What the model can do (capabilities) 2. What the model can’t do 3. What the output really means, technically What are confidence limits? What is variability or bias in simulated output? How sensitive is specific spring output to ET? NOT: How much is impact from junior spring users costing spring users? Can junior ground water users afford to mitigate or ??? 4. Is a particular method of use of the model defensible (statistically or analytically)

SUMMARY ESPAM2 is the best scientific tool available to IDWR Model needs to be completed and adopted as soon as possible Current calibration, uncertainty analysis, validation, and ESPAM1.1 comparison needs to be completed as planned Complete analysis and documentation including model capabilities and limits should be completed The current utilization of the trim line concept as a surrogate for model uncertainty is not defensible and other protocol utilizing documented uncertainty analyses should be adopted