“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Sensitivity and Importance Analysis Charles Yoe

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Sensitivity Analysis Defined Study of how the variation in the output of a model can be apportioned, qualitatively or quantitatively, to different “sources of variation” in the inputs for the purpose of increasing confidence in the analysis –Include assumptions –Input uncertainty –Scenario/model uncertainty

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions The Point Complex analysis may have dozens of input and output variables that are linked by a system of equations Analysts and decision makers must understand the relative importance of the components of an analysis Some outcomes and decisions are sensitive to minor changes in assumptions and input values

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Sensitivity Analysis If it is not obvious which assumptions and uncertainties most affect outputs, conclusions and decisions the purpose of sensitivity analysis is to systematically find this out

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Systematic Investigation of… Future scenarios Model parameters Model inputs Assumptions Model functional form

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Assumptions Sensitivity List the key assumptions (scenarios) of your analysis Explore what happens as you change/drop each one individually –Do your answers change? Challenging assumptions can be effective sensitivity analysis

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Input Sensitivity Parameter-how sensitive is our output to forecast error or other changes in inputs? Unexpected change or error Decision variables (Inputs we control)- might changes in our decisions/actions improve our outputs

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Sensitivity Analysis Methods Deterministic one-at-a-time analysis of each factor Deterministic joint analysis Scenario analysis Subjective estimates Parametric analysis--range of values Probabilistic analysis can be used for importance analysis

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions One-At-A-Time Analysis Hold each parameter constant –Expected value –Representative value Let one input vary –Assumption –Input –Parameter Common, useful, dangerous

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions One-At-A-Time Analysis Do not equate magnitude with influence A=U(10 7,10 8 ), B=U(2,6) C = A + B; A dominates C = A B ; B dominates

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions One-At-A-Time Analysis Dependence and branching in model creates flaws with this logic If A<50 then C = B + 1 Else C = B 100 What value do we set A equal to?

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Joint Analysis Change combinations of variables at same time Enables analysts to take dependencies explicitly into account Can have same limitations as OAAT analysis

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Subjective Estimates Subjective estimates of uncertain values can be used to identify threshold values of importance to the risk assessment

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Range of Values A specific (not subjective) range of values is used –E.g., 10 th, 50 th, 90 th percentiles Ceteris paribus approach All possible combinations approach –All 10 th percentiles, 10 th with 90 th and so on

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Importance Analysis How much does each model input contribute to the variation in the output? Typically a few key inputs account for most output variation –These are your important inputs. Not particularly good at identifying nonlinear or multivariate relationships

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Advanced Statistical Methods Apportion variation in output to inputs via –Regression analysis –Analysis of variance –Response surface methods –Fourier amplitude sensitivity test (FAST) –Mutual information index (MII) –Classification and regression trees (CART)

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions So What? When decision is sensitive to changes or uncertainties within realm of possibility then more precision and additional information may be required –More data (research) –Better models –Conservative risk management

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Take Away Points “What if” analysis is essential to good risk assessment Systematic investigations of model parameters, model inputs, assumptions, model functional form Essential to good risk management

“ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Charles Yoe, Ph.D. Questions?