Sensitivity and Importance Analysis Risk Analysis for Water Resources Planning and Management Institute for Water Resources 2008

Sensitivity Analysis Defined Study of how the variation in the output of a model can be apportioned, qualitatively or quantitatively, to different “sources of variation” Including assumptions Input uncertainty Scenario/model uncertainty

The Point Complex analysis may have dozens of input and output variables that are linked by a system of equations Analysts and decision makers must understand the relative importance of the components of an analysis Some outcomes and decisions are sensitive to minor changes in assumptions and input values

Sensitivity Analysis If it is not obvious which assumptions and uncertainties most affect outputs, conclusions and decisions the purpose of sensitivity analysis is to systematically find this out

Systematic Investigation of… Future scenarios Model parameters Model inputs Assumptions Model functional form

Why Sensitivity Analysis? Provides understanding of how output variables respond to changes in model inputs Increases confidence in analysis and its predictions

Assumptions Sensitivity List the key assumptions (scenarios) of your analysis Explore what happens as you change/drop each one individually Do your answers change? Challenging assumptions can be effective sensitivity analysis

Sensitivity Analysis Methods Deterministic one-at-a-time analysis of each factor Deterministic joint analysis Scenario analysis Subjective estimates Parametric analysis--range of values Probabilistic analysis can be used for importance analysis

One-At-A-Time Analysis Hold each parameter constant Expected value Representative value Let one input vary Assumption Input Parameter Common, useful, dangerous

One-At-A-Time Analysis Do not equate magnitude with influence A=U(10 7,10 8 ), B=U(2,6) C = A + B; A dominates C = A B ; B dominates

One-At-A-Time Analysis Dependence and branching in model creates flaws with this logic If A<50 then C = B + 1 Else C = B 100 What value do we set A equal to?

Joint Analysis Change combinations of variables at same time Enables analysts to take dependencies explicitly into account Can have same limitations as OAAT analysis

Subjective Estimates Subjective estimates of uncertain values can be used to identify threshold values of importance to the risk assessment

Range of Values A specific (not subjective) range of values is used E.g., 10 th, 50 th, 90 th percentiles Ceteris paribus approach All possible combinations approach All 10 th percentiles, 10 th with 90 th and so on

Importance Analysis How much does each model input contribute to the variation in the output? Typically a few key inputs account for most output variation These are your important inputs. Not particularly good at identifying nonlinear or multivariate relationships

Statistical Methods Apportion variation in output to inputs via Regression analysis Analysis of variance Response surface methods Fourier amplitude sensitivity test (FAST) Mutual information index (MII) Classification and regression trees (CART)

So What? When decision is sensitive to changes or uncertainties within realm of possibility then more precision and additional information may be required More data (research) Better models Conservative risk management

Take Away Points “What if” analysis is essential to good risk assessment Systematic investigations of model parameters, model inputs, assumptions, model functional form Essential to good risk management

One-At-A-Time Analysis aka Nominal range sensitivity analysis (NRSA). Individually varying one model input across its range of plausible values holding all other inputs at nominal or base-case values Resulting difference in model output is called the sensitivity or swing weight of model to the varied input

Automatic Differentiation (AD). Systematic evaluation of partial derivative of model output with respect to a given model input Similar to NRSA Only an arbitrarily small change is considered, rather than a possible range of values Provides indication of local sensitivity.