1. Computational statistics 2009 Sensitivity analysis (SA) - outline 1.Objectives and examples 2.Local and global SA 3.Importance measures involving derivatives or regression coefficients 4.A general variance-based importance measure 5.First-order indices and total indices

2. Computational statistics 2009 Objectives of sensitivity analysis SA is the study of how the variation in the output of a model can be apportioned to different sources of variation. JRC web-site: http://sensitivity-analysis.jrc.cec.eu.int/

3. Computational statistics 2009 Examples of questions addressed in SA  Supply-demand models: How sensitive is the predicted petrol consumption to a major change in price?  Weather forecasting models: How sensitive are the weather forecasts to the grid size used in the calculations?  Watershed models of the turnover of nitrogen: How sensitive is the predicted discharge of nitrogen to the sea to the model parameters describing the turnover of nitrogen in agricultural land?

4. Computational statistics 2009 The INCA-N ( Integrated Nitrogen in Catchments ) model of the flow of nitrogen and water through a river basin Model parameters: Initial conditions Nitrogen transformation rates Hydrogeological parameters Daily weather data INCA - N 20 40 60 80 100 120 11121 20 40 60 80 100 120 11121 Daily estimates of water discharge and NO 3 and NH 4 concentrations in river water Average annual riverine load of inorganic nitrogen

5. Computational statistics 2009 Examples of response surfaces produced by the INCA-N model Average annual nitrogen loss Denitrification rate Max. nitrate uptake rate Plant nitrate uptake rate

6. Computational statistics 2009 Local sensitivity analysis  A local SA is concerned with small changes about some central case of interest and their impact on the outcome of the calculations  Essentially, the objective of a local sensitivity analysis is to find the partial derivatives of the outcome with respect to the inputs at the point in question

7. Computational statistics 2009 Global sensitivity analysis A global sensitivity analysis is concerned with the whole set of potential inputs and aims to give an overall indication of the way that the outcome varies

8. Computational statistics 2009 Global sensitivity analysis - questions addressed  Which of the uncertain input factors is more important in determining the uncertainty in the output of interest?  If we could eliminate the uncertainty in one of the input factors, which factor should we choose to reduce the most the variance of the output?

9. Computational statistics 2009 Sensitivity with respect to factors A factor is anything in a model that can be changed prior to its execution  model input (driving force)  model parameter (unknown constant in the model)  one mesh size versus another  one model structure versus another

10. Computational statistics 2009 A simple portfolio model having three independent components Let where C s, C t, and C u are the quantities per item and P s, P t, and P u are hedged (delta-neutral) portfolios. Assume that P s, P t, and P u are independent with mean zero and different standard deviations (volatilities)

11. Computational statistics 2009 Sensitivity analysis using partial derivatives Our analysis indicates the largest portfolio to be the most important factor regardless of its volatility

12. Computational statistics 2009 Sensitivity analysis using partial derivatives of fractional change gives the fractional increase in Y corresponding to a unit fractional increase in P x at a given point in the input space

13. Computational statistics 2009 Sensitivity analysis using partial derivatives normalised with respect to the standard deviation of each factor When using as a sensitivity measure, the relative importance of P s, P t, and P u depends both on the volatility of these factors and on the weights C s, C t, and C u

14. Computational statistics 2009 Sensitivity analysis using derivatives normalised with respect to the standard deviation of each factor Let Then Fraction of variance attributable to each factor

15. Computational statistics 2009 A simple portfolio model having three independent factors - a Monte Carlo experiment Let Then, we can fit a regression model b x = C x, x = s, t, u

16. Computational statistics 2009 Standardised regression coefficients (SRCs) If we set then we obtain For linear models:

17. Computational statistics 2009 Standardised regression coefficients (SRCs) for non-linear models  The R 2 -value obtained by a linear regression model represents the fraction of the model output variance accounted for by that regression  Provided that the factors are independent, the standardised regression coefficients tell us how this fraction of the output variance can be decomposed according to the input factors, leaving us ignorant about the non-linear parts of the model The SRCs offer measures of sensitivity that are averaged over a set of possible values of other factors

18. Computational statistics 2009 A portfolio model having six independent factors Let where C s, C t, and C u are the quantities per item and P s, P t, and P u are hedged portfolios. Assume that all factors are independent and that

19. Computational statistics 2009 A general importance measure applicable to all models with random inputs  How does change if one can fix a factor P x at its mid-point? Can it decrease? Can it increase?  How does change if one can fix a factor P x at a point different from the midpoint? Can it decrease? Can it increase?  After averaging over all possible values of P x we obtain  The ratio is called importance measure, sensitivity index, or first order effect  The sensitivity index of a factor shows how much the variance of the model output would decrease if you were told the exact value of that factor

20. Computational statistics 2009 A portfolio model having six independent factors - calculation of first order effects FactorSxSx PsPs 0.36 PtPt 0.22 PuPu 0.08 CsCs 0.00 CtCt CuCu Sum0.66

21. Computational statistics 2009 Sums of importance measures for models having independent factors  In general,  For additive models,

22. Computational statistics 2009 Second-order effects in the six-factor portfolio model Calculations or simulations show that for this model we obtain:

23. Computational statistics 2009 Second-order effects in the six-factor portfolio model

24. Computational statistics 2009 A portfolio model having six independent factors - estimates of second order effects FactorSxSx Ps, CsPs, Cs 0.18 P t, C t 0.22 P j, C j 0.08 P s, C t 0.00.. Sum of second order terms 0.34 Sum of first order terms 0.66 Grand sum1.00

25. Computational statistics 2009 General decomposition theorem for the effects of independent factors Total effect index ( S Tx ) of a factor: The sum of all effects involving the factor under consideration First order effects Second order effects Third order effects

26. Computational statistics 2009 A portfolio model having six independent factors - main effects and total effect indices FactorSxSx S Tx PsPs 0.360.57 PtPt 0.220.35 PjPj 0.080.14 CsCs 0.000.19 CtCt 0.000.12 CjCj 0.000.06 Sum0.661.43

27. Computational statistics 2009 Computational aspects of sensitivity analysis  We need to estimate integrals representing variances of random variables  The estimation of integrals is performed using Monte-Carlo techniques involving pseudo- or quasi-random numbers  The computed indices depend on what assumptions that are made regarding the uncertainty of the input factors

28. Computational statistics 2009 Homework: Decomposition of V(Y) and relative value of V(E(Y | X i )) and E(V(Y | X -i )) for different model classes 1.Independent factors For example: Y = X 1 + X 2 + X 1 X 2 2.Additive models having dependent factors For example: Y = X 1  X 1 + X 2

29. Computational statistics 2009 Estimation of first order sensitivity indices Consider the sensitivity indices where These indices can be estimated by computing where and are pseudo – or quasi -random vectors

30. Computational statistics 2009 Estimation of total sensitivity indices Consider the sensitivity indices where the symbol –i indicates that all variables except x i are kept fixed These indices can be estimated by computing