1. France 2008 1 Recent advances in Global Sensitivity Analysis techniques S. Kucherenko Imperial College London, UK s.kucherenko@imperial.ac.uk

2. France 2008 2 Introduction of Global Sensitivity Analysis and Sobol’ Sensitivity Indices Why Quasi Monte Carlo methods (Sobol’ sequence sampling) are much more efficient than Monte Carlo (random sampling) ? Effective dimensions and their link with Sobol’ Sensitivity Indices Classification of functions based on global sensitivity indices Link between Sobol’ Sensitivity Indices and Derivative based Global Sensitivity Measures Quasi Randon Sampling - High Dimensional Model Representation with polynomial approximation Application of parametric GSA for optimal experimental design Outline

3. France 2008 3 Model Output x i : input factors Propagation of uncertainty Input x1x1 x2x2 … x3x3 … 1 2n … x4x4 xkxk y

4. France 2008 4 Consider a model x is a vector of input variables Y is the model output. Variance decomposition: Sobol’ SI: Sensitivity Indices (SI) ANOVA decomposition (HDMR):

5. France 2008 5 Sobol’ Sensitivity Indices (SI) Definition: - partial variances - variance  Requires 2 n integral evaluations for calculations Sensitivity indices for subsets of variables: Introduction of the total variance:  Corresponding global sensitivity indices:

6. France 2008 6 How to use Sobol’ Sensitivity Indices? accounts for all interactions between y and z, x=(y,z). The important indices in practice are and  does not depend on ;  does only depend on ;  corresponds to the absence of interactions between and other variables  If then function has additive structure: Fixing unessential variables  If does not depend on so it can be fixed  complexity reduction, from to variables

7. France 2008 7 Evaluation of Sobol’ Sensitivity Indices Straightforward use of Anova decomposition requires 2 n integral evaluations – not practical ! There are efficient formulas for evaluation of Sobol’ Sensitivity Indices ( Sobol’ 1990): of Sobol’ Sensitivity Indices ( Sobol’ 1990): Evaluation is reduced to high-dimensional integration. Monte Carlo method is the only way to deal with such problems

8. France 2008 8 Original vrs Improved formulae for evaluation of Sobol’ Sensitivity Indices

9. France 2008 9 Improved formula for Sobol’ Sensitivity Indices

10. France 2008 10 Comparison deterministic and Monte Carlo integration methods

11. France 2008 11 Monte Carlo integration methods

12. France 2008 12 How to improve MC ?

13. France 2008 13 Sobol’ Sequences vrs Random numbers and regular grid Unlike random numbers, successive Sobol’ points “know" about the position of previously sampled points and fill the gaps between them

14. France 2008 14 Quasi random sequences

15. France 2008 15 What is the optimal way to arrange N points in two dimensions? Regular GridSobol’ Sequence Low dimensional projections of low discrepancy sequences are better distributed than higher dimensional projections

16. France 2008 16 Comparison between Sobol sequences and random numbers

17. France 2008 17 Normally distributed Sobol’ Sequences Normal probability plots Histograms Uniformly distributed Sobol’ sequences can be transformed to any other distribution with a known distribution function

18. France 2008 18 Are QMC efficient for high dimensional problems ? “For high-dimensional problems (n > 12), QMC offers no practical advantage over Monte Carlo” ( Bratley, Fox, and Niederreiter (1992)) ?!

19. France 2008 19 Discrepancy I. Low Dimensions

20. France 2008 20 Discrepancy II. High Dimensions MC in high-dimensions has smaller discrepancy

21. France 2008 21 Is MC more efficient for high-dimensional problems than QMC ? Pros:  MC in high-dimensions has smaller discrepancy  Some studies show degradation of the convergence rate of QMC methods in high-dimensions to O(1/√N) Cons: Huge success of QMC methods in finance: QMC methods were proven to be much more efficient than MC even for problems with thousands of variables Many tests showed superior performance of QMC methods for high-dimensional integration

22. France 2008 22 Effective dimension ___________________________________________________________

23. France 2008 23 For many problems only low order terms in the ANOVA decomposition are important Consider an approximation error Theorem 1: Link between an approximation error and effective dimension in superposition sense Approximation errors Set of variables can be regarded as not important if If and Consider an approximation error Theorem 2: Link between an approximation error and effective dimension in truncation sense ___________________________________________________________________

24. France 2008 24 Type B. Dominant low order indices Classification of functions Type B,C. Variables are equally important Type A. Variables are not equally important Type C. Dominant higher order indices

25. France 2008 25 Sensitivity indices for type A functions

26. France 2008 26 Integration error vs. N. Type A Integration error vs. N. Type A (a) f(x) = ∑ n j=1 (-1) i  i j=1 x j, n = 360, (b) f(x) =  s i=1 │4x i -2│/(1+a i ), n = 100 (a) (b)

27. France 2008 27 Sensitivity indices for type B functions Sensitivity indices for type B functions Dominant low order indices

28. France 2008 28 Integration error vs. N. Type B Integration error vs. N. Type B Dominant low order indices (a) (b)

29. France 2008 29 Sensitivity indices for type C functions Sensitivity indices for type C functions Dominant higher order indices

30. France 2008 30 The integration error vs. N. Type C The integration error vs. N. Type C Dominant higher order indices: (a) (b)

31. France 2008 31 The Morris method Model Elementary Effect for the i th input factor in a point X o

32. France 2008 32 r elem. effects EE 1 i EE 2 i … EE r i are computed at X 1, …, X r and then averaged. Average of EEi’s   (x i ) Standard deviation of the EEi’s  σ (x i ) The EEi is still a local measure Solution: take the average of several EE

33. France 2008 33 A graphical representation of results Factors can be screened on the  (x i ), σ (x i ) plane

34. France 2008 34 Implemention of the Morris method r trajectories of (k+1) sample points are generated, each providing one EE per input A trajectory of the EE design Total cost = r (k + 1) r is in the range 4 -10

35. France 2008 35  *(x i ) and S Ti give similar ranking Problems: large Δ -> incorrect  *(x i ) a=99 a=9 a=0.9 A comparison with variance-based methods:  *(x i ) is related to S Ti Test: the g-function of Sobol’

36. France 2008 36 Derivative based Global Sensitivity Measures Morris measure in the limit Δ → 0 Sample X1, …, Xr Sobol points, estimate finite differences E 1 i, E 2 i … E r i and then averaged. Average of Ei’s  M*(x i )

37. France 2008 37 The integration error vs. N. Type A The integration error vs. N. Type A g-function of Sobol’. (a) (b)

38. France 2008 38 Comparison of Sobol’ SI and Derivative based Global Sensitivity Measures (a) (b) (c) There is a link between and

39. France 2008 39 Comparison of Sobol’ SI and Derivative based Global Sensitivity Measures 1. Small values of imply small values of. 2. For highly nonlinear functions ranking based on global SI can be very different from that based on derivative based sensitivity measures

40. France 2008 40 For many problems only low order terms in the ANOVA decomposition are important. Sobol’ SI: Quasi Randon Sampling HDMR is a metamodel (HDMR), Rabitz et al: It is assumed that effective dimension in superposition sense d s =2.

41. France 2008 41 Polynomial Approximation Properties: Orthonormal polynomial base First few Legendre polynomials:

42. France 2008 42 Global Sensitivity Analysis (HDMR) The number of function evaluations is N(n+2) for original Sobol’ method N for sensitivity indices based on RS-HDMR

43. France 2008 43 How to define maximum polynomial order ? Homma-Saltelli function

44. France 2008 44 RMSE for Homma-Saltelli function Root mean square error: QMC outperforms MC RS-HDMR has higher convergence than Sobol SI method

45. France 2008 45 g-function: with 2 important and 8 unimportant variables Sobol g-function QRS-HDMR converges faster Values of S i tot can be inaccurate.

46. France 2008 46 Sobol g-function Error measure: Function Approximation

47. France 2008 47 QRS-HDMR method requires 10 to 10 3 times less model evaluations than Sobol SI method ! Computational costs

48. France 2008 48 Optimal experimental design (OED) for parameter estimation Find values of experimentally manipulable variables (controls) and the time sampling strategy for a set of N exp experiments which provides maximum information for the subsequent parameter estimation problem Non-linear programming problem (NLP) with partial differential-algebraic (PDAEs) constraints subject to: System dynamics (ODEs, DAEs) Other algebraic constraints Upper and lower bounds:

49. France 2008 49 Case study: fed-batch reactor Biomass: Substrate: Reaction rate: Parameters to be estimated: p 1, p 2 0.05 < p 1 < 0.98, 0.05 < p 2 < 0.98 Control variables: u 1, u 2 Dilution factor: 0.05 < u 1 < 0.5 Feed substrate concentration: 5 < u 2 < 50

50. France 2008 50 OED traditional approach Fisher Information Matrix ( FIM ) based criteria:  A criterion =  D criterion =  E criterion =  Modified-E criterion = Main drawback: based on local SI non-realistic linear and local assumptions

51. France 2008 51 Parametric GSA Optimal experimental design: identification of a set of experiments with conditions that deliver measurement data that are the most sensitive to the unknown parameters

52. France 2008 52 Application of ParametricGSA for parameter optimization Application of Parametric GSA for parameter optimization Main advantage: based on global SI allows to consider a range of values for the parameters to be estimated n objective function: n Application of Global Optimization method

53. France 2008 53 Case study: fed-batch reactor Biomass: Substrate: Reaction rate: Parameters to be estimated: p 1, p 2 0.05 < p 1 < 0.98, 0.05 < p 2 < 0.98 Control variables: u 1, u 2 Dilution factor: 0.05 < u 1 < 0.5 Feed substrate concentration: 5 < u 2 < 50

54. France 2008 54 Optimal Experimental Design Problem constraints:  Experiment duration: 10 h  Number of measurement times: 10  Controls varied every 2 hours  n Results: Optimal input profile for u 1 and u 2 :

55. France 2008 55 Setting of the Parameter Estimation Problem Steps to find p: Take experimental or generated pseudo-experimental points Maximum likelihood optimization subject to: System dynamics (ODEs, DAEs) Other algebraic constraints Upper and lower bounds: Non-linear programming problem (NLP) with partial differential-algebraic (PDAEs) constraints p: set of parameters to be estimated : model prediction : measurements variance : experimental measures

56. France 2008 56 Results of parameter estimation p 1 = 0.37 ± 0.02, p 2 = 0.72 ± 0.12 p 1 = 0.5 ± 0.05, p 2 = 0.5 ± 0.11

57. France 2008 57 Publications Publications Hung WY, Kucherenko S., Samsatli N.J. and Shah N., The Proceedings of the 2003 Summer Computer Simulation Conference, Canada. Simulation Series, V 35, N3, pp. 101-106 (2003) Hung W.Y., Kucherenko S., Samsatli N.J. and Shah N (2004). Journal of the Operational Research Society 55, 801-813. Sobol’ I., Kucherenko S. Monte Carlo Methods and Simulation, 11, 1, 1-9 (2005). Sobol’ I., Kucherenko S. Wilmott, 56-61, 1 (2005). Kucherenko S., Shah N. Wilmott, 82-91, 4 (2007). Sobol, I.M., S. Tarantola, D. Gatelli, S.S. Kucherenko, W. Mauntz Reliability Engineering & System Safety, 957-960, 92 (2007 ). Rodriguez-Fernandez M., Kucherenko S., Pantelides C., Shah N. Proc. ESCAPE17, V. Plesu and P.S. Agachi (Editors), p66-71, (2007) Kucherenko S., Mauntz W. Submitted to Journal of Comp. Physics (2007). S. Kucherenko. Fifth International Conference on Sensitivity Analysis of Model Output, Budapest, (2007) S. Kucherenko, M. Rodriguez-Fernandez, C. Pantelides, N. Shah. Submitted to Reliability Engineering Systems Safety (2007) D. Gatelli, S. Kucherenko, M. Ratto, S. Tarantola, Submitted to Reliability Engineering Systems Safety (2007) I.M. Sobol’, S. Kucherenko. Submitted to Journal of Comp. Physics (2008). Application of Global Sensitivity Analysis to Biological Models A.Kiparissides, M.Rodriguez-Fernandez, S. Kucherenko, A. Mantalaris, E.Pistikopoulos Application of Global Sensitivity Analysis to Biological Models, Submitted to ESCAPE18 (2008).

58. France 2008 58 Summary uasi MC methods based on Sobol’ sequences outperform MC The error generated by the factors fixing is bounded by the total sensitivity index of the fixed factors Functions can be classified according to their effective dimension The method of derivative based global sensitivity measures (DGSM) is more efficient than the Morris and the Sobol’ SI methods. There is a link between DGSM and Sobol’ SI Quasi Randon Sampling - High Dimensional Model Representation with polynomial approximation can be orders of magnitude more efficient than Sobol’ SI for evaluation of main effects Application of global SI to OED results in the reduction of the required experimental work and the increased accuracy of parameter estimation Summary Quasi MC methods based on Sobol’ sequences outperform MC The error generated by the factors fixing is bounded by the total sensitivity index of the fixed factors Functions can be classified according to their effective dimension The method of derivative based global sensitivity measures (DGSM) is more efficient than the Morris and the Sobol’ SI methods. There is a link between DGSM and Sobol’ SI Quasi Randon Sampling - High Dimensional Model Representation with polynomial approximation can be orders of magnitude more efficient than Sobol’ SI for evaluation of main effects Application of global SI to OED results in the reduction of the required experimental work and the increased accuracy of parameter estimation

59. France 2008 59 Thank you for inviting me !Acknowledgments Prof. Sobol’ Imperial College London, UK: N. Shah, M. Rodríguez Fernández, B. Feil, W. Mauntz, C. Pantelides Joint Research Centre, ISPRA, Italy: S. Tarantola, D. Gatelli, M. Ratto Financial support: EPSRC Grant EP/D506743/1