1. Data Mining of Environmental Models for Sensitivity Analysis Tom Stockton Paul Black, Andy Schuh, Kate Catlett, John Tauxe Neptune and Company, Inc. www.neptuneandco.com Knowledge Discovery re

2. Issue How to conduct a sensitivity analysis of a complex high dimensional probabilistic environmental model?

3. Decision Modeling 1.Decision Model, build and solve –Decision Actions and Outcomes –Utility (costs, liabilities, desires) –Probabilistic model Scenario Model Parameter 2.Sensitivity analysis (knowledge re-discovery) 3.Value of information analysis (OUT-path) 4.Data collection 5.Update model (Bayesian or ad hoc)

4. Decision Modeling U(d | I) = sup d   S  M  Y U(d | y, S, M,  M ) utility function p(S) scenario uncertainty p(M | S) model uncertainty p(  M | S) parameter uncertainty p(I |  M  M, S) data likelihood p(y |  M, M,  S) risk predictive dist dy dS dM d  M where: U= utility, loss, costM= model structure d= decision  M = model parameters I= information/dataS= scenario y= risk

5. Sensitivity Analysis Given a model: Y = f (X) [Y = GoldSim(X)] Sensitivity analysis is aimed at describing the influence of each input variable X i on the model response Y

6. Sensitivity Measures One-At-A-Time (OAT) Differential Analysis Global –Statistical scatter plots, correlation, regression, rank transformations –Data mining Sobol, FAST, MARS, MART

7. Desirable Properties of a SA Measure Efficiency –account for all effects while being computationally affordable Simplicity –implementable and interpretable Model Independent –The method can handle non-linearity, non- monotonicity (across time and space) K. Chan, S. Tarantola and A. Saltelli, 2000, Variance-Based Methods, in Sensitivity Analysis, A. Saltelli, K. Chan, E.M.Scott.John Wiley and Sons.

8. Sensitivity Measures OAT and Differential Analysis, for complex probabilistic models, often are –not efficient, and –not model independent

9. Global Sensitivity Measures Sensitivity Measure Build a statistical model of the model response and the model inputs using the Monte Carlo simulation results Decompose variance of the output and attribute to input variables

10. Standardized Rank Regression SRR –Rank Y and X i and scale the ranks to mean of 0 and variance of 1 for convenience Based on the ranks of Y and X i Assuming the X i are independent

11. Fourier Amplitude Sensitivity Test FAST –Explores the multidimensional input space of the input factors by a search curve using Fourier transform function. –Handles main and interaction effects K. Chan, S. Tarantola and A. Saltelli, 2000, Variance-Based Methods, in Sensitivity Analysis, A. Saltelli, K. Chan, E.M.Scott.John Wiley and Sons.

12. Issues Differential Analysis –not feasible: derivatives of complex models SRR and OAT –not model independent: trouble with nonmonotonic nonlinear models. –not efficient: trouble with interaction effects in high dimensional models FAST –not efficient: Separate model runs

13. Possible Solutions Data mine the probabilistic model output –Multivariate Adaptive Regression Splines (MARS) –Multiple Additive Regression Trees (MART)

14. Data Mining MARS –Non-parametric recursive partitioning approach that fits separate splines to distinct intervals of the predictor variables. MART –Explores the multidimensional input space of the input factors using gradient boosting of additive regression models. Advantages –Search for interactions between variables, allowing any degree of interaction to be considered. –Tracks very complex data structures in high-dimensional data.

15. Sensitivity Indices via ANOVA decomposition Sensitivity indices are calculated using basis functions not including x s

16. Analytical Example Sobol’ g-function Saltelli A., Tarantola S., and Chan K.P.-S. (1999), “A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output,” Technometrics, 41, 39-55.

17. Example: Sobol’ g-function Inputa  Sensitivities Analytic MARTMARSFASTSRR x 1 0230.730.5650.7330.7730.0005 x 2 1550.230.2810.2240.1930.0015 x 3 4.5770.0320.0940.0360.0250.045 x 4 9970.0090.050.0090.0080.197 x 5 991070.00010.0050.00060.00020.207 x 6 991130.00010.0040.00000.00050.437 x7x7 991210.00010.00.00000.00010.007 x8x8 991250.00010.00.00000.00020.105 Saltelli A., Tarantola S., and Chan K.P.-S. (1999), “A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output,” Technometrics, 41, 39-55.

18. Radioactive Waste Disposal Example NTS GCD GoldSim model Simulated data is exported to MS Access SQL query data from and run sensitivity analysis ( is an Open Source statistical programming language )

19. Public BenefitAnalysis Costs ALARA Costs Monitoring Costs Disposal Fees Cumulative (CA) Management Options - Institutional Controls - Site Maintenance - Waste Acceptance - Closure - Monitoring/Surveillance Potential Liabilities Closure Costs Research, Monitoring, Information & Data Collection Choose Management Options & Update Management Plan YES NO Ecosystem MOP & IHI Occupational Regulations & Guidance Can the risk be managed to regulatory thresholds at an acceptable cost with an acceptable level of uncertainty? Disposal Costs Budgets Maintenance Review Periodic Review Waste Acceptance DecisionWaste Acceptance Decision Closure Decision Fate & Transport Existing Inventory Future Inventory 1 2 3 4 5 Cost Management Risk Contamination Uncertainty analysis Sensitivity analysis Value of Information 6 7 Iteration loop Legend 1 Sequence number 8

20. Simulation Results Model Inputs ( X ) –Inventory –Fate and transport Upward advection Biotic transport Model response ( Y ) –“EPA-SUM”

21. Model Response

22. Relative Influence Plot

23. Partial Dependence Plots

24. Co-partial Dependence Plot

25. Variation Explained MART/ TimeSRRMARS GCD 10,0000.910.99 LANL 500.870.94 1000.860.96 5000.750.91 1,0000.710.95 10,0000.710.93 MART/ TimeSRRMARS GCD 10,0000.910.99 LANL 500.870.94 1000.860.96 5000.750.91 1,0000.710.95 10,0000.710.93

26. Sensitivity Convergence

27. Upward Flux OAT

28. Summary MART and MARS appear to provide an –Efficient –Simple (?) –Model Independent approach to data mining probabilistic model results for sensitivity analysis

29. Finally… The decision context: –Is the uncertainty in the model response too high? –Is there value in reducing input uncertainty? –SA and cost used to estimate the value of collecting additional information.

30. FAST

31. MARS Non-parametric recursive partitioning approach that fits separate splines to distinct intervals of the predictor variables. Both the selected variables and the knots are found via a brute force, exhaustive search procedure optimized simultaneously by evaluating a "loss of fit" criterion. Searches for interactions between variables, allowing any degree of interaction to be considered. Tracks very complex data structures in high-dimensional data. J.H. Friedman, (1991), “Multivariate Adaptive Regression Splines,” The Annals of Statistics, 19, 1-14 Software: Trevor Hastie and Robert Tibshirani, MDA Library for R (‘GNU S’). Ross Ihaka and Robert Gentleman, (1996) R: A Language for Data Analysis and Graphics, Journal of Computational and Graphical Statistics, 5, 3, 299-314. www.r-project.org. www.r-project.org

32. MART Multiple Additive Regression Trees –Explores the multidimensional input space of the input factors using gradient boosting of additive regression models. –Handles main and interaction effects. –Fast K. Chan, S. Tarantola and A. Saltelli, 2000, Variance-Based Methods, in Sensitivity Analysis, A. Saltelli, K. Chan, E.M.Scott.John Wiley and Sons.