1. http://www.jrc.cec.eu.int/uasa1 QUANTITATIVE METHODS TO MANAGE UNCERTAINTY IN SCIENCE by Andrea Saltelli, Silvio Funtowicz, Stefano Tarantola, Joint Research Centre of the European Communities in Ispra (I), Andrea.Saltelli@jrc.it MUST: Managing Uncertainty in science for suSTainability: future research challenges for Europe FP6 launch event in Brussels, November 11, 2002

2. http://www.jrc.cec.eu.int/uasa2 Uncertainty is not an accident of the scientific method, but its substance. Peter Høeg, a Danish novelist, writes in Borderliners (Høeg, 1995): "That is what we meant by science. That both question and answer are tied up with uncertainty, and that they are painful. But that there is no way around them. And that you hide nothing; instead, everything is brought out into the open". Uncertainty. Høeg.

3. http://www.jrc.cec.eu.int/uasa3 Models mimic systems Rosen’s formalisation of the modelling process

4. http://www.jrc.cec.eu.int/uasa4 “World” (the natural system) and “Model” (the formal system) are internally entailed - driven by a causal structure. Nothing entails with one another “World” and “Model”; the association is hence the result of a craftsmanship. But this does not apply to natural systems only: give 10 engineers the blueprint of the same plant and they will return you 10 model based risk assessments for the same plant. Models mimic systems (Rosen)

5. http://www.jrc.cec.eu.int/uasa5 It can help the craftsman that the uncertainty in the information provided by the model (the substance of use for the decoding exercise) is carefully apportioned to the uncertainty associated with the encoding process. Models mimic systems (Rosen)

6. http://www.jrc.cec.eu.int/uasa6 >, HORNBERGER G.M., and R. C. Spear (1981) An approach to the preliminary analysis of environmental systems. Journal of Environmental management, 12, 7-18. >, The Economist, 1998. Models maps assumptions onto inferences... but often too narrowly

7. http://www.jrc.cec.eu.int/uasa7 But yet models are used... … and a legitimate question is the following: “If we had mapped the space of uncertain assumptions honestly and judiciously, would the space of inference still be of use 1 ?” 1 Read: do we still have peak around some useful inference (e.g. YES or NO, safe or unsafe, hypothesis accepted or rejected, policy effective or ineffective etc. ) or do we have as many YES as NO etc.? Use of models in the scientific discourse

8. http://www.jrc.cec.eu.int/uasa8 > Edward E. Leamer, “Sensitivity Analysis would help”, in Modelling Economic Series, Edited by CWJ Granger, 1990, Clarendon Press, Oxford. Models maps assumptions onto inferences …

9. http://www.jrc.cec.eu.int/uasa9 Space of estimated parameters Space of plausible models space Simulation... inference Models maps assumptions onto inferences … Leamer’s view of global Sensitivity Analysis (SA) Other assumptions (Copes with Equifinality)

10. http://www.jrc.cec.eu.int/uasa10 Estimated parameters Input data Model Uncertainty and sensitivity analysis Models maps assumptions onto inferences … (Parametric bootstrap version of UA/SA ) Inference (  Parametric bootstrap: we sample from the posterior parameter probability) (  Estimation)

11. http://www.jrc.cec.eu.int/uasa11 Estimation of parameters Loop on boot- replica of the input data Model Bootstrapping-of-the-modelling-process version of UA/SA, after Chatfield, 1995 Inference (  Bootstrap of the modelling process) (  Estimation) (  Model Identification)

12. http://www.jrc.cec.eu.int/uasa12 Inference Data Prior of Model Bayesian Uncertainty and Sensitivity Analysis (Draper 1995, Planas and Depoutot 2000) Prior of Parameters Prior of Model(s) Posterior of Parameters Posterior of Model(s) (  Sampling)

13. http://www.jrc.cec.eu.int/uasa13 The space of the model induced choices (the inference) swells and shrinks by our swelling and shrinking the space of the input assumptions. How many of the assumptions are relevant at all for the choice? And those that are relevant, how do they act on the outcome; singularly or in more or less complex combinations? (ANOVA-type analysis) Use of models in the scientific discourse … and role of uncertainty - sensitivity analysis

14. http://www.jrc.cec.eu.int/uasa14 I desire to have a given degree of robustness in the choice, what factor/assumptions should be tested more rigorously? (=> look at how much “fixing” any given f/a can potentially reduce the variance of the output) E.g. can I confidently “fix” a subset of the input factors/assumptions? The Beck and Ravetz “relevance” issue. How do I find these f/a? (=> total sensitivity indices). Use of models in the scientific discourse … and role of uncertainty - sensitivity analysis

15. http://www.jrc.cec.eu.int/uasa15 Use of models in the scientific discourse … and role of uncertainty - sensitivity analysis Reduced variance Expected reduced variance First order effect Total effect

16. 16 Environmental sustainability Index, From The Economist, Green and growing, The Economist, Jan 25th 2001, Produced on behalf of the World Economic Forum (WEF), and presented to the annual Davos summit this year. Robustness...

17. 17 Mathis Wackernagel, mental father of the “Ecological Footprint” and thus an authoritative source in the Sustainable Development expert community, concludes an argumented critique of the study done presented at Davos by noting: Robustness...

18. http://www.jrc.cec.eu.int/uasa18 "Overall, the report would gain from a more extensive peer review and a sensitivity analysis. The lacking sensitivity analysis undermines the confidence in the results since small changes in the index architecture or the weighting could dramatically alter the ranking of the nations.” Robustness - Wackernagel’s critique

19. http://www.jrc.cec.eu.int/uasa19 Monte Carlo Analysis: countries’ score for the Technology Development Index, (UN), a composite indicator. Modified as to include variability in weights (e.g. as when using budget allocation or Analytic Hierarchy Process). Robustness of composite indicator - a worked example

20. http://www.jrc.cec.eu.int/uasa20 -> Monte Carlo of score of country A minus score of country B. -> Country A generally better off…mostly due to the weight factors a and b Robustness of composite indicator a worked example b a

21. http://www.jrc.cec.eu.int/uasa21 -> Scatter-plot of score of country A minus score of country B (colours) as function of the two most important weights (red=negative, blue=zero, green=positive) Robustness of composite indicator a worked example country A – country B weight of 2 nd indicator weight of 1 st indicator

22. http://www.jrc.cec.eu.int/uasa22 How can I identify model structures in the simultaneous presence of several uncertainty sources ? Use of models in the scientific discourse … and role of uncertainty - sensitivity analysis

23. 23 ? ? Campolongo et al., 1999, JAC Saltelli et al., 1995, JAC

24. http://www.jrc.cec.eu.int/uasa24 Is the model-induced choice weak (non robust) because there is an insufficient number of observations, or because the experts cannot agree on an accepted theory? Use of models in the scientific discourse … and role of uncertainty - sensitivity analysis

25. http://www.jrc.cec.eu.int/uasa25 Useful inference versus falsification of the analysis Example: imagine the inference is Y = the logarithm of the ratio between the two pressure-on-decision indices (Tarantola et als. 2000). Y=Log(PI 1/PI 2) Region where Incineration Landfill is preferred Frequency of occurrence

26. http://www.jrc.cec.eu.int/uasa26 Useful inference versus falsification of the analysis

27. http://www.jrc.cec.eu.int/uasa27 What happens if I address the space of the policy options? Use of models in the scientific discourse … and role of uncertainty - sensitivity analysis

28. 28 Gauging the leverage of the policy options latitude

29. 29 Use of models in the scientific discourse … and role of uncertainty - sensitivity analysis A broader context of knowledge production and its quality as input to the policy process. Merging formal with informal tools. Ongoing work at the JRC with Angela and Silvio.

30. http://www.jrc.cec.eu.int/uasa30 © ULYSSES: De Marchi et. al, 1998.

31. http://www.jrc.cec.eu.int/uasa31 Conclusions The output from global Unc. A./Sens. A. can feed back into the extended peer review process via e.g. - refocusing of the critical issues, - re-assignment of weights for multiple criteria, or - of inference falsification (or otherwise) - identification of policy relevance/ irrelevance

32. http://www.jrc.cec.eu.int/uasa32 Further reading on SA Saltelli et als. Computer Physics Communications 2002, Saltelli et als. Statistical Science, 2000 Saltelli et als. JASA, 2000 (available here) Saltelli et al. Eds., Sensitivity Analysis, John Wiley & Sons publishers, Probability and Statistics series (buy it!)