1 Andrea Saltelli, European Commission, Joint Research Centre ECOINFORMATICS meeting US Environmental Protection Agency, Research Triangle Park, North Carolina, April 2008 Models, Uncertainty and Sensitivity

2 On uncertainty – 1 "That is what we meant by science. That both question and answer are tied up with uncertainty, and that they are painful. But that there is no way around them. And that you hide nothing; instead, everything is brought out into the open". Borderliners, Peter Høeg, Delta publisher, 1995

3 On uncertainty – 2 […] EPAs science panel found that quantitative evidence […] must … be characterized as having high uncertainties. What to do in the face of uncertainty is a policy question, not a scientific question. [..] The debate is about […] what kinds of uncertainty can be tolerated as a basis for decision-making. Hazy reasoning behind clean air David Goldston, Nature 452|3, April 2008 Science alone cant determine how regulations are written

4 Source: Scientific American, Jun2005, Vol. 292, Issue 6 How to play uncertainties in environmental regulation … On uncertainty – 3

5 - Fabrication (and politicisation) of uncertainty The example of the US Data quality act and of the OMB Peer Review and Information Quality which seemed designed to maximize the ability of corporate interests to manufacture and magnify scientific uncertainty.

6 About the OFFICE OF MANAGEMENT AND BUDGET (OMB) Proposed Risk Assessment Bulletin (January 9, 2006) OMB under attack by US legislators and scientists The aim is to bog the process down, in the name of transparency (Robert Shull). Source: Colin Macilwain, Safe and sound? Nature, 19 July Main Man. John Graham has led the White House mission to change agencies' approach to risk ibidem in Nature

7 The critique of models and what sensitivity analysis has to do with it

8 Jared Diamonds Collapse versus Michael Crichtons State of Fear

9 Rising sea level will threaten … cities of the United Kingdom (e.g. London), India Japan and the Philippines., p. 493.

10 Michael Crichton presents adversarial opinion on retreating glaciers and thickness of Antarctic ice cap – and contends that sea levels are not rising.

11 They talk as if simulation were real-world data. They re not. That s a problem that has to be fixed. I favor a stamp: WARNING: COMPUTER SIMULATION – MAY BE ERRONEOUS and UNVERIFIABLE. Like on cigarettes […] p. 556

12 For sure modelling is subject toady to an unprecedented critique, which is no longer limited to post-modern philosophers but involves intellectuals and scientists of different political hues. Have models fallen out of grace?

13 Useless Arithmetic: Why Environmental Scientists Can't Predict the Future by Orrin H. Pilkey and Linda Pilkey-Jarvis Quantitative mathematical models used by policy makes and government administrators to form environmental policies are seriously flawed

14 One of the examples discussed concerns the Yucca Mountain repository for radioactive waste disposal, where a very large model called TSPA (for total system performance assessment) is used to guarantee the safe containment of the waste. TSPA is Composed of 286 sub-models.

15 TSPA (like any other model) relies on assumptions -- a crucial one being the low permeability of the geological formation and hence the long time needed for the water to percolate from the desert surface to the level of the underground disposal. Evidence was produced which could lead to an upward revision of water permeability of 4 orders of magnitude (The 36 Cl story)

16 The narratives: How bad is the modeling that supports the Department of Energy's assertions about the safety and permanency of the Yucca Mountain nuclear waste dump? Execrable, according to legendary Duke University geologist Orrin Pilkey and his geologist daughter, Linda Pilkey-Jarvis, who works for the Washington state ecology department. Ken Maize Power Blog

17 We just cant predict, concludes N. N. Taleb, and we are victims of the ludic fallacy, of delusion of uncertainty, and so on. Modelling is just another attempt to Platonify reality … Nassim Nichola Taleb, The Black Swan, Penguin, London 2007

18 Many will disagree with Pilkey and Taleb. Yet, stakeholders and media alike expect instrumental use of models, amplification or dampening of uncertainty as a function of convenience and so on.

19 The IFPRI had raised about $460,000 for the modeling, which would have provided insights to help policymakers […] [… ] But Greenpeaces Haerlin and others objected that the models were not transparent. Source: Dueling visions for an hungry world, Erik Stokstad, 14 MARCH 2008, 319 SCIENCE

20 The critique of models The nature of models, after Rosen

21 The critique of models After Robert Rosen, 1991, World (the natural system) and Model (the formal system) are internally entailed - driven by a causal structure. [Efficient, material, final for world – formal for model] Nothing entails with one another World and Model; the association is hence the result of a craftsmanship. N Natural system F Formal system Decoding Entailment Encoding

22 The critique of models George M. Hornberger 1981 Hydrogeologist Naomi Oreskes 1994 Historian Jean Baudrillard 1999 Philosopher …

23 Other Newspaper headlines: Environmental institute lies and deceits Fuss in parliament after criticism on environmental numbers The bankruptcy of the environmental numbers Society has a right on fair information, RIVM does not provide it Just philosophy? Maybe not: A title during the RIVM media scandal (1999): RIVM over-exact prognoses based on virtual reality of computer models Jeroen van der Sluijs

24 Science for the post normal age is discussed in Funtowicz and Ravetz (1990, 1993, 1999) mostly in relation to Science for policy use. Jerry Ravetz Silvio Funtowicz

25 Post Normal Science Funtowicz and Ravetz, Science for the Post Normal age, Futures, 1993 Remark: on Post Normal Science diagram increasing stakes increases uncertainty

26 GIGO (Garbage In, Garbage Out) Science - where uncertainties in inputs must be suppressed lest outputs become indeterminate Jerry Ravetz

27 The critique of models Sensitivity Peter Kennedy, A Guide to Econometrics One of the ten commandments of applied econometrics according to Peter Kennedy: Thou shall confess in the presence of sensitivity. Corollary: Thou shall anticipate criticism

28 When reporting a sensitivity analysis, researchers should explain fully their specification search so that the readers can judge for themselves how the results may have been affected.

29 Definition. The study of how uncertainty in the output of a model (numerical or otherwise) can be apportioned to different sources of uncertainty in the model input. A related practice is `uncertainty analysis', which focuses rather on quantifying uncertainty in model output. The two should be run in tandem. Sensitivity

30 In sensitivity analysis: Type I error: assessing as important a non important factor Type II: assessing as non important an important factor Type III: analysing the wrong problem

31 Type III in sensitivity: Examples: In the case of TSPA (Yucca mountain) a range of 0.02 to 1 millimetre per year was used for percolation of flux rate. Applying sensitivity analysis to TSPA could or could not identify this as a crucial factor, but this would be of scarce use if the value of the percolation flux were later found to be of the order of 3,000 millimetres per year.

32 Prescriptions for sensitivity analysis EPAs 2004 guidelines on modelling Models Guidance Draft - November 2003 Draft Guidance on the Development, Evaluation, and Application of Regulatory Environmental Models Prepared by: The Council for Regulatory Environmental Modeling,

33 CREM Prescriptions for sensitivity analysis methods should preferably be able to (a)deal with a model regardless of assumptions about a models linearity and additivity; (b)consider interaction effects among input uncertainties; and (c)…an so on

34 CREM prescriptions are good. We at JRC works on practices that that take them into proper account. What these practices have in common the aspiration to tackle the curse of dimensionality.

35 Want to to know more? Buy our book! GLOBAL SENSITIVITY ANALYSIS. The primer John Wiley & Sons, 2008

36 Sensitivity analysis and the White House In the US the Proposed Risk Assessment Bulletin mentioned before also puts forward prescription for sensitivity analysis.

37 4. Standard for Characterizing Uncertainty Influential risk assessments should characterize uncertainty with a sensitivity analysis and, where feasible, through use of a numeric distribution […] Sensitivity analysis is particularly useful in pinpointing which assumptions are appropriate candidates for additional data collection to narrow the degree of uncertainty in the results. Sensitivity analysis is generally considered a minimum, necessary component of a quality risk assessment report. Source: OFFICE OF MANAGEMENT AND BUDGET Proposed Risk Assessment Bulletin (January 9, 2006)

38 The OMB about transparency

39 The primary benefit of public transparency is not necessarily that errors in analytic results will be detected, although error correction is clearly valuable. The more important benefit of transparency is that the public will be able to assess how much an agencys analytic result hinges on the specific analytic choices made by the agency. Concreteness about analytic choices allows, for example, the implications of alternative technical choices to be readily assessed. This type of sensitivity analysis is widely regarded as an essential feature of high-quality analysis, yet sensitivity analysis cannot be undertaken by outside parties unless a high degree of transparency is achieved. The OMB guidelines do not compel such sensitivity analysis as a necessary dimension of quality, but the transparency achieved by reproducibility will allow the public to undertake sensitivity studies of interest. Friday, February 22, 2002 Graphic - Federal Register, Part IX Office of Management and Budget Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; Notice; Republication

40 Conclusions: Role of sensitivity analysis (in a post-normal science context) Good practice and due diligence (e.g. test models, obtain parsimonious model representations …) Check if policy options are distinguishable given the uncertainties Contribute to the pedigree of the assessment. Falsify an analysis and or / make sure that you are not falsified.

41 Falsify the analysis (Popperian demarcation): *Scientific mathematical modelling should involve constant efforts to falsify the model (Pilkey and Pilkey Jarvis, op. cit.) ** Fight the white swan syndrome (Nassim N. Taleb, 2007)