1. Quantitative methods to manage uncertainty in science by
Andrea Saltelli, Stefano Tarantola and Michela Saisana, Joint Research Centre of the European Communities in Ispra (I),
Mini-symposium “The management of uncertainty in risk science and policy”, World Congress on Risk
Brussels, June 2003.

2. Rosen’s formalisation of the modelling process
Models mimic systems
Rosen’s formalisation of the modelling process

3. Models mimic systems (Rosen)
“World” (the natural system) and “Model” (the formal system) are internally entailed - driven by a causal structure. Nothing entails with one another “World” and “Model”; the association is hence the result of a craftsmanship.
But this does not apply to natural systems only: give 10 engineers the blueprint of the same plant and they will return you 10 model based risk assessments for the same plant.

4. Models mimic systems (Rosen)
It can help the craftsman that the uncertainty in the information provided by the model (the substance of use for the decoding exercise) is carefully apportioned to the uncertainty associated with the encoding process.

5. Models maps assumptions onto inferences ... but often too narrowly
<<[…] most simulation models will be complex, with many parameters, state-variables and non linear relations. Under the best circumstances, such models have many degrees of freedom and, with judicious fiddling, can be made to produce virtually any desired behaviour, often with both plausible structure and parameter values.>>, HORNBERGER and Spear (1981)
<<Cynics say that models can be made to conclude anything provided that suitable assumptions are fed into them.>>, The Economist, 1998.
KONIKOV and Bredehoeft,  Oreskes et al

6. Use of models in the scientific discourse
But yet models are used ...
… and a legitimate question is the following:
“If we had mapped the space of uncertain assumptions honestly and judiciously, would the space of inference still be of use1?”
1Read: do we still have peak around some useful inference (e.g. YES or NO, safe or unsafe, hypothesis accepted or rejected, policy effective or ineffective etc. ) or do we have as many YES as NO etc.?

7. Models maps assumptions onto inferences …
<<I have proposed a form of organised sensitivity analysis that I call “global sensitivity analysis” in which a neighborhood of alternative assumptions is selected and the corresponding interval of inferences is identified. Conclusions are judged to be sturdy only if the neighborhood of assumptions is wide enough to be credible and the corresponding interval of inferences is narrow enough to be useful.>>
Leamer, “Sensitivity Analysis would help”, 1990

8. Leamer’s view of global Sensitivity Analysis (SA)
Models maps assumptions onto inferences …
Leamer’s view of global Sensitivity Analysis (SA)
Space of estimated parameters
Simulation
inference
Space of plausible models space
...
Other assumptions

9. Models maps assumptions onto inferences …
(Parametric bootstrap version of UA/SA )
Model
Input data
(Estimation)
Estimated parameters
(Parametric bootstrap: we sample from the posterior parameter probability)
Uncertainty and sensitivity analysis
Inference

10. Estimation of parameters
Bootstrapping-of-the-modelling-process version of UA/SA, after Chatfield, 1995
Model
(Model Identification)
Loop on boot-replica of the input data
(Estimation)
Estimation of parameters
(Bootstrap of the modelling process)
Inference

11. Posterior of Parameters
Bayesian Uncertainty and Sensitivity Analysis
(Draper 1995, Planas and Depoutot 2000)
Posterior of Model(s)
Prior of Model(s)
Prior of Model
Data
(Sampling)
Inference
Prior of Parameters
Posterior of Parameters

12. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
The space of the model induced choices (the inference) swells and shrinks by our swelling and shrinking the space of the input assumptions. How many of the assumptions are relevant at all for the choice? And those that are relevant, how do they act on the outcome; singularly or in more or less complex combinations?

13. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
I desire to have a given degree of robustness in the choice, what factor/assumptions should be tested more rigorously? (=> look at how much “fixing” any given f/a can potentially reduce the variance of the output)
Can I confidently “fix” a subset of the input factors/assumptions? The Beck and Ravetz “relevance” issue. How do I find these factors?

14. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
“Reduced” variance
Expected reduced variance – it is small if the factor is important.

15. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Big if factor important
Small if factor important
First order effect

16. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Also used is the total effect term:
This is the expected fractional value of the variance that would be left if all factor but Xi were fixed.
The use of different sensitivity measures should be seen as the answer to a rigorous question concerning the relative importance of input factors.

17. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
One can thus relate the total effect term
to a question relative to the possibility to fix factor(s), (Factor Fixing Setting),
while the first order effect
frames into the Factors’ Prioritisation Setting.

18. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Other setting (questions) can easily be imagined. Settings to frame the uncertaitny and sensitivity analyses are crucial.
The alternative would be to have different SA methods suggesting different factors relative imortance.
Settings should be audited! = Let us agree on what “importance” means before we engage in the analysis.

19. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Is the model-induced choice weak (non robust) because there is an insufficient number of observations, or because the experts cannot agree on an accepted theory?

20. Region where Region where Incineration Landfill
Useful inference versus falsification of the analysis
Example: imagine the inference is Y = the logarithm of the ratio between the two pressure-on-decision indices (Tarantola et als. 2000).
Region where Region where
Incineration Landfill
is preferred is preferred
Frequency of occurrence
Y=Log(PI 1/PI 2)

21. Useful inference versus falsification of the analysis

22. Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
What happens if I address the space of the policy options?

23. Gauging the leverage of the policy options latitude

24. - refocusing of the critical issues/factor,
Conclusions
The output from global uncertainty and sensitivity analyses can feed back into the extended peer review process via e.g.
- refocusing of the critical issues/factor,
- (re-assignment of weights for multiple criteria, or)
- inference falsification
identification of policy relevance/ irrelevance
Note: EC Guidelines for Extended Impact Assessment inlcude explicit and detailed indication for global SA!

25. References
ROSEN R., Life Itself - A Comprehensive Inquiry into Nature, Origin, and Fabrication of Life. Columbia University Press 1991.
HORNBERGER G.M., and R. C. Spear (1981) An approach to the preliminary analysis of environmental systems. Journal of Environmental management, 12, 7-18.
KONIKOV and Bredehoeft, 1992, "Groundwater models cannot be validated" Advances in Water Resources 15(1),
ORESKES, N. , Shrader-Frechette K., Belitz, K., 1994, Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences, SCIENCE, 263,
Edward E. Leamer, “Sensitivity Analysis would help”, in Modelling Economic Series, Edited by CWJ Granger, 1990, Clarendon Press, Oxford.
CHATFIELD C., Model uncertainty , data mining and statistical inference,
J. R. Statist. Soc. A, 158 (3) , , 1993

26. A forum - http://sensitivity-analysis.jrc.cec.eu.int/
Further reading on SA
Papers - Saltelli et als., Statistical Science, 2000; Saltelli and Tarantola, JASA, 2002
Book - Saltelli et al. Eds., Sensitivity Analysis, 2000, John Wiley & Sons publishers, Probability and Statistics series
Book - A primer (Sensitivity Analysis in Practice) will appear by end 2003, with Wiley.
A forum -
Presentations of the mini-symposium on