Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Uncertainty Assessment and Communication Jeroen van der Sluijs Copernicus Institute for Sustainable Development and Innovation Utrecht University

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Copernicus Institute Research topics Energy and material demands and efficiencies (Dr. Martin Patel) Possibilities for a more sustainable energy supply (Dr. Andre Faaij) Land use, biodiversity and ecosystem functioning (Prof. Dr. Peter de Ruiter) Innovation: systems, processes and policies (Prof. Dr. Ruud Smits) Governance for sustainable development (Prof. Dr. Pieter Glasbergen) Integrative models and methods and the management of risks and uncertainties (Dr. Jeroen van der Sluijs)

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Complex environmental risks Typical characteristics (Funtowicz & Ravetz): Decisions will need to be made before conclusive scientific evidence is available; Decision stakes are high: potential error costs of wrong decisions can be huge Values are in dispute Knowledge base is mixture of knowledge and ignorance:  large (partly irreducible) uncertainties, knowledge gaps, and imperfect understanding; Assessment dominated by models, scenarios, and assumptions Many (hidden) value loadings in problem frames, indicators, assumptions Coping with uncertainty is essential

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 RIVM / De Kwaadsteniet (1999) “RIVM over-exact prognoses based on virtual reality of computer models” Newspaper headlines: Environmental institute lies and deceits Fuss in parliament after criticism on environmental numbers The bankruptcy of the environmental numbers Society has a right on fair information, RIVM does not provide it

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Once environmental numbers are thrown over the disciplinary fence, important caveats tend to be ignored, uncertainties compressed and numbers used at face value e.g. Climate Sensitivity, see Van der Sluijs, Wynne, Shackley, 1998: °C ? ! Crossing the disciplinary boundaries Resulting misconception: Worst case = 4.5°C

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 The certainty trough (McKenzie, 1990)

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 In model based assessment and foresight of complex environmental problems, unquantifiable uncertainties dominate the quantifiable ones Unquantifiable uncertainties include those associated with problem framings system boundaries model structures assumptions indeterminacies value ladenness

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Insights on uncertainty Uncertainty is partly socially constructed and its assessment always involves subjective judgement Omitting uncertainty management can lead to scandals and crisis More research does not necessarily reduce uncertainty –may reveal unforeseen complexities –irreducible uncertainty (intrinsic or practically) High quality  low uncertainty Quality relates to fitness for function (robustness, PP) Shift in focus needed from reducing uncertainty towards systematic attempts to explicitly cope with uncertainty

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Dimensions of uncertainty Technical (inexactness) Methodological (unreliability) Epistemological (ignorance) Societal (limited social robustness)

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 RIVM Uncertainty Guidance Goals: Structure an approach to environmental assessment that facilitates an awareness, identification and incorporation of uncertainty Specifically address and relate the role of uncertainties in the context of policy advice May not reduce uncertainties, but provides means to assess their potential consequences and avoid pitfalls associated with ignoring or ignorance of uncertainties Guidance for use and help against misuse of uncertainty tools Provide useful uncertainty assessments Promote the adoption of uncertainty methods Facilitate design of effective strategies for communicating uncertainty.

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Reporting Problem framing & context analysis Process assessment Environmental Assessment methods Uncertainty identification & prioritization Uncertainty Analysis Uncertainty Management Review & Evaluation Communication Steps in uncertainty management

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Systematic reflection on uncertainty issues in: Problem framing Involvement of stakeholders Selection of indicators Appraisal of knowledge base Mapping and assessment of relevant uncertainties Reporting of uncertainty information

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Detailed Guidance RIVM-MNP Uncertainty Guidance Quickscan Hints & Actions List Quickscan Questionnaire Mini-Checklist Download all volumes: Reminder list Invokes Reflection Portal to QS Further Guidance Advice Hints & Implications Advice on Quantitative + Qualitative tools for UA Tool Catalogue for Uncertainty Assessment

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Problem framing and context Explore rival problem frames Relevant aspects / system boundary Typify problem structure Problem lifecycle / maturity Role of study in policy process Uncertainty in socio-political context

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Type-III error: Assessing the wrong problem by incorrectly accepting the false meta-hypothesis that there is no difference between the boundaries of a problem, as defined by the analyst, and the actual boundaries of the problem (Dunn, 1997). Context validation (Dunn, 1999). The validity of inferences that we have estimated the proximal range of rival hypotheses. Context validation can be performed by a participatory bottom-up process to elicit from scientists and stakeholders rival hypotheses on causal relations underlying a problem and rival problem definitions.

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 In different phases of problem lifecycle, different uncertainties are salient Different problem-types need different uncertainty management strategies

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Involvement of stakeholders Identify relevant stakeholders. Identification of areas of agreement and disagreement among stakeholders on value dimensions of the problem. Recommendations on when to involve different stakeholders in the assessment process.

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Indicators How well do indicators used address key aspects of the problem? Alternative indicators? Limitations of indicators used? Controversies in science and society about these indicators?

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Adequacy of available knowledge base? What are strong and weak points in the knowledgebase? –Use of proxies, empirical basis, theoretical understanding, methodological rigor, validation NUSAP Pedigree analysis What parts of the knowledge are contested (scientific and societal controversies)? Is the assessment feasible in view of available resources? (limitations implied)

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Example Pedigree matrix parameter strength

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Example Pedigree results Trafic-light analogy 2.6 green This example is the case of VOC emissions from paint in the Netherlands, calculated from national sales statistics (NS) in 5 sectors (Ship, Building & Steel, Do It Yourself, Car refinishing and Industry) and assumptions on additional thinner use (Th%) and a lump sum for imported paint and an assumption for its VOC percentage. See full research report on for details.

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Sensitivity Pedigree weakstrong low high NUSAP Diagnostic Diagram Danger zone Safe zone Combine results from sensitivity analysis and pedigree analysis by plotting each model parameter in this diagram. This reveals the parameters whose uncertainty is most problematic.

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 VOC% imp.paint Thin % Ind Overlap VVVF/CBS imp NUSAP Diagnostic Diagram Imp. Below threshold Thin.% DIY-rest Gap VVVF-RNS Thin.% Car NS Decor NS Ind Imp. Paint NS DIY NS Car NS Ship Th.% decor Example result from the case of emission monitoring of VOC from paint in NL. The most problematic parameter is the assumed VOC percentage in imported paint.

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Mapping and prioritization of relevant uncertainties Highlight uncertainties in typology relevant to this problem Set priorities for uncertainty assessment Select uncertainty assessment tools from the tool catalogue

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Typology of uncertainties Location Level of uncertainty statistical uncertainty, scenario uncertainty, recognised ignorance Nature of uncertainty knowledge-related uncertainty, variability-related uncertainty Qualification of knowledge base (backing) weak, fair, strong Value-ladenness of choices small, medium, large

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Locations of uncertainties: Context ecological, technological, economic, social and political representation Expert judgement narratives, storylines, advices Model model structure, technical model, model parameters, model inputs Data measurements, monitoring data, survey data Outputs indicators, statements

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Tool catalogue For each tool: Description Goals and use What sorts and locations of uncertainty does this tool address? What resources are required to use it? Strengths and limitations guidance on application & complementarity Typical pitfalls of each tool References to handbooks, example case studies, web-sites, experts etc.

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Tool catalogue Sensitivity Analysis Error propagation Monte Carlo NUSAP Expert Elicitation Scenario analysis PRIMA Checklist model quality assistance Assumption analysis …...

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Reporting Make uncertainties explicit Assess robustness of results Discuss implications of uncertainty findings for different settings of burden of proof Relevance of results to the problem Progressive disclosure of information -> traceability and backing

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 IPCC WGI Proposal for Interpretation and Use of Probabilistic Terms

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Weiss 2003 uncertainty scale 10. Beyond any doubt 9. Beyond a reasonable doubt 8. Clear and Convincing Evidence 7. Clear Showing 6. Substantial and credible evidence 5. Preponderance of the Evidence 4. Clear indication 3. Probable cause: reasonable grounds for belief 2. Reasonable, articulable grounds for suspicion 1. No reasonable grounds for suspicion 0. Impossible

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 Triggers that increase policy relevance of uncertainty Close to a norm or target Near a threshold of severe impact On steep part of cost-curve or impact curve Low pedigree High (potential) valueladenness ‘temperatute’ of scientific or societal controversies

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 More information:

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 References: Funtowicz, S.O. and J.R. Ravetz, Uncertainty and Quality in Science for Policy. Kluwer, 229 pp., Dordrecht, James Risbey, J.P. van der Sluijs and J. Ravetz, Quality assistance in environmental modelling, Environmental Modelling and Assessment, forthcoming Van der Sluijs, Jeroen P., A way out of the credibility crisis of models used in integrated environmental assessment, Futures, Vol. 34, 2002, pp Jeroen van der Sluijs, Matthieu Craye, Silvio Funtowicz, Penny Kloprogge, Jerry Ravetz, and James Risbey Combining Quantitative and Qualitative Measures of Uncertainty in Model based Environmental Assessment: the NUSAP System, Risk Analysis, forthcoming Walker, W., Harremoës, P., Rotmans, J., Van der Sluijs, J., Van Asselt, M., Jansen, P., Krayer von Krauss, M.P., Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support. Journal of Integrated Assessment, 4 (1) Weiss, C., Scientific Uncertainty and Science Based Precaution. International Environmental Agreements: Politics, Law and Economics 3, 2003, p Weiss, C., Expressing Scientific Uncertainty. Law, probability and risk, 2003, 2, p The four volumes of the RIVM uncertainty guidance: J.P. van der Sluijs, J.S. Risbey, P. Kloprogge, J.R. Ravetz, S.O. Funtowicz, S.Corral Quintana, Â Guimarães Pereira, B. De Marchi, A.C. Petersen, P.H.M. Janssen, R. Hoppe, and S.W.F. Huijs. RIVM/MNP Guidance for Uncertainty Assessment and Communication: Detailed Guidance Utrecht University & RIVM, 2003 (available from: A.C. Petersen, P.H.M. Janssen, J.P. van der Sluijs, J.S. Risbey, J.R. Ravetz RIVM/MNP Guidance for Uncertainty Assessment and Communication: Mini-Checklist & Quickscan Questionairre.. RIVM/MNP; ISBN , 2003 (available from: P.H.M. Janssen, A.C. Petersen, J.P. van der Sluijs, J.S. Risbey, J.R. Ravetz. RIVM/MNP Guidance for Uncertainty Assessment and Communication: Quickscan Hints & Actions List. RIVM/MNP, ISBN , 2003 (available from: J.P. van der Sluijs, P.H.M. Janssen, A.C. Petersen, P. Kloprogge, J.S. Risbey, W. Tuinstra, M.B.A. van Asselt, J.R. Ravetz, RIVM/MNP Guidance for Uncertainty Assessment and Communication: Tool Catalogue for Uncertainty Assessment Utrecht University & RIVM,, 2004 (available from: Most refs available on request as pdf file, just mail me at