2. Copernicus Institute Knowledge Quality Assessment rethinking uncertainty challenges for science and society Jeroen van der Sluijs Copernicus Institute for Sustainable Development and Innovation Utrecht University

3. Copernicus Institute Knowledge Quality Assessment Complex environmental risks Typical characteristics (Funtowicz & Ravetz): Decisions will need to be made before conclusive scientific evidence is available; Decision stakes are high: potential error costs of wrong decisions can be huge Values are in dispute Knowledge base is mixture of knowledge and ignorance:  large (partly irreducible) uncertainties, knowledge gaps, and imperfect understanding; Assessment dominated by models, scenarios, and assumptions Many (hidden) value loadings in problem frames, indicators, assumptions Coping with uncertainty is essential

4. Copernicus Institute Knowledge Quality Assessment Problematic definitions of uncertainty Example 1: Walker et al. 2003 “We adopt a general definition of uncertainty as being any departure from the unachievable ideal of complete determinism” Might make more sense to talk about unreality or uncomplexity as being any departure from or reduction of the inherent complexity of systems outside the controlled environment of the laboratory.

5. Copernicus Institute Knowledge Quality Assessment Definition from HarmoniCa uncertainty guidance document A person is uncertain if s/he lacks confidence about the specific outcomes of an event or action. Reasons for this lack of confidence might include a judgement of the information as incomplete, blurred, inaccurate or potentially false or might reflect intrinsic limits to the deterministic predictability of complex systems or of stochastic processes. Similarly, a person is certain if s/he is confident about the outcome of an event. It is possible that a person feels certain but has misjudged the situation (i.e. s/he is wrong).

6. Copernicus Institute Knowledge Quality Assessment The definition above defines uncertainty as a property (state of confidence) of the decision. Alternatively uncertainty can be defined as a property (state of perfection) of the total body of knowledge or information that is available at the moment of judgement. Uncertainty is then seen as an expression of the various forms of imperfection of the available information and depends on the state-of-the- art of scientific knowledge on the problem at the moment that the decision needs to be made (assuming that the decision maker has access to the state-of-the-art knowledge).

7. Copernicus Institute Knowledge Quality Assessment Challenges Increase society’s capacity to manage and surmount uncertainties surrounding knowledge production and use in designing and implementing precautionary (or should I say: responsible) policies and sustainable development New epistemology that does not see uncertainty as deviation from deterministic ideal, nor as imperfect knowledge, nor as low quality Need for a new (multidimensional) definition of uncertainty (maybe even a new word)

8. Copernicus Institute Knowledge Quality Assessment Insights on uncertainty Omitting uncertainty management can lead to scandals, crisis and loss of trust in science and institutions More research tends not to reduce uncertainty –Usually reveals unforeseen complexities –Meets irreducible uncertainty (intrinsic or practically) High quality  low uncertainty Quality relates to fitness for function (robustness, PP) In many complex problems unquantifiable uncertainties dominate the quantifiable uncertainty Shift in focus needed from reducing uncertainty towards systematic ways to explicitly cope with uncertainty and quality -> knowledge quality assessment

9. Copernicus Institute Knowledge Quality Assessment Uncertainty as a “monster” A monster is a phenomenon that at the same moment fits into two categories that were considered to be mutually excluding (Smits, 2002; Douglas 1966)

10. Copernicus Institute Knowledge Quality Assessment Cultural categories that we thought to be mutually exclusive and that now tend to get increasingly mixed up: knowledge – ignorance objective – subjective facts – values prediction – speculation science – policy

11. Copernicus Institute Knowledge Quality Assessment Responses to monsters Different degrees of tolerance towards the abnormal: monster-exorcism (expulsion) monster-adaptation (transformation) monster-embracement (acceptance) monster-assimilation (rethinking)

12. Copernicus Institute Knowledge Quality Assessment Footnote: compare to Lakatos (1976) preserving mathematical models against apparent refutations Surrender (throw the model away and start all again), Monster barring, Monster adjustment Lemma incorporation.

13. Copernicus Institute Knowledge Quality Assessment monster-exorcism Uncertainty causes discomfort Reduce uncertainties! Strong believe in “objective science”: “the puzzle can be solved” Example: “We are confident that the uncertainties can be reduced by further research” (IPCC 1990)

14. Copernicus Institute Knowledge Quality Assessment But…. For each head science chops off of the uncertainty monster, several new monster heads tend to pop up (unforeseen complexities) 1994 IGBP dropped objective to reduce uncertainty: ”full predictability of the earth system is almost certainly unattainable”

15. Copernicus Institute Knowledge Quality Assessment "We cannot be certain that this can be achieved easily and we do know it will take time. Since a fundamentally chaotic climate system is predictable only to a certain degree, our research achievements will always remain uncertain. Exploring the significance and characteristics of this uncertainty is a fundamental challenge to the scientific community." (Bolin, 1994) Former chairman IPCC on objective to reduce uncertainties:

16. Copernicus Institute Knowledge Quality Assessment Monster adaptation Fit the uncertainty monster back in the categories: purification Quantify uncertainty, subjective probability & Bayesian Tendency to build system models based on “objective science” and externalise the subjective parts and uncertainties into scenario’s and storylines Boundary work

17. Copernicus Institute Knowledge Quality Assessment IPCC 10 years after “we are confident that the uncertainties can be reduced…”

18. Copernicus Institute Knowledge Quality Assessment Monster adaptation meets its limits Different models fed with the same scenarios produce very different results “Integrated Assessment Modeling of Global Climate Change: Transparent Rational Tool for Policy Making or Opaque Screen Hiding Value-laden Assumptions?” (Steve Schneider)

19. Copernicus Institute Knowledge Quality Assessment Monster Embracement Uncertainty is welcomed: an appreciated property of life fascination about the unfathomable complexity of our living planet Gaia room for spirituality and wonder as counterweight to the engineering worldview of “managing the biosphere” Plea for a humble science Holism; Inclusive Science --------------------- Or: Uncertainty is welcomed because it fits well in other political agenda’s (strategic) Denial of realness of environmental risks by emphasizing all those uncertainties Manufacturing uncertainty

20. Copernicus Institute Knowledge Quality Assessment Monster Assimilation Rethink the categories by which the knowledge base is judged Create a place for monsters in the science policy interface Post Normal Science; Reflexive science; Complex systems research

21. Copernicus Institute Knowledge Quality Assessment Uncertainty has multiple dimensions Technical (inexactness) Methodological (unreliability) Societal (limited social robustness) Epistemological (ignorance)

22. Copernicus Institute Knowledge Quality Assessment Inexactness Intrinsic uncertainty: –Variability / heterogeneity Technical limitations: –Resolution error –Aggregation error –Unclear definitions

23. Copernicus Institute Knowledge Quality Assessment Unreliability Methodological limitations Limited internal strength in: –Use of proxies –Empirical basis –Methodological rigour –Validation Bias in knowledge production –Motivational bias (interests, incentives) –Disciplinary bias –Cultural bias –Choice of (modelling) approach –Subjective judgement Future scope

24. Copernicus Institute Knowledge Quality Assessment Limited social robustness Limited external strength in: –Bias / Value ladenness –Insufficient exploration of rival problem framings –Management of dissent –Extended peer acceptance / stakeholder involvement –Transparency –Access & availability –Intelligibility Strategic/selective knowledge use

25. Copernicus Institute Knowledge Quality Assessment Ignorance Epistemological limitations Limited theoretical understanding System indeterminacy –Open-endedness –Chaotic behavior –Intrinsic unknowability Active ignorance –Model fixes for reasons understood –Limited domains of applicability of functional relations –Numerical error –Surprise A Passive ignorance –Bugs (software error, hardware error, typos) –Model fixes for reasons not understood –Surprise B

26. Copernicus Institute Knowledge Quality Assessment

27. Copernicus Institute Knowledge Quality Assessment RIVM/MNP Uncertainty Guidance Systematic reflection on uncertainty and quality in: Problem framing Involvement of stakeholders Selection of indicators Appraisal of knowledge base Mapping and assessment of relevant uncertainties Reporting of uncertainty information

28. Copernicus Institute Knowledge Quality Assessment Detailed Guidance RIVM-MNP Uncertainty Guidance Quickscan Hints & Actions List Quickscan Questionnaire Mini-Checklist Downloads: www.nusap.net Reminder list Invokes Reflection Portal to QS Further Guidance Advice Hints & Implications Advice on Quantitative + Qualitative tools for UA Tool Catalogue for Uncertainty Assessment

29. Copernicus Institute Knowledge Quality Assessment

30. Copernicus Institute Knowledge Quality Assessment “Wisdom is to know, that you do not know” (Socrates)