1. Universiteit Utrecht Copernicus Institute “Useless arithmetic” or “the best of our knowledge”? Does probabilistic risk assessment of long- term geological storage of CO 2 make sense? Dr. Jeroen van der Sluijs, Ferhat Yavuz MSc, Joris Koorneef MSc, and Prof Dr. Wim Turkenburg Presentation at the 3rd Risk Assessment Network Meeting, organised by IEA Greenhouse Gas R&D Programme, London 15-16 August 2007 Copernicus Institute for Sustainable Development and Innovation Utrecht University

2. Copernicus Institute Universiteit Utrecht Pilkey & Pilkey, 2007 book

3. Copernicus Institute Universiteit Utrecht Yucca Mountain: bizarre mismatch Regulatory standard implied need for scientific certainty for up to one million years State of knowledge –limitations of a quantitative modeling approach (US-DOE’s Total System Performance Assessment, TSPA) –radical uncertainty and ignorance –uncontrolled conditions of very long term unknown and indeterminate future. Ignorance: Percolation flux: TSPA model assumed 0.5 mm per year (expert guess) Elevated levels of Chlorine-36 isotope in faults uncovered by tunnel boring: percolation flux > 3000 mm per year over the past 50 yr...

4. Copernicus Institute Universiteit Utrecht Bow Tie approach Consequences Top Event Threads Hazard modified from http://nmishrag.mishc.uq.edu.au/NMISHRAG_Chapter4_4.1.5.asp

5. Copernicus Institute Universiteit Utrecht Probabilistic risk analysis—sequence of analysis steps Source: Kirchsteiger, 1999

6. Copernicus Institute Universiteit Utrecht Vulnerable objects (housing, schools, hospitals, etc) <10 -6 per year (area A) Less vulnerable objects < 10 -5 per year (area B) NL Acceptability criteria for individual risk NL External Safety The individual risk for a point-location around a hazardous activity: probability that an average unprotected person permanently present at that point location, would get killed due to an accident at the hazardous activity. Bottelberghs 2000, Journal of Hazardous Materials 71, 59–84.

7. Copernicus Institute Universiteit Utrecht F N Example of a societal risk curve plot (F,N plot) societal risk: Probability that a group of more than N persons would get killed due to an accident at the hazardous activity N = number of lethal victims; F = probability per year for an accident at the hazardous activity that would cause >N victims.

8. Copernicus Institute Universiteit Utrecht Strengths of PRA Integrative and quantitative approach Allows ranking of issues and results, explicit treatment of uncertainties, and optimisation Can be used to both enhance safety and manage operability. Results and decisions can be communicated on a clearly defined basis Its use is beneficial even if the models generated are not (fully) quantified Lack of accuracy of the data does not hamper the use of probabilistic approaches as comparative tools to rank alternatives

9. Copernicus Institute Universiteit Utrecht Weaknesses of PRA complex, time consuming, data-intensive unavoidably requires mixtures of ‘subjective’ (expert judgement) and ‘objective’ data (observations, measurements) - limits scientific rigor of result - feels uncomfortable large potential for misunderstanding of scientific status of the outcomes – undue sense of certainty – pitfall of “quasi precision” models of open (uncontrolled) systems can never be validated, only ‘confirmed’ by non-contradiction between observation and prediction (Oreskes et al. 1994) dangers of too early standardization & benchmarking (anchoring bias)

10. Copernicus Institute Universiteit Utrecht PRA of geological CO 2 storage versus PRA of industrial installations: Natural reservoir much less defined and way more heterogeneous Reservoir is not an engineered system >> time horizon The longer the time horizon, the more open the system is >> stored volume of substance << past experience >> dependency on expert judgement in specific case of CO 2 storage all general weaknesses of PRA are amplified...

11. Copernicus Institute Universiteit Utrecht 3 paradigms of uncertain risks 'deficit view' Uncertainty is provisional Reduce uncertainty, make ever more complex models Tools: quantification, Monte Carlo, Bayesian belief networks 'evidence evaluation view' Comparative evaluations of research results Tools: Scientific consensus building; multi disciplinary expert panels focus on robust findings 'complex systems view / post-normal view' Uncertainty is intrinsic to complex systems Uncertainty can be result of production of knowledge Acknowledge that not all uncertainties can be quantified Openly deal with deeper dimensions of uncertainty (problem framing indeterminacy, ignorance, assumptions, value loadings, institutional dimensions) Tools: Knowledge Quality Assessment Working deliberatively within imperfections

12. Copernicus Institute Universiteit Utrecht Dimensions of uncertainty Technical (inexactness) Methodological (unreliability) Epistemological (ignorance) Societal (limited social robustness)

13. Copernicus Institute Universiteit Utrecht Qualified Quantities: NUSAP: Numeral, Unit, Spread, Assessment, Pedigree Assessment expresses expert judgement on the unreliability Pedigree evaluates the strength of a number by looking at: Background history by which the number was produced Underpinning and scientific status of the number

14. Copernicus Institute Universiteit Utrecht Example pedigree matrix for model parameters

15. Copernicus Institute Universiteit Utrecht Model Quality Assessment Models are tools, not truths A model is not good or bad but there are ‘better’ and ‘worse’ forms of modelling practice Models are ‘more’ or ‘less’ useful when applied to a particular problem. Model Quality Assessment can provide: insurance against pitfalls in process insurance against irrelevance in application refs: www.mnp.nl/guidance Risbey, J., J. van der Sluijs, et al. (2005): Application of a Checklist for Quality Assistance in Environmental Modelling to an Energy Model. Environmental Modeling & Assessment 10 (1), 63-79.

16. Copernicus Institute Universiteit Utrecht Valid uses of PRA of geological CO 2 storage: Comparative assessment of different reservoirs and storage options “Validation” of simpler methods Gain insight in key-characteristics that determine reservoir safety Gain insight in what factors should be monitored for early detection of leakage risks Improvement of operational practices Support of safer designs Informed debate with regulators and society (but it is essential to make pedigree of results explicit!)

17. Copernicus Institute Universiteit Utrecht Tricky and invalid uses of PRA of geological CO 2 storage Invalid: Demonstration of safety Interpreting outcomes as absolute Tricky: Demonstration of compliance to a quantified safety requirement Comparison to other (e.g. industrial) risks

18. Copernicus Institute Universiteit Utrecht Uncertainty and model complexity

19. Copernicus Institute Universiteit Utrecht Casman et al. 1999: Mixed levels of uncertainty Risk Analysis, 1999, 19 (1), 33-42

20. Copernicus Institute Universiteit Utrecht High uncertainty is not the same as low quality, but..... methodological uncertainty of choice of (risk) indicator can be dominant (Example taken from Saltelli et al., 2000 book “Sensitivity Analysis”)

21. Copernicus Institute Universiteit Utrecht Conclusions (1) Specific characteristics of CO 2 storage amplify all generic weaknesses of PRA Strong dependence on expert judgement Need for systematic reflection on knowledge quality Need for systematic elicitation and documentation of ARGUMENTS behind each judgment by each expert Be very open and very transparent about uncertainty and pedigree of results Be explicit about all assumptions on which outcomes are conditioned Avoid mismatch between regulatory requirements and the limited level of rigor that state-of-the-art science can realistically achieve

22. Copernicus Institute Universiteit Utrecht Alternatives for regulation Precautionary Principle (1)measures that constrain the possibility of the harm to occur (2) measures that contain the harm (c.q. increase the controllability of the harm) when it would occur –Flexible standards: Step by step, case by case approach –First decades off-shore only? –Availability of control measures/remediation –Reversibility? Maximum Credible Accident approach?