1. Uncertainty assessment of the IMAGE/TIMER B1 CO 2 emissions scenario using the NUSAP method Jeroen P. van der Sluijs 1 Jose Potting 1 James Risbey 1 Detlef van Vuuren 2 Bert de Vries 2 Arthur Beusen 2 Peter Heuberger 2 Serafin Corral Quintana 3 Silvio Funtowicz 3 Penny Kloprogge 1 David Nuijten 1 Arthur Petersen 5 Jerry Ravetz 4 1 UU-STS 2 RIVM 3 EC-JRC 4 RMC 5 VUA j.p.vandersluijs@chem.uu.nl

2. Objective Develop a framework for uncertainty assessment and management addressing quantitative & qualitative dimensions test & demonstrate usefulness in IA models

3. IMAGE 2: Framework of models and Linkages World Economy (WorldScan) Population (Popher) Change in GDP, population & others (i.e. scenario assumptions) Land-use emissions Energy demand & supply (TIMER) Energy & industry emissions Land demand, use & cover Emissions & land-use changes Carbon cycle Atmospheric chemistry Concentration changes Climate (Zonal Climate Model or ECBilt) Climatic changes Natural systems Agricultural Impacts Water Impacts Land degradation Sea level rise FeedbacksFeedbacks Impacts

4. TIMER Model : five submodels Energy Demand (ED) Liquid Fuel supply (LF) Gaseous Fuel supply (GF) Electric Power Generation (EPG) Solid Fuel supply (SF) Population (POPHER) Inputs:Population, GDP capita -1, activity in energy sectors, assumptions regarding technological development, depletion and others. Outputs:End-use energy consumption, primary energy consumption. Fuel demand Prices Economy (WorldScan) Electricity demand

5. Main objectives TIMER To analyse the long-term dynamics of the energy system, and in particular changes in energy demand and the transition to non-fossil fuels within an integrated modeling framework; To construct and simulate greenhouse gas emission scenarios that are used in other submodels of IMAGE 2.2 or that are used in meta-models of IMAGE;

6. Key questions  What are key uncertainties in TIMER?  What is the role of model structure uncertainties in TIMER?  Uncertainty in which input variables and parameters dominate uncertainty in model outcome?  What is the strength of the sensitive parameters (pedigree)?

7. Location of uncertainty Input data Parameters Technical model structure Conceptual model sruct. /assumptions Indicators Problem framing System boundary Socio-political and institutional context

8. Sorts of uncertainty Inexactness Unreliability Value loading Ignorance

9. Inexactness Variability / heterogeneity Lack of knowledge Definitional vagueness Resolution error Aggregation error

10. Unreliability Limited internal strength in: –Use of proxies –Empirical basis –Theoretical understanding –Methodological rigour (incl. management of anomalies) –Validation Limited external strength in: –Exploration of rival problem framings –Management of dissent –Extended peer acceptance / stakeholder involvement –Transparency –Accessibility Future scope Linguistic imprecision

11. Value loading Bias –In knowledge production –Motivational bias (interests, incentives) –Disciplinary bias –Cultural bias –Choice of (modelling) approach (e.g. bottom up, top down) –Subjective Judgement –In knowledge utilization –Strategic/selective knowledge use Disagreement –about knowledge –about values

12. Ignorance System indeterminacy –open endedness –chaotic behavior Active ignorance –Model fixes for reasons understood –limited domains of applicability of functional relations –Surprise A Passive ignorance –Bugs (numerical / software / hardware error) –Model fixes for reasons not understood –Surprise B

13. Method  Checklist for model quality assistance  Meta-level analysis SRES scenarios to explore model structure uncertainties  Global sensitivity analysis (Morris)  NUSAP expert elicitation workshop to assess pedigree of sensitive model components  Diagnostic diagram to prioritise uncertainties by combination of criticality (Morris) and strength (pedigree)

14. Checklist Assist in quality control in complex models Not models are good or bad but ‘better’ and ‘worse’ forms of modelling practice Quality relates to fitness for function Help guard against poor practice Flag pittfalls

15. Checklist structure Screening questions Model & problem domain Internal strength Interface with users Use in policy process Overall assessment

16. Maria Message Aim Minicam ASF Image (Van Vuuren et al. 2000) (SRES scenarios reported to IPCC (2000) by six different modelling groups) B1-marker

17. Morris (1991) facilitates global sensitivity analysis in minimum number of model runs covers entire range of possible values for each variable parameters varied one step at a time in such a way that if sensitivity of one parameter is contingent on the values that other parameters may take, Morris captures such dependencies

18. Most sensitive model components: Population levels and economic activity Intra-sectoral structural change Progress ratios for technological improvements Size and cost supply curves of fossil fuels resources Autonomous and price-induced energy efficiency improvement Initial costs and depletion of renewables

19. NUS A P: Pedigree Evaluates the strength of the number by looking at: Background history by which the number was produced Underpinning and scientific status of the number

20. Parameter Pedigree Proxy Empirical basis Theoretical understanding Methodological rigour Validation

21. Proxy Sometimes it is not possible to obtain direct measurements or estimates of the parameter and so some form of proxy measure is used. Proxy refers to how good or close a measure of the quantity which we model is to the actual quantity we represent. An exact measure of the quantity would score four. If the measured quantity is not clearly related to the desired quantity the score would be zero.

22. Empirical basis Empirical quality typically refers to the degree to which direct observations are used to estimate the parameter. When the parameter is based upon good quality observational data, the pedigree score will be high. Sometimes directly observed data are not available and the parameter is estimated based on partial measurements or calculated from other quantities. Parameters determined by such indirect methods have a weaker empirical basis and will generally score lower than those based on direct observations.

23. Theoretical understanding The parameter will have some basis in theoretical understanding of the phenomenon it represents. This criterion refers to the extent en partiality of the theoretical understanding. Parameters based on well established theory will score high on this metric, while parameters whose theoretical basis has the status of crude speculation will score low.

24. Methodological rigour Some method will be used to collect, check, and revise the data used for making parameter estimates. Methodological quality refers to the norms for methodological rigour in this process applied by peers in the relevant disciplines. Well established and respected methods for measuring and processing the data would score high on this metric, while untested or unreliable methods would tend to score lower.

25. Validation This metric refers to the degree to which one has been able to cross-check the data against independent sources. When the parameter has been compared with appropriate sets of independent data to assess its reliability it will score high on this metric. In many cases, independent data for the same parameter over the same time period are not available and other datasets must be used for validation. This may require a compromise in the length or overlap of the datasets, or may require use of a related, but different, proxy variable, or perhaps use of data that has been aggregated on different scales. The more indirect or incomplete the validation, the lower it will score on this metric.

27. Elicitation workshop 18 experts (in 3 parallel groups of 6) discussed parameters, one by one, using information & scoring cards Individual expert judgements, informed by group discussion

30. Example result gas depletion multiplier Same data represented as kite diagram: Green = min. scores, Amber= max scores, Light green = min. scores if outliers omitted (Traffic light analogy) Radar diagram: Each coloured line represents scores given by one expert

31. Average scores (0-4) proxy 2½±½ empirical2±½ theory2±½ method 2±½ validation 1±½ valueladeness2½±1 competence 2±½

33. Conclusions (1)  Model quality assurance checklist proves quick scan to flag major areas of concern and associated pitfalls in the complex mass uncertainties.  Meta-level intercomparison of TIMER with the other SRES models gave us some insight in the potential roles of model structure uncertainties.

34. Conclusions (2)  Global sensitivity analysis supplemented with expert elicitation constitutes an efficient selection mechanism to further focus the diagnosis of key uncertainties.  Our pedigree elicitation procedure yields a differentiated insight into parameter strength.

35. Conclusions (3)  The diagnostic diagram puts spread and strength together to provide guidance in prioritisation of key uncertainties.

36. Conclusions (4) NUSAP method: can be applied to complex models in a meaningful way helps to focus research efforts on the potentially most problematic model components pinpoints specific weaknesses in these components

37. More information: www.nusap.net