1. Exploratory Modelling and Analysis Jan Kwakkel Erik Pruyt 1 an approach for model-based foresight under deep uncertainty

2. Exploratory Modeling and Analysis (EMA) Exploratory modeling is using computational experiments to assist in reasoning about systems where there is significant (or deep) uncertainty (Bankes, 1993)* EMA was developed at the RAND Corporation EMA represents a new way of thinking about the use of computer models to support policy making Traditional modeling consists of consolidating known facts about a system that are then used as a surrogate for the system when confronted with uncertainties about details or mechanisms, modelers use educated guesses (resulting in best estimate predictive models). 2 *BANKES, S. 1993. Exploratory Modeling for Policy Analysis, Operations Research, 43 (3), p. 435-449.

3. Predictive Modeling vs. Exploratory Modeling Exploratory Modeling Model is used as a hypothesis generator (what if...) Take into account external (scenario) uncertainty, structural (model) uncertainty, and uncertainty about valuation of outcomes The objective is to reason about system behavior: under which circumstances would a policy succeed or fail? Uses rapid assessment models, because the uncertainties may swamp model results Predictive Modeling Model is used to predict Take into account (external) uncertainty; deal with internal uncertainty using educated guesses The objective is to predict system behavior and whether a policy will succeed or fail Aims at detailed models that capture the state of the art 3

4. EMA Approach Specify policy problem Analyze the uncertainties associated with the problem Develop one or more fast and simple models consistent with the available information and knowledge that allow for exploring the specified uncertainties Explore the behavior of these models across the ranges of the uncertainties Assess the implications of the exploration for policy Iteratively modify plan in light of revealed weaknesses until a satisfying plan emerges 4

5. Explore the parameter space Exploration versus directed search Exploration (sampling techniques) Factorial methods Monte Carlo sampling Latin Hypercube sampling Directed search (optimization techniques) Conjugant Gradient Optimization Genetic Algorithms Simulated Annealing Etc. 5

6. Mineral Scarcity Problem Many crucial high-volume minerals are expected to become exhausted in the coming decades The disparity between the expected exponential growth of metal demand and the expected limited growth of metal supply may result in temporary and/or chronic scarcity; and Strategic and/or speculative behavior of countries that have a quasi- monopoly on the extraction of (rare earth) metals may seriously hinder the transition of modern societies towards more sustainable ones. The asynchronous dynamics of supply and demand, aggravated by reinforcing behaviors and knock-on effects, is a breeding ground for acute and/or chronic crises What kinds of dynamics can happen? 6

7. System Dynamics Model 7 PRUYT, E. 2010. Scarcity of Minerals and Metals: A Generic Exploratory System Dynamics Model. In: MOON, T. H. (ed.) The 28th International Conference of the System Dynamics Society. Seoul, Korea.

8. Uncertainties to be explored Parametric variations Lifetime of mines Lifetime of recycling facilities Initial values for most stock variables Price elasticity and desired profit margins Order of time delays Building time of mines, recycling capacity Non linear relations captured in table functions Learning effect Impact shortages on price Substitution behaviour In total, 27 uncertainties are jointly explored 8

9. Results for 100 LHS runs 9

10. Results Number of runs Number of behavior patters 1000371 50001214 100002042 150002742 200003386 250003894 300004547 350004976 400005511 450005972 500006404 Behavioral clustering of time series Each run is specified as a concatenation of atomic behavior patters 10

11. Airport Planning Problem Schiphol Airport is a environmentally constrained airport It loses demand to other airport Low cost and charter to regional airports Long haul and transfer to other hubs in Europe How can the airport invest to remain competitive, despite a wide variety of uncertainties? 11

12. A Fast and Simple Model for Calculating Airport Performance A variety of tools are readily available for aspects of airport performance (e.g. noise, emissions, capacity) Uncertainty about the airport system itself is low The Fast and Simple Component integrates – FAA capacity tool (FCM) – FAA emissions tool (EDMS) – FAA noise tool (AEM) – NATS external safety methodology Outcomes: – Ratio capacity to demand, latent demand, size of noise contour, average casualty expectancy, emissions

13. Uncertainties to be explored The main uncertainties faced by airports come from external forces We developed generators for key external forces: –Engine technology (exponential and logistic performance increase) –Air Traffic Management technology (exponential and logistic performance increase) –Population (logistic growth, logistic growth followed by logistic decline) –Aviation Demand (exponential, logistic, decline) –Composition of fleet (logistic change, linear change) –Weather (parametric uncertainty) Together, these uncertainties result in 48 structurally different scenario generators, each of which can generate an infinite range of quantitatively different scenarios

14. Results Basic plan Try to limit movements to 510.000 in 2020 From 2015 move up to 70.000 movements to regional airport In 2020, a new runway should become operational What is the bandwidth of outcomes for this plan given the uncertainties? Approach: Conjugant gradient optimization across all the uncertainties Multiple different initializations for each optimization to handle local vs. global optima Time required: roughly a week of computer simulations on a normal desktop PC 14

15. Performance Bandwidth of Basic Plan The basic plan has a very wide bandwidth Plan unsuccessful in guiding the future development of the airport Overinvestment in runways Unnecessary moving of operations to regional airports How to improve the plan? Introduce flexibility Specify the conditions under which pre-specified actions are taken E.g. build runway only if there is a certain level of demand or certain deterioration in capacity do to wind 15 Basic plan Noise13 – 64 km 2 Emissions2,1 – 19,6 ton CO External Safety 0,9 – 2,7 ACE Ratio capacity versus demand 0,3 – 2,5

16. Performance Bandwidth of Adaptive Plan The adaptive plan significantly reduces the bandwidth of outcomes on the shown indicators across the same uncertainties How big is the difference in performance between the two plans? Are there regions were the initial static plan is still better? 16 Initial Static PlanAdaptive Plan Noise13 – 64 km 2 10 – 47 km 2 Emissions2,1 – 19,6 ton CO1,9 – 10,3 ton CO External Safety0,9 – 2,7 ACE1,1 – 2,3 ACE Ratio capacity versus demand 0,3 – 2,50, 9 – 1,1

17. Results 17

18. Results 18 Static plan performs better

19. Results 19 Modify the adaptive plan to deal with these regions

20. Concluding remarks Deep uncertainty has to be addressed explicitly in any long-term decision making problem EMA offers a useful technique that allows the utilization of models to explore the implications of the uncertainties EMA can be used to develop dynamic adaptive strategies capable of coping with the multiplicity of plausible futures Research is needed Visualizing and analyzing results of exploration Communication of results to clients Efficient techniques for both directed searches and open exploration 20

21. 21

22. DEEP UNCERTAINTY 22

23. What is uncertainty? Any departure from the unachievable ideal of complete determinism. Uncertainty is not simply a lack of knowledge, since an increase in knowledge might lead to an increase of knowledge about things we dont know and thus increase uncertainty (Walker et al. 2003)* Sources of uncertainty can be specified according to their nature, location and level Nature: character of the uncertainty Limited knowledge, inherent variability, ambiguity Location: which aspect of the system or model are we uncertain about Inputs, model structure, outputs, valuation of outputs Level: degree of uncertainty Ranges from complete certainty to absolute ignorance 23 *WALKER, W. E., HARREMOËS, J., ROTMANS, J. P., VAN DER SLUIJS, J. P., VAN ASSELT, M. B. A., JANSSEN, P. H. M. & KRAYER VON KRAUSS, M. P. 2003. Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model- Based Decision Support. Integrated Assessment, 4, p. 5-17.

24. Deep Uncertainty Situation where the relevant actors do not know or cannot agree on (Aspects of) how the system works How likely or plausible various (paths to) future states are How to value the various outcomes of interest In almost all long-term decision making problems, deep uncertainty is encountered e.g. the climate change debate Few techniques are readily available for offering decision support Decision making should be based on robustness instead of optimality Robustness is to be achieved in part through adaptiveness Only take near term actions that overall have desirable consequences Prepare actions to be taken in light of how the future enfolds 24

25. Treating Uncertainty in Model-Based Decision Support A wide variety of methods and techniques are available for dealing with low levels of uncertainty in the context of model-based decision support Sensitivity analysis, Monte-Carlo simulation, Multi-Criteria Decision Analysis, etc. Deep uncertainty is more problematic in model-based decision support e.g. what about disagreement between experts about a functional relationship in a model? Conclusion: Limited capabilities for dealing with deep uncertainty in the context of model-based decision support 25

26. EXPLORATION DETAILS 26

27. Exploration Generic approach Specify the ranges for the parameters Choose a sampling strategy Specify the number of samples Useful for Open exploration of what kind of outcomes are possible Open exploration of what kinds of behavior can occur Most frequently employed strategy in EMA Easy to execute, but big risk of information overload 27

28. Choosing a distribution 28

29. Choosing a distribution 29

30. DIRECTED SEARCH 30

31. Directed search Generic approach: Conclusions are derived from a model for a specific set of parameter values For each parameter a range of possible values is specified Identify under which combinations of parameter values the conclusions are invalidated Directed search is useful for Identifying the worst possible performance of a policy option Identifying the maximum difference in performance between several policy options Identifying the conditions under which model behavior changes (so called tipping points) 31

32. IDENTIFICATION OF PLAUSIBLE TRANSITION PATHWAYS FOR THE FUTURE DUTCH ELECTRICITY GENERATION SYSTEM 32

33. Case details Problem Many electricity companies have to replace a large part of their generation capacity in the coming 20 years Will this enable a transition towards sustainable generation? ElecTrans Model Agent based model of the Dutch system Covers generator companies, network companies, and users Uncertainties Operational costs of options Investment costs Planning horizon Desired Return on investment Various demand developments 33

34. Results 34

35. EMA AND SA 35

36. Sensitivity Analysis vs. EMA Sensitivity analysis (SA) is the study of how the variation in the output of a mathematical model can be apportioned, qualitatively or quantitatively, to different sources of variation in the input of the model.* EMA is interested in exploring the behavior of a modeled system across a wide variety of uncertainties to determine modes of behavior support the development of adaptive robust strategies provide insight into the combinatorial effects of the uncertainties EMA and SA have a different purpose EMA not only interested in model inputs, but also structural or model paradigmatic variations EMA is directly related to supporting policy development 36 *Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D. Saisana, M., and Tarantola, S., 2008, Global Sensitivity Analysis. The Primer, John Wiley & Sons.