Decision Making Under Deep Uncertainty Robert Lempert Director RAND Pardee Center for Longer Range Global Policy and the Future Human Condition July 8, 2009

2 Traditional Planning Methods Can Illuminate Trees Rather Than Forest Traditional analytic methods characterize uncertainties as a prelude to assessing alternative decisions Predict Act Climate change confronts decisionmakers with deep uncertainty, where –They do not know, and/or key parties to the decision do not agree on, the system model, prior probabilities, and/or “cost” function Decisions can go awry if decisionmakers assume risks are well- characterized when they are not –Uncertainties are underestimated –Competing analyses can contribute to gridlock –Misplaced concreteness can blind decision-makers to surprise Climate change confronts decisionmakers with deep uncertainty, where –They do not know, and/or key parties to the decision do not agree on, the system model, prior probabilities, and/or “cost” function Decisions can go awry if decisionmakers assume risks are well- characterized when they are not –Uncertainties are underestimated –Competing analyses can contribute to gridlock –Misplaced concreteness can blind decision-makers to surprise

3 Forecasting the Unpredictable Can Contribute to Bad Decisions Gross national product (trillions of 1958 dollars) Energy use (10 15 Btu per year) 0 Historical trend continued Scenarios In the early 1970s forecasters made projections of U.S energy use based on a century of data In the early 1970s forecasters made projections of U.S energy use based on a century of data

4 Forecasting the Unpredictable Can Contribute to Bad Decisions Gross national product (trillions of 1958 dollars) Energy use (10 15 Btu per year) 0 Historical trend continued Actual Scenarios 2000 Actual In the early 1970s forecasters made projections of U.S energy use based on a century of data … they all were wrong In the early 1970s forecasters made projections of U.S energy use based on a century of data … they all were wrong

5 Outline Robust Decision Making (RDM) Climate vulnerability and response option analysis for Inland Empire Utilities Agency (IEUA) Observations Robust Decision Making (RDM) Climate vulnerability and response option analysis for Inland Empire Utilities Agency (IEUA) Observations

6 New Technology Allows Computer to Serve As “Prosthesis for the Imagination” Robust Decision Making (RDM) is a quantitative decision analytic approach that –Characterizes uncertainty with multiple, rather than single, views of the future –Evaluates alternative decision options with a robustness, rather than optimality, criterion –Iteratively identifies vulnerabilities of plans and evaluates potential responses Robust Decision Making (RDM) is a quantitative decision analytic approach that –Characterizes uncertainty with multiple, rather than single, views of the future –Evaluates alternative decision options with a robustness, rather than optimality, criterion –Iteratively identifies vulnerabilities of plans and evaluates potential responses Candidate strategy Identify vulnerabilities Assess alternatives for ameliorating vulnerabilities RDM combines key advantages of scenario planning and quantitative decision analysis in ways that –Decision makers find credible –Contribute usefully to contentious debates RDM combines key advantages of scenario planning and quantitative decision analysis in ways that –Decision makers find credible –Contribute usefully to contentious debates

7 RDM Has Effectively Addressed Many Types of Decisions Under Deep Uncertainty Energy, Environment, and Climate Change Long-Range Natural Resource Management Renewable portfolios standards Center on climate change decision making National Security Terrorism Insurance Force procurement and deployment Pre-conflict shaping strategies Commercial- Sector Applications Electric utilities’ strategies under deregulation Product and technology planning in the auto industry

8 Compare Alternative Approaches to Managing Catastrophic Event with Unknown Probability Consider town on shore of pristine lake –Lake can switch abruptly to undesirable and potentially irreversible eutrophic state at unknown pollution concentration Citizens must decide how much pollution to emit –Gain small utility from emitting pollution to lake and lose significant utility if lake goes eutrophic –Deeply uncertain about location of concentration threshold Alternative decision approaches include: –Optimum expected utility –Precautionary principle –Robust decision making Consider town on shore of pristine lake –Lake can switch abruptly to undesirable and potentially irreversible eutrophic state at unknown pollution concentration Citizens must decide how much pollution to emit –Gain small utility from emitting pollution to lake and lose significant utility if lake goes eutrophic –Deeply uncertain about location of concentration threshold Alternative decision approaches include: –Optimum expected utility –Precautionary principle –Robust decision making Robert J. Lempert and Myles T. Collins., 2007: “Managing the Risk of Uncertain Threshold Response: Comparison of Robust, Optimum, and Precautionary Approaches” Risk Analysis 27 (4), 1009–1026

9 Use Simple Simulation Model of Lake System to Assess Consequences of Town’s Decisions Three “policy levers” describe town’s citizens’ adaptive strategy –Initial pollution emissions (L 0 ) –Maximum yearly increase in emissions (  L) –Safety margin (S) – buffer between pollution emissions and estimate of critical threshold (X crit ) Over time, citizens learn true value of critical threshold –Observations increasingly accurate as level of pollution approaches unknown threshold Nutrients in Lake Natural Emissions Nutrient sink Recycling when eutrophic L t = f(L 0,  L,S) Anthropogenic Emissions Learning Lempert and Collins (2007)

10 Well Characterized Uncertainty Suggests An Optimal Strategy Optimal Strategy w Uncertainty L init 0.37 Safety Margin3.0 LL 0.11 Mean PVU13. Probability distribution for critical threshold X crit Lempert and Collins (2007)

11 Robust Strategies Trade Some Optimal Performance for Less Sensitivity to Assumptions Represent uncertainty about critical threshold with set of multiple, plausible distributions Define expected regret of strategy s contingent on distribution i where strategy s regret is Compared to optimal strategy, a robust strategy has small weighted average of best and worst expected regret Represent uncertainty about critical threshold with set of multiple, plausible distributions Define expected regret of strategy s contingent on distribution i where strategy s regret is Compared to optimal strategy, a robust strategy has small weighted average of best and worst expected regret Lempert and Collins (2007)

12 Find Vulnerabilities of Optimal Strategy Regret of Strategy A over different values of X crit Vulnerabilities Present value utility of optimum strategy Lempert and Collins (2007)

13 Compare Regret Over Range of Futures Optimal Strategy Lempert and Collins (2007)

14 Town’s Citizens Have More Robust Options Than Strategy A Optimal Strategy Potential Robust Strategy Lempert and Collins (2007)

15 Outline Robust Decision Making (RDM) Climate vulnerability and response option analysis for Inland Empire Utilities Agency (IEUA) –What impacts may climate change have on IEUA’s current plans? –What should IEUA do in response? Observations Robust Decision Making (RDM) Climate vulnerability and response option analysis for Inland Empire Utilities Agency (IEUA) –What impacts may climate change have on IEUA’s current plans? –What should IEUA do in response? Observations

16 Climate Change Poses Significant Planning Challenge for Water Managers Climate change will likely have large but uncertain impacts on supply and demand for water “Stationarity is dead” –Most agencies already include climate (often implicitly) in many decisions –Amidst all the uncertainty one thing we do know for sure -- tomorrow’s climate will not be like the past’s Relaxing this assumption poses key challenges –How do you adjust plans based on uncertain climate projections? –How do you communicate these plans, especially when uncertain long-term benefits require near-term costs? Climate change will likely have large but uncertain impacts on supply and demand for water “Stationarity is dead” –Most agencies already include climate (often implicitly) in many decisions –Amidst all the uncertainty one thing we do know for sure -- tomorrow’s climate will not be like the past’s Relaxing this assumption poses key challenges –How do you adjust plans based on uncertain climate projections? –How do you communicate these plans, especially when uncertain long-term benefits require near-term costs?

17 Conducted Vulnerability and Options Analysis for Inland Empire Utilities Agency (IEUA) –IEUA currently serves 800,000 people May add 300,000 by 2025 –Water presents a significant challenge –IEUA currently serves 800,000 people May add 300,000 by 2025 –Water presents a significant challenge David G. Groves, Debra Knopman, Robert J. Lempert, Sandra H. Berry, and Lynne Wainfan, Presenting Uncertainty About Climate Change to Water Resource Managers, RAND TR-505-NSF, 2007.

18 –Current water sources include: Groundwater 56% Imports32% Recycled1% Surface8% Desalter2% Conducted Vulnerability and Options Analysis for Inland Empire Utilities Agency (IEUA) –IEUA currently serves 800,000 people May add 300,000 by 2025 –Water presents a significant challenge –IEUA currently serves 800,000 people May add 300,000 by 2025 –Water presents a significant challenge Groves et. al. (2007)

19 –Current water sources include: Groundwater 56% Imports32% Recycled1% Surface8% Desalter2% Focus of IEUA’s 20 year plan Conducted Vulnerability and Options Analysis for Inland Empire Utilities Agency (IEUA) –IEUA currently serves 800,000 people May add 300,000 by 2025 –Water presents a significant challenge –IEUA currently serves 800,000 people May add 300,000 by 2025 –Water presents a significant challenge Groves et. al. (2007)

20 Model Performance of plans IEUA Plans System data & climate forecasts We Built a Model to Assess Performance of IEUA Plans in Different Future States of World –Model projects future water supply and demand for IEUA service area Consistent with IEUA management plans and assumptions Reflect plausible trends of climate change –Model projects future water supply and demand for IEUA service area Consistent with IEUA management plans and assumptions Reflect plausible trends of climate change Based on WEAP software tool Groves et. al. (2007)

21 GCMs Project Plausible Temperature and Precipitation Ranges for Southern California –Derived from forecasts from 21 GCMs with A1B emissions scenario –Each forecast weighted by ability to reproduce past climate and level of agreement with other forecasts –Derived from forecasts from 21 GCMs with A1B emissions scenario –Each forecast weighted by ability to reproduce past climate and level of agreement with other forecasts (Tebaldi et al.) Temperature Precipitation Groves et. al. (2007)

22 Generate Future Weather Sequences by Resampling Historic Local Climate Records KNN method produces hundreds of local weather sequences –Daily and monthly variability that matches historic Chino climate –Temperature and precipitation trends that match climate model forecasts KNN method produces hundreds of local weather sequences –Daily and monthly variability that matches historic Chino climate –Temperature and precipitation trends that match climate model forecasts (Yates et al.) Groves et. al. (2007)

23 Model Performance of plans IEUA Plans System data & climate forecasts Model Assess Performance of IEUA Plans in Many Different Scenarios Temp: +1.6 o C Precip: -10% Scenario B Plan suffers shortages in adverse future climate Year Annual supply (taf) Recycled Groundwater Local Supplies Imports Dry-year yield Surplus Shortage Temp: +0.7 o C Precip: +3% Scenario A Plan generates surpluses in benign future climate Year Annual supply (taf) Recycled Groundwater Imports Surplus Local Supplies Groves et. al. (2007)

24 Many Uncertain Factors Could Impact the Performance of Current IEUA Plan Natural Processes Future temperatures Future precipitation Changes in groundwater processes Performance of Management Strategies Development of aggressive waste-water recycling program Implementation of groundwater replenishment Costs of Future Supplies and Management Activities Imported supplies Water use efficiency Groves et. al. (2007)

25 Planners in S. California, for Instance, Face a Range of Possible Future Climate Conditions Summer-time temperature change ( ) +.1C +2.1C 0 Likely range Results based on statistical summary of 21 of the world’s best Global Climate Models Winter-time precipitation change ( ) +8% -19% 0 Likely range No changeHotter WetterMuch drier Groves et. al. (2007)

26 Many Uncertain Factors Could Impact the Performance of Current IEUA Plan Natural Processes Future temperatures Future precipitation Changes in groundwater processes Performance of Management Strategies Development of aggressive waste-water recycling program Implementation of groundwater replenishment Costs of Future Supplies and Management Activities Imported supplies Water use efficiency Groves et. al. (2007)

PV shortage cost ($ billions) PV supply cost ($ billions) Scenario A Scenario B Adverse climate $3.4 billion in supply cost $1.9 billion in shortage cost Current IEUA 2005 Urban Water Management Plan Benign climate $3.3 billion in supply cost $0 in shortage cost “Scenario Maps” Help Decision Makers Visualize How Plans Evolve Over Many Futures David G. Groves, Robert J. Lempert, Debra Knopman, Sandra H. Berry: Preparing for an Uncertain Climate Future: Identifying Robust Water Management Strategies, RAND DB-550-NSF, 2008.

PV shortage cost ($ billions) PV supply cost ($ billions) Current IEUA Plan (200 Scenarios) “Scenario Maps” Help Decision Makers Visualize How Plans Evolve Over Many Futures Groves et. al. (2008)

PV shortage cost ($ billions) PV supply cost ($ billions) Current IEUA Plan $3.75 billion cost threshold Current plan generates high costs in 120 of 200 Scenarios “Scenario Maps” Help Decision Makers Visualize How Plans Evolve Over Many Futures Groves et. al. (2008)

30 Discover Key Scenarios in Ensembles of Many Model Runs 1.Ran the model 200 times under different combinations of uncertain factors (e.g. temperature and precipitation trends and others) 2.Used statistical algorithms to identify conditions that lead to 2005 UWMP to perform poorly 3.These factors become key driving forces for “policy-relevant” scenarios 1.Ran the model 200 times under different combinations of uncertain factors (e.g. temperature and precipitation trends and others) 2.Used statistical algorithms to identify conditions that lead to 2005 UWMP to perform poorly 3.These factors become key driving forces for “policy-relevant” scenarios Number of runs High Cost (120 runs) Statistical analysis suggests factors that contribute most to these undesirable outcomes Groves et. al. (2007)

PV shortage cost ($ billions) PV supply cost ($ billions) Current IEUA Plan Statistical Analysis Suggests Key Factors That Create Vulnerabilities for Existing Plan Natural Processes Future temperatures Future precipitation Changes in groundwater processes Performance of Management Strategies Development of aggressive waste-water recycling program Implementation of groundwater replenishment Costs of Future Supplies and Management Activities Imported supplies Water use efficiency These three factors explain 70% of vulnerabilities of IEUA’s current plans Groves et. al. (2008)

32 Response Options May Help IEUA Address These Vulnerabilities Groves et. al. (2008)

33 Can Quantify Some, But Not All, Of These Costs Costs increase over time Groves et. al. (2008)

34 Should IEUA Act Now or Later to Reduce Potential Climate Vulnerabilities? Act now to augment 2005 Plan? NO Monitor, and take additional action if supplies drop too low In 2015, 2020, 2025, …. YES Implement additional efficiency, recycling, and replenishment In 2015, 2020, 2025, …. Monitor, and take additional action if supplies drop too low Groves et. al. (2008)

35 Compare Nine Strategies Over 200 Scenarios Reflecting Key Uncertainties Static options Update options Number of Scenarios (PV Costs > $3.75 billion) Current Plan forever Current Plan + DYY and recycling Current Plan + replenishment Current Plan with updates Current Plan + replenishment with updates Current Plan + efficiency Current Plan + efficiency with updates Current Plan + DYY and recycling with updates Current Plan + all enhancements Groves et. al. (2008)

36 Just Allowing IEUA’s Current Plan to Update Reduces Vulnerability Substantially Static options Update options From 120 Down to 30 Number of Scenarios (PV Costs > $3.75 billion) Current Plan forever Current Plan + DYY and recycling Current Plan + replenishment Current Plan with updates Current Plan + replenishment with updates Current Plan + efficiency Current Plan + efficiency with updates Current Plan + DYY and recycling with updates Current Plan + all enhancements Groves et. al. (2008)

37 Acting Now Reduces Future Vulnerabilities Even More Current Plan with updates Current Plan + replenishment with updates Current Plan + efficiency Current Plan + efficiency with updates Current Plan + DYY and recycling with updates Current Plan + all enhancements Static options Update options Number of Scenarios (PV Costs > $3.75 billion) Groves et. al. (2008)

38 Acting Now Reduces Future Vulnerabilities Even More Current Plan with updates Current Plan + replenishment with updates Current Plan + efficiency Current Plan + efficiency with updates Current Plan + DYY and recycling with updates Current Plan + all enhancements Static options Update options Number of Scenarios (PV Costs > $3.75 billion) Implementation becomes more challenging This analysis helped IEUA decide to make more near-term efficiency investments, and to monitor performance and adapt as needed down the road Groves et. al. (2008)

39 Outline Robust Decision Making (RDM) Climate vulnerability and response option analysis for Inland Empire Utilities Agency (IEUA) Observations Robust Decision Making (RDM) Climate vulnerability and response option analysis for Inland Empire Utilities Agency (IEUA) Observations

40 Conducted Elicitations Among IEUA’s Planners and Community to Estimate Likelihood of Achieving Goals RecyclingReplenishment Goal Miss goal Miss goal Probability of meeting UWMP goals Meet Goals Miss Goals Groves et. al. (2007)

41 Many Uncertain Factors Could Impact the Performance of Current IEUA Plan Natural Processes Future temperatures Future precipitation Changes in groundwater processes Performance of Management Strategies Development of aggressive waste-water recycling program Implementation of groundwater replenishment Costs of Future Supplies and Management Activities Imported supplies Water use efficiency Groves et. al. (2007)

42 Meet recycling goal Meet replenishment goal Future climate New conservation Percolation decrease Climate on imports MissExceedMeet MissExceedMeet DrierWetter -5%+20% -20%0% WeakStrong Explains 70% of high cost cases Analysis Suggests Factors That Cause Severe Shortages for IEUA’s 20 Year Plan Climate-related uncertainties facing IEUA Groves et. al. (2007)

43 RDM Enables Effective Planning Based on Multiple Views of Future Use many scenarios to imagine the future –Not a single forecast Seek robust strategies that do well across many scenarios assessed according to several values –Not optimal strategies Employ strategies that evolve over time in response to changing conditions –Not "fixed" strategies Use computer as “prosthesis for the imagination” –Not a calculator Use many scenarios to imagine the future –Not a single forecast Seek robust strategies that do well across many scenarios assessed according to several values –Not optimal strategies Employ strategies that evolve over time in response to changing conditions –Not "fixed" strategies Use computer as “prosthesis for the imagination” –Not a calculator

44 More Information David G. Groves, Robert J. Lempert, Debra Knopman, Sandra H. Berry: Preparing for an Uncertain Climate Future: Identifying Robust Water Management Strategies, RAND DB-550-NSF, David G. Groves, Debra Knopman, Robert J. Lempert, Sandra H. Berry, and Lynne Wainfan, Presenting Uncertainty About Climate Change to Water Resource Managers, RAND TR-505-NSF, Groves, David G, David Yates, Claudia Tebaldi, 2008: “Developing and Applying Uncertain Global Climate Change Projections for Regional Water Management Planning,” Water Resources Research, 44(12): W12413 Robert J. Lempert and Myles T. Collins., 2007: “Managing the Risk of Uncertain Threshold Response: Comparison of Robust, Optimum, and Precautionary Approaches” Risk Analysis 27 (4), 1009–1026 David G. Groves and Robert J. Lempert, 2007: A new analytic method for finding policy-relevant scenarios, Global Environmental Change 17, Robert J. Lempert, Steven W. Popper, Steven C. Bankes, 2003: Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis, RAND MR-1626-RPC, Aug. David G. Groves, Robert J. Lempert, Debra Knopman, Sandra H. Berry: Preparing for an Uncertain Climate Future: Identifying Robust Water Management Strategies, RAND DB-550-NSF, David G. Groves, Debra Knopman, Robert J. Lempert, Sandra H. Berry, and Lynne Wainfan, Presenting Uncertainty About Climate Change to Water Resource Managers, RAND TR-505-NSF, Groves, David G, David Yates, Claudia Tebaldi, 2008: “Developing and Applying Uncertain Global Climate Change Projections for Regional Water Management Planning,” Water Resources Research, 44(12): W12413 Robert J. Lempert and Myles T. Collins., 2007: “Managing the Risk of Uncertain Threshold Response: Comparison of Robust, Optimum, and Precautionary Approaches” Risk Analysis 27 (4), 1009–1026 David G. Groves and Robert J. Lempert, 2007: A new analytic method for finding policy-relevant scenarios, Global Environmental Change 17, Robert J. Lempert, Steven W. Popper, Steven C. Bankes, 2003: Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis, RAND MR-1626-RPC, Aug.

45 Thank you!

46 We Also Evaluated How This Analysis Affected Policy-Makers’ Views Four IEUA workshops presented modeling results to participants including: –Agency professional managers and technical staff –Local elected officials –Community stakeholders “Real-time” surveys measured participants’ –Understanding of concepts –Willingness to adjust policy choices based on information presented –Views on RDM Four IEUA workshops presented modeling results to participants including: –Agency professional managers and technical staff –Local elected officials –Community stakeholders “Real-time” surveys measured participants’ –Understanding of concepts –Willingness to adjust policy choices based on information presented –Views on RDM Groves et. al. (2007)

47 First Three Workshops Compared Alternative Approaches to Uncertainty Workshop design approximates on-going laboratory experiments Compared three approaches Traditional qualitative scenarios Probabilistic forecasts RDM with Scenario discovery Groves et. al. (2007)

48 RDM Scenarios More Useful, But More Difficult to Understand Questionnaire itemTraditional Scenarios Scenario Discovery Provides results that can be used in planning Agree somewhat Agree strongly Provides information on how to improve plan Agree somewhat Is easy to explain to decisionmakers Agree somewhat Disagree strongly Traditional scenarios –Gave IEUA much of the information they needed –Emphasized the importance of achieving goals in IEUA’s plan Scenario Discovery –Provided more useful information –Sparked discussion of adaptive strategies Traditional scenarios –Gave IEUA much of the information they needed –Emphasized the importance of achieving goals in IEUA’s plan Scenario Discovery –Provided more useful information –Sparked discussion of adaptive strategies Groves et. al. (2007)

49 RDM Scenarios More Useful, But More Difficult to Understand Questionnaire itemTraditional Scenarios Scenario Discovery Provides results that can be used in planning Agree somewhat Agree strongly Provides information on how to improve plan Agree somewhat Is easy to explain to decisionmakers Agree somewhat Disagree strongly Traditional scenarios –Gave IEUA much of the information they needed –Emphasized the importance of achieving goals in IEUA’s plan Scenario Discovery –Provided more useful information –Sparked discussion of adaptive strategies Traditional scenarios –Gave IEUA much of the information they needed –Emphasized the importance of achieving goals in IEUA’s plan Scenario Discovery –Provided more useful information –Sparked discussion of adaptive strategies

50 Fourth (Adaptive Strategy) Workshop Compared Different Presentations of RDM Participants reported: –RDM helped support comparison of climate-related risks and choice among plans –Preference for scatter plot over histogram scenario displays After the workshop: –35% said consequences of bad climate change now appeared “more serious” than before –40% thought the likelihood of of bad climate change outcomes for the IEUA was “greater” than before –75% though the ability of IEUA planners to plan for and manage effects was “greater” than before Overall, analysis increased: –Perceived likelihood of serious climate impacts –Confidence that IEUA could take effective actions to reduce its vulnerability to climate change –Support for near-term efficiency enhancements to current IEUA plan Participants reported: –RDM helped support comparison of climate-related risks and choice among plans –Preference for scatter plot over histogram scenario displays After the workshop: –35% said consequences of bad climate change now appeared “more serious” than before –40% thought the likelihood of of bad climate change outcomes for the IEUA was “greater” than before –75% though the ability of IEUA planners to plan for and manage effects was “greater” than before Overall, analysis increased: –Perceived likelihood of serious climate impacts –Confidence that IEUA could take effective actions to reduce its vulnerability to climate change –Support for near-term efficiency enhancements to current IEUA plan Groves et. al. (2008)

51 Observations Analysis suggests IEUA’s current long-range plans: –Vulnerable to climate change –Can be made more resilient by near-term conservation, attention to storm intensity, and effective future monitoring and updating Measurements suggest –RDM analysis effectively shifted views on seriousness of climate challenges and appropriate responses, but requires more work to be easily understood by policy-makers –Importance of linking effective response options with presentation of climate uncertainty Currently using this approach to help several major water agencies include climate in their long-range plans Analysis suggests IEUA’s current long-range plans: –Vulnerable to climate change –Can be made more resilient by near-term conservation, attention to storm intensity, and effective future monitoring and updating Measurements suggest –RDM analysis effectively shifted views on seriousness of climate challenges and appropriate responses, but requires more work to be easily understood by policy-makers –Importance of linking effective response options with presentation of climate uncertainty Currently using this approach to help several major water agencies include climate in their long-range plans