1. © 2009 Warren B. Powell© 2008 Warren B. Powell Slide 1 SMART: A Stochastic Multiscale Energy Policy Model using Approximate Dynamic Programming Power Systems Modeling Conference University of Florida March, 2009 Warren Powell Abraham George Alan Lamont Jeffrey Stewart © 2009 Warren B. Powell, Princeton University

2. © 2009 Warren B. Powell Goals for an energy policy model Potential policy questions »How do we design policies to achieve energy goals (e.g. 20% renewables by 2015) with a given probability? »How does the imposition of a carbon tax change the likelihood of meeting this goal? »What might happen if ethanol subsidies are reduced or eliminated? »How would a long-term rise or fall in oil prices affect investments? Some challenges: »The marginal value of wind and solar farms depends on the ability to work with intermittent supply. »The impact of intermittent supply will be affected by our use of storage. »The need for storage (and the value of wind and solar) depends on the entire portfolio of energy producing technologies.

3. © 2009 Warren B. Powell Demand modeling Commercial electric demand 7 days

4. © 2009 Warren B. Powell Intermittent energy sources Wind speed Solar energy

5. © 2009 Warren B. Powell Wind 30 days 1 year

6. © 2009 Warren B. Powell Storage Batteries Ultracapacitors Flywheels Hydroelectric

7. © 2009 Warren B. Powell Long term uncertainties…. 2010 2015 2020 2025 2030 Tax policy Batteries Solar panels Carbon capture and sequestration Price of oil Climate change

8. © 2009 Warren B. Powell Goals for an energy policy model Model capabilities we are looking for »Multi-scale Multiple time scales (hourly, daily, seasonal, annual, decade) Multiple spatial scales Multiple technologies (different coal-burning technologies, new wind turbines, …) Multiple markets –Transportation (commercial, commuter, home activities) –Electricity use (heavy industrial, light industrial, business, residential) –…. »Stochastic (handles uncertainty) Hourly fluctuations in wind, solar and demands Daily variations in prices and rainfall Seasonal variations in weather Yearly variations in supplies, technologies and policies

9. © 2009 Warren B. Powell Prior work Existing deterministic models »MARKAL, NEMS, EFOM, META*Net »Use linear programming with nonlinear market equilibration »Typically focus on long horizons (decades) with long time steps (1-5 years) »Not set up to handle uncertainty or fine-grained spatial and temporal features Techniques incorporating uncertainty »LP-averaging – Stochastic MARKAL »Decision trees – SOCRATES »Stochastic programming – SETSTOCH »Markov decision processes (you have to be kidding)

10. © 2009 Warren B. Powell Decision making under uncertainty Mixing optimization and uncertainty Time Energy sources ? ? ? ? ? large optimization model (e.g. NEMS, MARKAL, …)

11. © 2009 Warren B. Powell Decision making under uncertainty Mixing optimization and uncertainty Time Energy sources 6.2 12.334 No Solar 142 Scenario 1

12. © 2009 Warren B. Powell Decision making under uncertainty Mixing optimization and uncertainty Time Energy sources 3.6 18.917 No Wind 89.1 Scenario 2

13. © 2009 Warren B. Powell Decision making under uncertainty Mixing optimization and uncertainty Time Energy sources 5.9 22.314 Yes Solar 117 Scenario 3

14. © 2009 Warren B. Powell 5.9 22.314 Yes Solar 117 3.6 18.917 No Wind 89.1 Decision making under uncertainty 6.2 12.334 No Solar 142 Scenario 1 Scenario 2 Scenario 3 Now we have to combine the results of these three optimizations to make decisions.

15. © 2009 Warren B. Powell Making decisions Stochastic programming using scenario trees JanFebMarAprMay…Sep…Jan Now Known Unknown 0.3 0.4 probability (weight) 0.3 Wetter Drier Average over 30 Flow forecast scenarios 15% exceedence 50% exceedence 85% exceedence 15% exceedence 50% exceedence 85% exceedence Stochastic programming does not cheat, but it does not scale. It is best designed for coarse-grained sources of uncertainty, but will not handle fine- grained temporal resolution.

16. © 2009 Warren B. Powell Stochastic MARKAL Received Wednesday, March 18 2009

17. © 2009 Warren B. Powell Energy resource modeling SMART: A Stochastic, Multiscale Allocation model for energy Resources, Technology and policy »Stochastic – able to handle different types of uncertainty: Fine-grained – Daily fluctuations in wind, solar, demand, prices, … Coarse-grained – Major climate variations, new government policies, technology breakthroughs »Multiscale – able to handle different levels of detail: Time scales – Hourly to yearly Spatial scales – Aggregate to fine-grained disaggregate Activities – Different types of demand patterns »Decisions Hourly dispatch decisions Yearly investment decisions Takes as input parameters characterizing government policies, performance of technologies, assumptions about climate

18. © 2009 Warren B. Powell Energy resource modeling System state : Resource state Market demands System parameters: State of technology (costs, performance) Climate, weather (temperature, rainfall, wind) Government policies (tax rebates on solar panels) Market prices (oil, coal)

19. © 2009 Warren B. Powell Slide 19 Energy resource modeling The decision variable:

20. © 2009 Warren B. Powell Energy resource modeling Exogenous information: Slide 20 Information can be: Fine-grained: Wind, solar, demand, prices, … Coarse-grained: Changes in technologies, policies, climate

21. © 2009 Warren B. Powell Slide 21 Energy resource modeling The transition function Also known as the system model, plant model or just model. All the physics of the problem.

22. © 2009 Warren B. Powell Slide 22 Introduction to ADP The challenge of dynamic programming: Problem: Three curses of dimensionality »State space »Outcome space (the expectation) »Action space (x is a vector)

23. © 2009 Warren B. Powell Slide 23 Introduction to ADP The computational challenge: How do we find ? How do we compute the expectation? How do we find the optimal solution?

24. © 2009 Warren B. Powell Slide 24 Introduction to ADP We first use the value function around the post-decision state variable, removing the expectation: We then replace the value function with an approximation that we estimate using machine learning techniques:

25. © 2009 Warren B. Powell© 2008 Abraham George 25 Making decisions Following an ADP policy

26. © 2009 Warren B. Powell© 2008 Abraham George 26 Making decisions Following an ADP policy

27. © 2009 Warren B. Powell© 2008 Abraham George 27 Making decisions Following an ADP policy

28. © 2009 Warren B. Powell 28 Approximate dynamic programming With luck, the objective function will improve steadily

29. © 2009 Warren B. Powell© 2008 Warren B. Powell Slide 29 Approximate dynamic programming Step 1: Start with a pre-decision state Step 2: Solve the deterministic optimization using an approximate value function: to obtain. Step 3: Update the value function approximation Step 4: Obtain Monte Carlo sample of and compute the next pre-decision state: Step 5: Return to step 1. Simulation Deterministic optimization Recursive statistics

30. © 2009 Warren B. Powell© 2008 Warren B. Powell Slide 30 Approximate dynamic programming Step 1: Start with a pre-decision state Step 2: Solve the deterministic optimization using an approximate value function: to obtain. Step 3: Update the value function approximation Step 4: Obtain Monte Carlo sample of and compute the next pre-decision state: Step 5: Return to step 1. Simulation Deterministic optimization Recursive statistics

31. © 2009 Warren B. Powell Approximate dynamic programming OptimizationSimulation Cplex Approximate dynamic programming combines simulation and optimization in a rigorous yet flexible framework. Optimization Strengths Produces optimal decisions. Mature technology Weaknesses Cannot handle uncertainty. Cannot handle high levels of complexity Simulation Strengths Extremely flexible High level of detail Easily handles uncertainty Weaknesses Models decisions using user-specified rules. Low solution quality.

32. © 2009 Warren B. Powell Slide 32 The investment problem 2008 2009

33. © 2009 Warren B. Powell Slide 33 The dispatch problem Hourly electricity dispatch problem

34. © 2009 Warren B. Powell Slide 34 Energy resource modeling Hourly model »Decisions at time t impact t+1 through the amount of water held in the reservoir. Hour t Hour t+1

35. © 2009 Warren B. Powell Slide 35 Energy resource modeling Hourly model »Decisions at time t impact t+1 through the amount of water held in the reservoir. Value of holding water in the reservoir for future time periods. Hour t

36. © 2009 Warren B. Powell Slide 36 Energy resource modeling

37. © 2009 Warren B. Powell Slide 37 Energy resource modeling 2008 Hour 1 2 3 4 8760 2009 1 2

38. © 2009 Warren B. Powell Slide 38 Energy resource modeling 2008 Hour 1 2 3 4 8760 2009 1 2

39. © 2009 Warren B. Powell Slide 39 Energy resource modeling 2008 2009

40. © 2009 Warren B. Powell Slide 40 Energy resource modeling 2008 2009

41. © 2009 Warren B. Powell Slide 41 Energy resource modeling 2008 2009 2010 2011 2038

42. © 2009 Warren B. Powell Slide 42 Energy resource modeling 2008 2009 2010 2011 2038

43. © 2009 Warren B. Powell Slide 43 Energy resource modeling 2008 2009 2010 2011 2038 ~5 seconds

44. © 2009 Warren B. Powell Slide 44 Energy resource modeling Use statistical methods to learn the value of resources in the future. Resources may be: »Stored energy Hydro Flywheel energy … »Storage capacity Batteries Flywheels Compressed air »Energy transmission capacity Transmission lines Gas lines Shipping capacity »Energy production sources Wind mills Solar panels Nuclear power plants Amount of resource Value

45. © 2009 Warren B. Powell Energy resource modeling Following sample paths »Demands, prices, weather, technology, policies, … Slide 45 2030 Achieved goal w/ Prob. 0.70 Metric (e.g. % renewable) Need to consider: Little noise (wind, rain, demand, prices, …) Big noise (technology, policy, climate, …)

46. © 2009 Warren B. Powell Convergence analysis »For scalar, piecewise linear approximations: Nascimento, J. and W. B. Powell, An Optimal Approximate Dynamic Programming Algorithm for the Lagged Asset Acquisition Problem, Mathematics of Operations Research, Vol. 34, No. 1, pp. 210-237 (2009). Nascimento, J. and W. B. Powell, Optimal approximate dynamic programming algorithms for a general class of storage problems, under review at SIAM J. on Control and Optimization. »For general continuous states and actions: J. Ma and W. B. Powell, Convergence Proofs for Least Squares Policy Iteration Algorithm of High-Dimensional Infinite Horizon Markov Decision Process Problems, under review at Machine Learning. »Stepsizes: George, A. and W. B. Powell, Adaptive Stepsizes for Recursive Estimation with Applications in Approximate Dynamic Programming, Machine Learning, Vol. 65, No. 1, pp. 167-198, (2006).

47. © 2009 Warren B. Powell Convergence analysis Research on approximations: »George, A., W.B. Powell and S. Kulkarni, Value Function Approximation Using Hierarchical Aggregation for Multiattribute Resource Management, Journal of Machine Learning Research, Vol. 9, pp. 2079-2111 (2008). »L. Hannah, D. Blei and W. B. Powell, Density Estimation and Regression with Dirichlet Process-Generalized Linear Mixture Models, in preparation for submission to Machine Learning. »J. Ma and W. B. Powell, A Convergent Algorithm for Continuous State Value Function Approximations Using Kernel Regression, in preparation.

48. © 2009 Warren B. Powell Energy resource modeling Benchmarking »Compare ADP to optimal LP for a deterministic problem Annual model –8,760 hours over a single year –Focus on ability to match hydro storage decisions 20 year model –24 hour time increments over 20 years –Focus on investment decisions »Comparisons on stochastic model Stochastic rainfall analysis –How does ADP solution compare to LP? Carbon tax policy analysis –Demonstrate nonanticipativity Slide 48

49. © 2009 Warren B. Powell 49 Iterations Percentage error from optimal 0.06% over optimal Energy resource modeling ADP objective function relative to optimal LP 2.50 2.00 1.50 1.00 0.50 0.00

50. © 2009 Warren B. Powell Optimal from linear program Energy resource modeling Optimal from linear program Reservoir level Demand Rainfall

51. © 2009 Warren B. Powell 51 Approximate dynamic programming Energy resource modeling ADP solution Reservoir level Demand Rainfall

52. © 2009 Warren B. Powell Optimal from linear program Energy resource modeling Optimal from linear program Reservoir level Demand Rainfall

53. © 2009 Warren B. Powell Approximate dynamic programming Energy resource modeling ADP solution Reservoir level Demand Rainfall

54. © 2009 Warren B. Powell 54 Annual energy model Time period Precipitation Sample paths

55. © 2009 Warren B. Powell 55 Reservoir level Optimal for individual scenarios Time period ADP Energy resource modeling

56. © 2009 Warren B. Powell Slide 56 Multidecade energy model Optimal vs. ADP – daily model over 20 years 0.24% over optimal

57. © 2009 Warren B. Powell Energy policy modeling Traditional optimization models tend to produce all-or-nothing solutions Cost differential: IGCC - Pulverized coal Pulverized coal is cheaper IGCC is cheaper Investment in IGCC Traditional optimization Approximate dynamic programming

58. © 2009 Warren B. Powell Energy policy modeling Policy study: »What is the effect of a potential (but uncertain) carbon tax in year 8? 1 2 3 4 5 6 7 8 9 Year Carbon tax 0

59. © 2009 Warren B. Powell Energy policy modeling Renewable technologies Carbon-based technologies No carbon tax

60. © 2009 Warren B. Powell Energy policy modeling With carbon tax Carbon-based technologies Renewable technologies Carbon tax policy unknown Carbon tax policy determined

61. © 2009 Warren B. Powell Energy policy modeling With carbon tax Carbon-based technologies Renewable technologies

62. © 2009 Warren B. Powell Conclusions Capabilities »SMART can handle problems with over 300,000 time periods so that it can model hourly variations in a long- term energy investment model. »It can simulate virtually any form of uncertainty, either provided through an exogenous scenario file or sampled from a probability distribution. »Accurate modeling of climate, technology and markets requires access to exogenously provided scenarios. »It properly models storage processes over time. »Current tests are on an aggregate model, but the modeling framework (and library) is set up for spatially disaggregate problems.

63. © 2009 Warren B. Powell Conclusions Limitations »Value function approximations capture the resource state vector, but are limited to very simple exogenous state variations. »More research is needed to test the ability of the model to use multiple storage technologies. »Extension to spatially disaggregate model will require significant engineering and data. »Run times will start to become an issue for a spatially disaggregate model.

64. © 2009 Warren B. Powell© 2008 Warren B. Powell Slide 64