1. Slide 1 Stochastic Optimization in Energy Systems DIMACS Workshop on Algorithmic Decision Theory Rutgers University October 27, 2010 Warren Powell CASTLE Laboratory Princeton University http://www.castlelab.princeton.edu © 2010 Warren B. Powell, Princeton University

2. © 2010 Warren B. Powell Slide 2 Lecture outline  The problem of uncertainty  Modeling stochastic optimization problems  Energy storage portfolios  The unit commitment problem for PJM  Long-term energy resource planning

3. Challenges in models and algorithms  Strategic questions in energy policy and economics »How do we design market mechanisms to control green house gases? »How do we design policies to achieve energy goals (e.g. 30% renewables by 2030) with a given probability? »How does the imposition of a carbon tax or investment tax credit change the likelihood of meeting this goal? »We need models of competitive equilibria and cooperative games in the presence of uncertainties about technology, policy and climate change. »How do we quantify the risks associated with the pairing of energy from wind with dispatchable energy from hydroelectric facilities in the presence of climate change?

4. Challenges in models and algorithms  Design and control of energy systems »How do we balance storage of energy from the grid in electric vehicles in the presence of variability of energy from wind, volatile prices, and uncertainty in usage patterns for vehicles? »How do we control the storage of energy across a portfolio of energy storage devices? »How should a utility work with urban building managers to balance energy consumption from the grid, storage devices and backup generators? »How can grid operators modify their unit commitment models to capture significant contributions of energy from wind and solar? »How can companies design efficient verification policies for forests being used as carbon offsets?

5. Challenges in models and algorithms  All of these problems can be formulated as some type of stochastic optimization problem.  Simple problems can be solved as decision trees…  Most problems are hard.

6. Decision making under uncertainty  Mixing optimization and uncertainty Time Energy sources ? ? ? ? ? “large optimization model (e.g. NEMS, MARKAL, …)”

7. Decision making under uncertainty  Mixing optimization and uncertainty Time Energy sources 6.2 12.334 No Solar 142 Scenario 1

8. Decision making under uncertainty  Mixing optimization and uncertainty Time Energy sources 3.6 18.917 No Wind 89.1 Scenario 2

9. Decision making under uncertainty  Mixing optimization and uncertainty Time Energy sources 5.9 22.314 Yes Solar 117 Scenario 3

10. 5.9 22.314 Yes Solar 117 3.6 18.917 No Wind 89.1 Decision making under uncertainty 6.2 12.334 No Solar 142 Scenario 1 Scenario 2 Scenario 3 Now we have to combine the results of these three optimizations to make decisions.

11. Decision making under uncertainty Don’t gamble; take all your savings and buy some good stock and hold it till it goes up, then sell it. If it don’t go up, don’t buy it. Will Rogers It is not enough to mix “optimization” (intelligent decision making) and uncertainty. You have to be sure that each decision has access to the information available at the time.

12. Decision making under uncertainty  Socrates – used by Pacific, Gas and Electric JanFebMarAprMay…Sep…Jan Now Known Unknown 0.3 0.4 probability (weight) 0.3 Wetter Drier Average over 30 Flow forecast scenarios 15% exceedence 50% exceedence 85% exceedence 15% exceedence 50% exceedence 85% exceedence Stochastic programming does not “cheat”, but it does not scale. It is best designed for coarse-grained sources of uncertainty, but will not handle fine- grained temporal resolution.

13. © 2010 Warren B. Powell Slide 13 Lecture outline  The problem of uncertainty  Modeling stochastic optimization problems  Energy storage portfolios  The unit commitment problem for PJM  Long-term energy resource planning

14. Slide 14 Modeling stochastic optimization problems  Attribute vectors:

15. Slide 15 Modeling stochastic optimization problems  Modeling resources: »The attributes of a single resource: »The resource state vector: »The information process:

16. Slide 16 Modeling stochastic optimization problems  The system state:

17. Slide 17 Modeling stochastic optimization problems  Making decisions:

18. Slide 18 Modeling stochastic optimization problems  Exogenous information:

19. Slide 19 Modeling stochastic optimization problems  The transition function Known as the: “Transition function” “Transfer function” “State transition model” “System model” “Plant model” “Model”

20. Slide 20 Modeling stochastic optimization problems Demands Resources

21. Slide 21 Modeling stochastic optimization problems t t+1 t+2

22. Slide 22 Modeling stochastic optimization problems t t+1 t+2 Optimizing at a point in time Optimizing over time

23. Modeling stochastic optimization problems  The objective function Decision function (policy) State variable Contribution function Finding the best policy Expectation over all random outcomes

24. Modeling stochastic optimization problems  Policies »1) Myopic policies Take the action that maximizes contribution (or minimizes cost) for just the current time period: »2) Lookahead policies Plan over the next T periods, but implement only the action it tells you to do now.

25. Modeling stochastic optimization problems  Policies (cont’d) »3) Policies based on value function approximations »4) Policy function approximations

26. Modeling stochastic optimization problems  Ways of approximating functions (policies or value functions) »1) Lookup tables When in a (discrete) state, returns an action When in a (discrete) state, returns the value of being in that state. There is one value (parameter) to determine for each state. »2) Parametric models A closed form, analytic function determined by one or more parameters.

27. Modeling stochastic optimization problems  Ways of approximating functions (policies or value functions) »3) Nonparametric models Nonparametric methods approximate a function by using a weighted sum of observations, where the weight declines with the distance between the observed state and the state we are trying to estimate.

28. © 2010 Warren B. Powell Slide 28 Lecture outline  The problem of uncertainty  Modeling stochastic optimization problems  Energy storage portfolios  The unit commitment problem for PJM  Long-term energy resource planning

29. © 2010 Warren B. Powell Slide 29 Energy storage portfolios  Premise »The optimal mix of energy resources depends on the entire portfolio. »The performance of any single source of energy (including storage) depends on the marginal difference between demand and all other forms of energy in the portfolio. This is what determines the residual demand. »It is not just what is happening (the physical process) but also how well you could predict what was going to happen (the information process).

30. © 2010 Warren B. Powell Slide 30 Energy storage portfolios  Wind »Varies with multiple freqeuencies (seconds, hours, days, seasonal). »Spatially uneven, generally not aligned with population centers. Solar »Shines primarily during the day (when it is needed), but not entirely reliably. »Strongest in south/southwest.

31. © 2010 Warren B. Powell Slide 31 Energy storage portfolios 30 days 1 year

32. © 2010 Warren B. Powell Slide 32 Energy storage portfolios Batteries Ultracapacitors Flywheels Hydroelectric

33. © 2010 Warren B. Powell Slide 33 Energy storage portfolios  Storage technologies »Storage technologies different in terms of: Charge rates Discharge rates Power rating Capacity Loss Slide 33 NameCharge rateDischarge ratePower ratingCapacityLossEfficiency Batteries sodium-sulfur5 hours 10 MW20 Wh/kg< lithium80-90% lead-acid10 hours10 mins.-hour100 kW20 Wh/kg10%/month75% lithium1 hour 10-100 kW150 Wh/kg10%/month90-95% Ultracapacitorsecondsfew seconds10-100 kW5 Wh/kg5%/day95-100% Flow battery5 hours 200 kW-1MWminimal70-78%? Pumped hydro storage (PHS)hours10 hours1 GWevaporation75%-80% Flywheelminutes 10kW-10MW10 Wh/kgfrictional95% SMES (supercond. mag. )seconds 10-100 MW.2 Wh/kglarge95% Comp. air energy storage (CAES) above groundhours2 hours15 MWminimal48% below groundhours10 hours100-300MWminimal48%

34. Optimal control of wind and storage  Controlling the storage process »Imagine that we would like to use storage to reduce demand when electricity prices are high. »We use a simple policy controlled by two parameters. Price

35. Energy storage portfolios  We now have to optimize over policies »This means finding the best values for »The policy refers to the structure of the decision function, and the parameters »This means we have to solve »We can write © 2010 Warren B. Powell Slide 35

36. 36 Optimizing storage policy

37.  Initially we think the concentration is the same everywhere: »We want to measure the value where the knowledge gradient is the highest. This is the measurement that teaches us the most. Optimizing storage policy Estimated contributionKnowledge gradient

38.  After four measurements: »Whenever we measure at a point, the value of another measurement at the same point goes down. The knowledge gradient guides us to measuring areas of high uncertainty. Optimizing storage policy Measurement Value of another measurement at same location. Estimated contributionKnowledge gradient New optimum

39. Optimizing storage policy  After five measurements: Estimated contributionKnowledge gradient

40. Optimizing storage policy  After six samples Estimated contributionKnowledge gradient

41. Optimizing storage policy  After seven samples Estimated contributionKnowledge gradient

42. Optimizing storage policy  After eight samples Estimated contributionKnowledge gradient

43. Optimizing storage policy  After nine samples Estimated contributionKnowledge gradient

44. Optimizing storage policy  After ten samples Estimated contributionKnowledge gradient

45.  After 10 measurements, our estimate of the surface: Optimizing storage policy Estimated contributionTrue concentration

46.  After 10 measurements, our estimate of the surface: Optimizing storage policy Estimated contributionKnowledge gradient

47. © 2010 Warren B. Powell Slide 47 Lecture outline  The problem of uncertainty  Modeling stochastic optimization problems  Energy storage portfolios  The unit commitment problem for PJM  Long-term energy resource planning

48. © 2010 Warren B. Powell Slide 48  Designing energy portfolios…. »… is like building a stone wall, where energy comes from sources which vary in terms of capital cost, operating cost and flexibility. The unit commitment problem

49.  The day-ahead problem »Determines which coal/nuclear/natural gas/…plants to turn on/off and when. »Will use wind/solar when available and if needed. »Requires point forecast of wind/solar/demand. ….subject to numerous constraints, including integrality. »The problem is solved using two adjustment parameters Fraction of generator capacity assumed (e.g. 93 percent) q Quantile of wind forecast assumed for advance commitments.

50. The unit commitment problem  The hour-ahead problem »Each hour, we can make modest adjustments. Plants that are “on” can be adjusted up and down. »Cannot turn coal plants on and off. »Actual may differ from forecast: Wind may be higher or lower than forecast. If higher, may not be able to use it because of inability to scale back other sources. Demand may exceed forecast. Wind/solar may fall below forecast. In this case, we find the least cost unit that can be scaled up quickly enough.

51. The unit commitment problem  Modeling »A deterministic model »Stochastic formulation – I Ambiguous whether decisions are deterministic or stochastic. Day ahead decisions are deterministic, but hour ahead decisions are stochastic.

52. The unit commitment problem  Modeling »Stochastic formulation – II is determined at time t, to be implemented at time t’ is determined at time t’, to be implemented at time t’+1 »Important to recognize information content At time t, is deterministic. At time t, is stochastic.

53. The unit commitment problem  The unit commitment problem »Rolling forward with perfect forecast of actual wind, demand, … hour 0-24 hour 25-48 hour 49-72

54. The unit commitment problem  The unit commitment problem »Rolling forward with perfect forecast of actual wind, demand, … hour 0-24 hour 25-48 hour 49-72

55. The unit commitment problem  The unit commitment problem »Using forecast of future wind. hour 0-24 hour 25-48 hour 49-72

56. The unit commitment problem  The unit commitment problem »Using forecast of future wind. hour 0-24 hour 25-48 hour 49-72

57. The unit commitment problem  The unit commitment problem »Stepping through the next day. hour 0-24

58. The unit commitment problem  The unit commitment problem »Stepping forward observing actual wind, making small adjustments hour 0-24

59. The unit commitment problem hour 0-24  The unit commitment problem »Stepping forward observing actual wind, making small adjustments

60. The unit commitment problem hour 0-24  The unit commitment problem »Stepping forward observing actual wind, making small adjustments

61. The unit commitment problem hour 0-24  The unit commitment problem »Stepping forward observing actual wind, making small adjustments

62. The unit commitment problem hour 0-24  The unit commitment problem »Stepping forward observing actual wind, making small adjustments

63. The unit commitment problem hour 0-24  The unit commitment problem »Stepping forward observing actual wind, making small adjustments

64. The unit commitment problem hour 0-24  The unit commitment problem »Stepping forward observing actual wind, making small adjustments

65. The unit commitment problem  Papers can be divided into four categories with respect to the handling of information: »The model is wrong, and the paper is implementing an incorrect algorithm. “Incorrect” means cheating by using information from the future. »The model is wrong (or imprecise), but the experimental work is correct. Many people know how to do a proper simulation, but don’t know how to model it. »The model is correct, but the experimental work is wrong. The programmer cheated by using information from the future; the modeler used proper mathematics that did not match the code. »The model is correct, and the experimental work is correct.

66. The unit commitment problem  Safest way to write out the objective function »Our policy combines: Lookahead policy to solve unit commitment problem Myopic policy to solve hourly adjustment problem. Myopic policy exploits tunable parameters (p,q) in the lookahead policy. We can write the dependence on (p,q) using

67. The unit commitment problem  The value of optimizing (p,q)

68. The unit commitment problem  Energy profile with 2 percent from wind © 2010 Warren B. Powell Slide 68

69. The unit commitment problem Slide 69 © 2010 Warren B. Powell  Forecast from unit commitment model

70.  Energy profile with 20 percent from wind The unit commitment problem Slide 70 © 2010 Warren B. Powell

71. The unit commitment problem  The effect of modeling uncertainty in wind

72.  With better wind forecasts The unit commitment problem Slide 72 © 2010 Warren B. Powell

73. The unit commitment problem  With better wind forecasts: »Using a stochastic model that properly captures the flow of information, we can quantify the value of better forecasts. ($ millions) Total Day-ahead system cost Total Real-time system cost Total Generator dispatch cost Total Generator turndown savings 20% Wind – forecast with past 7 days at corresponding hour Winter69.24079.58411.1090.766 Summer83.88690.2227.6061.269 20% Wind – forecast with past 7 hour Winter61.38968.7647.7300.354 Summer78.29985.0007.6080.907 Slide 73 © 2010 Warren B. Powell

74. Slide 74 Lecture outline  The problem of uncertainty  Modeling stochastic optimization problems  Energy storage portfolios  The unit commitment problem for PJM  Long-term energy resource planning

75. Slide 75 Goals for an energy policy model  Potential questions »Policy questions How do we design policies to achieve energy goals (e.g. 20% renewables by 2015) with a given probability? How does the imposition of a carbon tax change the likelihood of meeting this goal? What might happen if ethanol subsidies are reduced or eliminated? What is the impact of a breakthrough in batteries? »Energy economics What is the best mix of energy generation technologies? How is the economic value of wind affected by the presence of storage? What is the best mix of storage technologies? How would climate change impact our ability to use hydroelectric reservoirs as a regulating source of power? © 2010 Warren B. Powell

76. Slide 76 Intermittent energy sources Wind speed Solar energy © 2010 Warren B. Powell

77. Slide 77 Long term uncertainties…. 2010 2015 2020 2025 2030 Tax policy Batteries Solar panels Carbon capture and sequestration Price of oil Climate change © 2010 Warren B. Powell

78. Slide 78 SMART-Stochastic, multiscale model  SMART: A Stochastic, Multiscale Allocation model for energy Resources, Technology and policy »Stochastic – able to handle different types of uncertainty: Fine-grained – Daily fluctuations in wind, solar, demand, prices, … Coarse-grained – Major climate variations, new government policies, technology breakthroughs »Multiscale – able to handle different levels of detail: Time scales – Hourly to yearly Spatial scales – Aggregate to fine-grained disaggregate Activities – Different types of demand patterns »Decisions Hourly dispatch decisions Yearly investment decisions Takes as input parameters characterizing government policies, performance of technologies, assumptions about climate © 2010 Warren B. Powell

79. Slide 79 The annual investment problem 2008 2009 © 2010 Warren B. Powell

80. Slide 80 The hourly dispatch problem Hourly electricity “dispatch” problem © 2010 Warren B. Powell

81. Slide 81 The hourly dispatch problem  Hourly model »Decisions at time t impact t+1 through the amount of water held in the reservoir. Hour t Hour t+1 © 2010 Warren B. Powell

82. Slide 82 The hourly dispatch problem  Hourly model »Decisions at time t impact t+1 through the amount of water held in the reservoir. Value of holding water in the reservoir for future time periods. Hour t © 2010 Warren B. Powell

83. Slide 83 The hourly dispatch problem © 2010 Warren B. Powell

84. Slide 84 The hourly dispatch problem 2008 Hour 1 2 3 4 8760 2009 1 2 © 2010 Warren B. Powell

85. Slide 85 The hourly dispatch problem 2008 Hour 1 2 3 4 8760 2009 1 2 © 2010 Warren B. Powell

86. Slide 86 SMART-Stochastic, multiscale model 2008 2009 © 2010 Warren B. Powell

87. Slide 87 SMART-Stochastic, multiscale model 2008 2009 © 2010 Warren B. Powell

88. Slide 88 SMART-Stochastic, multiscale model 2008 2009 2010 2011 2038 © 2010 Warren B. Powell

89. Slide 89 SMART-Stochastic, multiscale model 2008 2009 2010 2011 2038 © 2010 Warren B. Powell

90. Slide 90 SMART-Stochastic, multiscale model 2008 2009 2010 2011 2038 ~5 seconds © 2010 Warren B. Powell

91. Slide 91 Approximate dynamic programming Step 1: Start with a pre-decision state Step 2: Solve the deterministic optimization using an approximate value function: to obtain. Step 3: Update the value function approximation Step 4: Obtain Monte Carlo sample of and compute the next pre-decision state: Step 5: Return to step 1. Simulation Deterministic optimization Recursive statistics © 2010 Warren B. Powell

92. Slide 92 SMART-Stochastic, multiscale model  Use statistical methods to learn the value of resources in the future.  Resources may be: »Stored energy Hydro Flywheel energy … »Storage capacity Batteries Flywheels Compressed air »Energy transmission capacity Transmission lines Gas lines Shipping capacity »Energy production sources Wind mills Solar panels Nuclear power plants Amount of resource Value © 2010 Warren B. Powell

93. Slide 9393 Iterations Percentage error from optimal 0.06% over optimal Benchmarking on hourly dispatch 2.50 2.00 1.50 1.00 0.50 0.00  ADP objective function relative to optimal LP © 2010 Warren B. Powell

94. Slide 94 Benchmarking on hourly dispatch  Optimal from linear program Optimal from linear program Reservoir level Demand Rainfall © 2010 Warren B. Powell

95. Slide 9595 Benchmarking on hourly dispatch ADP solution Reservoir level Demand Rainfall  Approximate dynamic programming © 2010 Warren B. Powell

96. Slide 96 Benchmarking on hourly dispatch Optimal from linear program Reservoir level Demand Rainfall  Optimal from linear program © 2010 Warren B. Powell

97. Slide 97 Benchmarking on hourly dispatch ADP solution Reservoir level Demand Rainfall  Approximate dynamic programming © 2010 Warren B. Powell

98. Slide 98 Multidecade energy model  Optimal vs. ADP – daily model over 20 years 0.24% over optimal © 2010 Warren B. Powell

99. Slide 99 Energy policy modeling  Traditional optimization models tend to produce all-or-nothing solutions Cost differential: IGCC - Pulverized coal Pulverized coal is cheaper IGCC is cheaper Investment in IGCC Traditional optimization Approximate dynamic programming © 2010 Warren B. Powell

100. Slide 100100 Time period Precipitation Sample paths Stochastic rainfall © 2010 Warren B. Powell

101. Slide 101101 Reservoir level Optimal for individual scenarios Time period ADP Stochastic rainfall © 2010 Warren B. Powell