1. © 2007 Warren B. Powell Slide 1 The Dynamic Energy Resource Model Lawrence Livermore National Laboratories September 24, 2007 Warren Powell Alan Lamont Jeffrey Stewart Abraham George © 2007 Warren B. Powell, Princeton University

2. © 2007 Warren B. Powell Slide 2 The dynamic energy resource model Questions: »How will the market evolve in terms of the adoption of competing energy technologies? How many windmills, and where? How much ethanol capacity? How will the capacity of coal, natural gas and oil evolve? »What government policies should be implemented? Carbon tax? Cap and trade? Tax credits for windmills and solar panels? Tax credits for ethanol? »Where should we invest R&D dollars? Ethanol or hydrogen? Batteries or windmills? Hydrogen production, storage or conversion?

3. © 2007 Warren B. Powell Slide 3 The dynamic energy resource model Features we need: »Multiple time scales Model will plan decades into the future, but reflect decisions and processes that occur on hourly, daily, seasonal and yearly levels. »Multiple forms of uncertainty We will model dynamic information processes that describe the evolution of technology, climate, weather, prices and wind. »Multiple levels of spatial granularity The model will be able to run at different levels of spatial aggregation, capturing the geographic substitution of different types of energy. »Multi-attribute representation of markets We want to be able to distinguish energy demands to capture usage and lifestyle patterns.

4. © 2007 Warren B. Powell Slide 4 Outline A deterministic model A stochastic, dynamic energy model ADP for energy capacity management

5. © 2007 Warren B. Powell Slide 5 Outline A deterministic model A stochastic, dynamic energy model ADP for energy capacity management

6. © 2007 Warren B. Powell Slide 6 Deterministic models Deterministic, linear programming-based models »Basic model: »Features: Can model flows of energy and substitution of energy resources over time. Assumes a deterministic view of the world (everything is known now).

7. © 2007 Warren B. Powell Slide 7 Deterministic models A single large linear program: Time Space

8. © 2007 Warren B. Powell Slide 8 Deterministic models Limitations »Unable to model uncertainty in technology, climate, prices. »Unable to model activities at a high level of detail. Large linear program limits the number of rows, which grows rapidly as we use finer representations of resources and markets. »Traditionally uses a discrete time representation, making it hard to handle fine time scales (e.g. hourly) over long horizons (e.g. 50 years).

9. © 2007 Warren B. Powell Slide 9 Outline A deterministic model A stochastic, dynamic energy model ADP for energy capacity management

10. © 2007 Warren B. Powell Slide 10 Stochastic, dynamic model The state of a resource:

11. © 2007 Warren B. Powell Slide 11 Stochastic, dynamic model Modeling multiple energy resources: »The attributes of a single resource: »The resource state vector: »The information process:

12. © 2007 Warren B. Powell Slide 12 Stochastic, dynamic model Modeling market demands: »The attributes of a single type of demand: »The demand state vector: »The information process:

13. © 2007 Warren B. Powell Slide 13 Stochastic, dynamic model The system state:

14. © 2007 Warren B. Powell Slide 14 Stochastic, dynamic model The three states of our system »The state of a single resource/entity »The resource state vector »The system state vector

15. © 2007 Warren B. Powell Slide 15 Stochastic, dynamic model The decision variable:

16. © 2007 Warren B. Powell Slide 16 Stochastic, dynamic model Exogenous information:

17. © 2007 Warren B. Powell Slide 17 Stochastic, dynamic model Hourly »Daily temperature variation »Wind »Equipment failures Daily »Fluctuation in spot prices »Short term demand »Major weather events »Transportation delays (movement of coal and oil) Monthly »Seasonal variation (temperature, water flow for hydro, population shifts) »Medium term weather patterns »Significant supply disruptions (major hurricane, wars) Yearly »Changes in technology »Demand patterns (SUV’s) »Long term climate cycles (including global warming) »Spatial patterns in population growth »New supply discoveries (major oil fields) »Intervention of foreign governments in markets »Long term supply contracts

18. © 2007 Warren B. Powell Slide 18 Stochastic, dynamic model The transition function t t+1

19. © 2007 Warren B. Powell Slide 19 Stochastic, dynamic model Our strategy: »Basic model: »Features: Simulation-based – We simulate forward in time using a very general-purpose transition model. Handles virtually any form of uncertainty. Can use a range of policies for different types of decisions, from simple dispatch rules to more sophisticated policies that look into the future.

20. © 2007 Warren B. Powell Slide 20 Information and decisions Information T – changes in technology S – changes in energy supplies P – changes in energy prices W – Weather Decisions I – Changes in energy capacity infrastructure (new plants, new fields) S – Short term changes in supplies R – R&D investments M – Market response T T T T W W W W W W W W W W W W W W W W W W S S S S S S S S P P P P P P P P P P P P P P P P P P P IIII R RRRRRRRRR S SSS S SSSSSSSS SS SSS MMM MMM MM M

21. © 2007 Warren B. Powell Slide 21 Making decisions Dispatch decisions: »Use the technology with the lowest marginal cost. »Small linear program to handle substitution of different types of power. GENERATORS MARKETS 2 1 4 3 A C B I II III ENERGY SOURCES

22. © 2007 Warren B. Powell Slide 22 Making decisions Hydro power management »Forecast inflow and outflow to reservoirs to determine amount available for generating electricity

23. © 2007 Warren B. Powell Slide 23 Making decisions Purchasing new capacity: »A decision to add capacity in year t changes the capacity available in year t+1. »Resource transition function Resource state vector Capacity change decisions Exogenous changes to resources

24. © 2007 Warren B. Powell Slide 24 Making decisions Purchasing new capacity: »We want our capacity acquisition decisions to mimic the intelligence that companies/financial markets make. »We propose to “simulate Wall St.” by solving the capacity acquisition problem as an optimization problem to find the policy that solves: »The optimal policy is characterized by Bellman’s equation: »Problem: Solving this equation is computationally intractable because of the “three curses of dimensionality.”

25. © 2007 Warren B. Powell Slide 25 Approximate dynamic programming Solving the dynamic program using approximate dynamic programming: »Step 1: Break transition into two steps: »Step 2: Formulate value function around post-decision state: »Step 3: Replace value function with approximation »Step 4: Design strategy for updating the approximation

26. © 2007 Warren B. Powell Slide 26 Part VII - CASTLE Lab News CASTLE Lab News New Modeling Language Captures Complexities of Real-World Operations! 75 cents Spans the gap between simulation and optimization. CASTLE Lab announced the development of a powerful new simulation environment for modeling complex operations in transportation and logistics. The dissertation of Dr. Joel Shapiro, it offers the flexibility of simulation environments, but the intelligence of optimization. The modeling language will allow managers to quickly test continued on page 3 Thursday, March 2, 1999

27. © 2007 Warren B. Powell Slide 27

28. © 2007 Warren B. Powell Slide 28 Outline A deterministic model A stochastic, dynamic energy model ADP for energy capacity management

29. © 2007 Warren B. Powell Slide 29 ADP for energy resource management 2008 2009

30. © 2007 Warren B. Powell Slide 30 We have to allocate resources before we know the demands for different types of energy in the future: ADP for energy resource management

31. © 2007 Warren B. Powell Slide 31 We use value function approximations of the future to make decisions now: ADP for energy resource management

32. © 2007 Warren B. Powell Slide 32 This determines how much capacity to provide: ADP for energy resource management

33. © 2007 Warren B. Powell Slide 33 Marginal value: ADP for energy resource management

34. © 2007 Warren B. Powell Slide 34 Using the marginal values, we iteratively estimate piecewise linear functions. ADP for energy resource management

35. © 2007 Warren B. Powell Slide 35 Right derivativeLeft derivative Using the marginal values, we iteratively estimate piecewise linear functions. ADP for energy resource management

36. © 2007 Warren B. Powell Slide 36 Using the marginal values, we iteratively estimate piecewise linear functions. ADP for energy resource management

37. © 2007 Warren B. Powell Slide 37 Piecewise linear, separable value function approximations: ADP for energy resource management

38. © 2007 Warren B. Powell Slide 38 Approximate dynamic programming t

39. © 2007 Warren B. Powell Slide 39 Approximate dynamic programming

40. © 2007 Warren B. Powell Slide 40 Approximate dynamic programming

41. © 2007 Warren B. Powell Slide 41 Approximate dynamic programming Features »Extremely flexible Simulation-based modeling is able to handle high level of detail about energy resources and demands, time scales and uncertainties. Can handle mixed policies: –Myopic policies for dispatch problem –Rolling horizon procedures for hydro –Dynamic programming-based policies for capacity acquisition »Challenges Value function approximations have to be designed to handle the state of technology and climate. Strategies have to be designed to guide the system to reach different goals. Measures for evaluating solution quality (is it realistic? near- optimal?) need to be designed.

42. © 2007 Warren B. Powell Slide 42

43. © 2007 Warren B. Powell Slide 43 The dynamic energy resource model Uncertainties (exogenous information processes) »Technology: Carbon sequestration The cost of batteries, fuel cells, solar panels The storage of hydrogen, efficiency of solar panels, … »Climate: Global and regional temperatures Changing patterns of snow storage on mountains Wind patterns »Markets: Global supplies of oil and natural gas International consumption patterns Domestic purchasing behaviors (SUV’s?) Tax policies The price of oil and natural gas

44. © 2007 Warren B. Powell Slide 44 The dynamic energy resource model Alternative ways of solving large stochastic optimization problems: »Simulation using myopic policies – Using rules to determine decisions based on the current state of the system. Rules are hard to design, and decisions now do not consider the impact on the future. »Deterministic optimization – Ignores uncertainty (and problems are still very large scale). »Rolling horizon procedures – Uses point estimates of what might happen in the future. Will not produce robust behaviors. »Stochastic programming – Cannot handle multiple sources of uncertainty over multiple time periods. »Markov decision processes – Discrete state, discrete action will not scale (“curse of dimensionality”)

45. © 2007 Warren B. Powell Slide 45 Dynamic energy resource management Proposed approach: Approximate dynamic programming »Our research combines mathematical programming, simulation and statistics in a dynamic programming framework. Math programming handles high-dimensional decisions. Simulation handles complex dynamics and high-dimensional information processes. Statistical learning is used to improve decisions iteratively. Solution strategy is highly intuitive – tends to mimic human behavior. »Features: Scales to very large scale problems. Easily handles complex dynamics and information processes. Rigorous theoretical foundation »Research challenge: Calibrating the model. Designing high quality policies using the tools of approximate dynamic programming. Evaluating the quality of these policies.