© 2008 Warren B. Powell Slide 1 The Dynamic Energy Resource Model Warren Powell Alan Lamont Jeffrey Stewart Abraham George © 2007 Warren B. Powell, Princeton University

© 2008 Warren B. Powell Slide 2 Dynamic energy resource management Questions: »How will the market evolve in terms of the adoption of competing energy technologies? How many windmills, and where? How much ethanol capacity? How will the capacity of coal, natural gas and oil evolve? »What government policies should be implemented? Carbon tax? Cap and trade? Tax credits for windmills and solar panels? Tax credits for ethanol? »Where should we invest R&D dollars? Ethanol or hydrogen? Batteries or windmills? Hydrogen production, storage or conversion?

© 2008 Warren B. Powell Slide 3 Dynamic energy resource management Uncertainties: »Technology: Carbon sequestration The cost of batteries, fuel cells, solar panels The storage of hydrogen, efficiency of solar panels, … »Climate: Global and regional temperatures Changing patterns of snow storage on mountains Wind patterns »Markets: Global supplies of oil and natural gas International consumption patterns Domestic purchasing behaviors (SUV’s?) Tax policies The price of oil and natural gas

© 2008 Warren B. Powell Slide 4 Dynamic energy resource management Research challenges: »Making decisions Finding the best decisions (capacity decisions, R&D decisions, government policies) requires solving high-dimensional stochastic, dynamic programs. How do we obtain practical solutions to stochastic, dynamic programs which exhibit state variables with millions of dimensions? »Modeling multiple time scales We have to represent wind, temperature, rain and snow fall, market prices and government policies. This requires modeling hourly, daily, seasonal and yearly dynamics. »Modeling multiple levels of resolution Spatial: We need to represent the location of windmills at state, regional and county levels. Behavioral: We need to capture the differences between travel behavior patterns (long commutes vs. short trips, commercial fleet vehicles vs. personal use), or the difference between light and heavy industrial power use.

© 2008 Warren B. Powell Slide 5 Dynamic energy resource management Alternative ways of solving large stochastic optimization problems: »Simulation using myopic policies – Using rules to determine decisions based on the current state of the system. Rules are hard to design, and decisions now do not consider the impact on the future. »Deterministic optimization – Ignores uncertainty (and problems are still very large scale). »Rolling horizon procedures – Uses point estimates of what might happen in the future. Will not produce robust behaviors. »Stochastic programming – Cannot handle multiple sources of uncertainty over multiple time periods. »Markov decision processes – Discrete state, discrete action will not scale (“curse of dimensionality”)

© 2008 Warren B. Powell Slide 6 Dynamic energy resource management Proposed approach: Approximate dynamic programming »Our research combines mathematical programming, simulation and statistics in a dynamic programming framework. Math programming handles high-dimensional decisions. Simulation handles complex dynamics and high-dimensional information processes. Statistical learning is used to improve decisions iteratively. Solution strategy is highly intuitive – tends to mimic human behavior. »Features: Scales to very large scale problems. Easily handles complex dynamics and information processes. Rigorous theoretical foundation »Research challenge: Calibrating the model. Designing high quality policies using the tools of approximate dynamic programming. Evaluating the quality of these policies.

© 2008 Warren B. Powell Slide 7 Version 2 of issues: Next two slides mimic the previous ones, but more compactly.

© 2008 Warren B. Powell Slide 8 The dynamic energy resource model Questions: »How will the market evolve in terms of the adoption of competing energy technologies? How many windmills, and where? How much ethanol capacity? How will the capacity of fossil fuel generation? »What government policies should be implemented? Carbon tax? Cap and trade? Tax credits for windmills and solar panels? Tax credits for ethanol? »Where should we invest R&D dollars? Ethanol or hydrogen? Batteries or windmills? Hydrogen production, storage or conversion?

© 2008 Warren B. Powell Slide 9 The dynamic energy resource model Features we need: »Multiple time scales Model will plan decades into the future, but reflect decisions and processes that occur on hourly, daily, seasonal and yearly levels. »Multiple forms of uncertainty We will model dynamic information processes that describe the evolution of technology, climate, weather, prices and wind. »Multiple levels of spatial granularity The model will be able to run at different levels of spatial aggregation, capturing the geographic substitution of different types of energy. »Multi-attribute representation of markets We want to be able to distinguish energy demands to capture usage and lifestyle patterns.

© 2008 Warren B. Powell Slide 10 Outline Traditional models: optimization and simulation

© 2008 Warren B. Powell Slide 11 Modeling The challenge: »We understand the problem of simulating physical systems Climate Hydrology Technology »To understand the economics for policy purposes, we need to model decisions. How will electricity flow from generating source to market demand on an hourly basis? How to operate different energy technologies (daily, weekly, seasonal)? How much generation capacity of each type will be added or retired each year?

© 2008 Warren B. Powell Slide 12 Deterministic models Deterministic, linear programming-based models »This is the current class of models used to inform policy-makers. »Basic model: »Features: Can model flows of energy and substitution of energy resources over time. Assumes a deterministic view of the world (everything is known now).

© 2008 Warren B. Powell Slide 13 Deterministic models A time-dependent linear program: Time Space

© 2008 Warren B. Powell Slide 14 Deterministic models Limitations »Unable to model uncertainty in technology, climate, prices. »Unable to model activities at a high level of detail. Large linear program limits the number of rows, which grows rapidly as we use finer representations of resources and markets. »Traditionally uses a discrete time representation, making it hard to handle fine time scales (e.g. hourly) over long horizons (e.g. 50 years). »Deterministic models do not adequately represent the decisions policy-makers are faced with when determining climate policy.

© 2008 Warren B. Powell Slide 15 Simulation models Strengths »Very flexible – able to handle a high level of detail. »Can handle any form of (quantifiable) uncertainty. Weaknesses »Hard to program rules that mimic the intelligence of markets, governments and companies. »Will not recommend the path we should follow to reach a goal.

© 2008 Warren B. Powell Slide 16 Modeling alternatives Simulation »Strengths Extremely flexible High level of detail »Weaknesses Low level of “intelligence” Lower solution quality May have difficulty “behaving” properly with new scenarios. Difficulty adapting to random outcomes. Optimization »Strengths High level of intelligence System behaves “optimally” even with new datasets Reduces data set preparation. »Weaknesses Strict rules on problem structure Low level of detail Inflexible! There is often a competition between deterministic “optimizers” and “simulators.”... Why are we asking this question?

© 2008 Warren B. Powell Slide 17 Outline Elements of a dynamic energy model

© 2008 Warren B. Powell Slide 18 Stochastic, dynamic model Modeling energy resources

© 2008 Warren B. Powell Slide 19 Stochastic, dynamic model The system state:

© 2008 Warren B. Powell Slide 20 Stochastic, dynamic model The decision variables:

© 2008 Warren B. Powell Slide 21 Stochastic, dynamic model Exogenous information:

© 2008 Warren B. Powell Slide 22 Stochastic, dynamic model Hourly »Daily temperature variation »Wind »Equipment failures Daily »Fluctuation in spot prices »Short term demand »Major weather events »Transportation delays (movement of coal and oil) Monthly »Seasonal variation (temperature, water flow for hydro, population shifts) »Medium term weather patterns »Significant supply disruptions (major hurricane, wars) Yearly »Changes in technology »Demand patterns (SUV’s) »Long term climate cycles (including global warming) »Spatial patterns in population growth »New supply discoveries (major oil fields) »Intervention of foreign governments in markets »Long term supply contracts

© 2008 Warren B. Powell Slide 23 Stochastic, dynamic model The transition function t t+1 Captures the evolution of all aspects of the system over time.

© 2008 Warren B. Powell Slide 24 Stochastic, dynamic model Our strategy: »Basic model: »Features: Simulation-based – We simulate forward in time using a very general-purpose transition model. Handles virtually any form of uncertainty. Can use a range of policies for different types of decisions, from simple dispatch rules to more sophisticated policies that look into the future.

© 2008 Warren B. Powell Slide 25 Information and decisions Information T – changes in technology S – changes in energy supplies P – changes in energy prices W – Weather Decisions I – Changes in energy capacity infrastructure (new plants, new fields) S – Short term dispatch decisions R – R&D investments M – Market response T T T T W W W W W W W W W W W W W W W W W W S S S S S S S S P P P P P P P P P P P P P P P P P P P IIII R RRRRRRRRR S SSS S SSSSSSSS SS SSS MMM MMM MM M

© 2008 Warren B. Powell Slide 26 Making decisions Dispatch decisions: »Use the technology with the lowest marginal cost. »Each hour solve a small linear program to handle substitution of different types of power. GENERATORS MARKETS A C B I II III ENERGY SOURCES

© 2008 Warren B. Powell Slide 27 Making decisions Hydro power management »Forecast inflow and outflow to reservoirs to determine amount available for generating electricity

© 2008 Warren B. Powell Slide 28 Making decisions Purchasing new capacity: »A decision to add capacity in year t changes the capacity available in year t+1. »We want our capacity acquisition decisions to mimic the intelligence that companies/financial markets make. »We propose to “simulate Wall St.” by solving the capacity acquisition problem as an optimization problem that balances value now with the expected value of the future.

© 2008 Warren B. Powell Slide 29 Outline General ADP algorithmic strategy

© 2008 Warren B. Powell Slide 30 Our general algorithm Step 1: Start with a post-decision state Step 2: Obtain Monte Carlo sample of and compute the next pre-decision state: Step 3: Solve the deterministic optimization using an approximate value function: to obtain. Step 4: Update the value function approximation Step 5: Find the next post-decision state: Simulation Optimization Statistics

© 2008 Warren B. Powell Slide 31 Competing updating methods Comparison to other methods: »Classical MDP (value iteration) »Classical ADP (pre-decision state): »Our method (update around post-decision state):

© 2008 Warren B. Powell Slide 32 Simulating a myopic policy Approximate dynamic programming t

© 2008 Warren B. Powell Slide 33 Simulating a myopic policy Approximate dynamic programming

© 2008 Warren B. Powell Slide 34 Using value functions to anticipate the future Approximate dynamic programming t “Here and now”Downstream impacts

© 2008 Warren B. Powell Slide 35 Approximate dynamic programming Using value functions to anticipate the future

© 2008 Warren B. Powell Slide 36 Approximate dynamic programming Using value functions to anticipate the future

© 2008 Warren B. Powell Slide 37 Approximate dynamic programming Using value functions to anticipate the future

© 2008 Warren B. Powell Slide 38 Part VII - CASTLE Lab News CASTLE Lab News New Modeling Language Captures Complexities of Real-World Operations! 75 cents Spans the gap between simulation and optimization. CASTLE Lab announced the development of a powerful new simulation environment for modeling complex operations in transportation and logistics. The dissertation of Dr. Joel Shapiro, it offers the flexibility of simulation environments, but the intelligence of optimization. The modeling language will allow managers to quickly test continued on page 3 Thursday, March 2, 1999

© 2008 Warren B. Powell Slide 39

© 2008 Warren B. Powell Slide 40 Outline Estimating the value functions

© 2008 Warren B. Powell Slide 41 Energy resource modeling

© 2008 Warren B. Powell Slide 42 We have to allocate resources before we know the demands for different types of energy in the future: Energy resource modeling

© 2008 Warren B. Powell Slide 43 We use value function approximations of the future to make decisions now: Energy resource modeling

© 2008 Warren B. Powell Slide 44 This determines how much capacity to provide: Energy resource modeling

© 2008 Warren B. Powell Slide 45 Marginal value: Energy resource modeling

© 2008 Warren B. Powell Slide 46 Using the marginal values, we iteratively estimate piecewise linear functions. Energy resource modeling

© 2008 Warren B. Powell Slide 47 Right derivativeLeft derivative Using the marginal values, we iteratively estimate piecewise linear functions. Energy resource modeling

© 2008 Warren B. Powell Slide 48 Using the marginal values, we iteratively estimate piecewise linear functions. Energy resource modeling

© 2008 Warren B. Powell Slide 49 Linear value function approximations: Two-stage stochastic programming

© 2008 Warren B. Powell Slide 50 Piecewise linear, separable value function approximations: Two-stage stochastic programming

© 2008 Warren B. Powell Slide 51 Research challenges Approximate dynamic programming: »At the heart of an ADP algorithm is the challenge of finding a value function approximation “that works” Can be used within commercial LP solvers Can be updated (estimated) easily Is stable Provides high quality solutions »Assessing solution quality Is it realistic? –Do we seem to mimic markets and public policy? Is it robust? –Do we achieve energy goals under different scenarios?

© 2008 Warren B. Powell Slide 52 For the dynamic energy resource model, it is not enough to have a value function that depends purely on the resource vector. »The value of coal plants depends on our ability to sequester carbon. »We need to capture the “state of the world” in our value function approximations. Strategies: »Let be the full system state vector, capturing the cost of technologies, government policies, etc. etc. »Let be a set of “features” that appear to be important explanatory variables. Identifying features is the “art” of ADP. »We can then fit value functions that depend on the features. Research challenges

© 2008 Warren B. Powell Slide 53 Research challenges Strategies for fitting »Lookup-table Very general, but suffers from curse of dimensionality »Linear regression with low-dimensional polynomials Can work –depends on the problem. »Kernel regression Powerful strategy that combines lookup-table with regression models. Use within ADP is surprisingly young. Variety of issues unique to ADP.

© 2008 Warren B. Powell Slide 54 Research challenges Approximate dynamic programming: »How do we establish that we are getting “good” solutions? Demonstrate techniques on simpler problems. Compare against other methods for larger problems. »We need algorithms that are fast and stable. Identifying variance reduction methods from the simulation community that work on this problem class. Developing kernel regression techniques for improved fitting of the value function. Finding the best smoothing techniques for recursive updating. Parallel processing for accelerating simulations.

© 2008 Warren B. Powell Slide 55 Research challenges System modeling »Modeling the evolution of technology using compact representations If we invest in technology, how do we describe the change process? »Modeling physical processes at multiple scales Wind, temperature, rainfall at different levels of discretization. »We need a software architecture that allows a larger community to participate in the modeling We need to tap into various types of domain expertise, such as climate modeling, transportation modeling, …

© 2008 Warren B. Powell Slide 56

© 2008 Warren B. Powell Slide 57 Notes: »We have developed a classical, deterministic linear programming model as a basis for comparison and calibration. »Early results show a close match between ADP and linear programming solution on a deterministic dataset (without storage). »We intend to continue to use the LP model as a benchmark. But it is limited in the number of time periods it can handle. We cannot model hourly wind fluctuations over 5+ year horizon.

© 2008 Warren B. Powell Slide 58 LP vs. ADP comparison Cost of ADP solution over LP optimal solution Iteration Cost over LP optimal solution

© 2008 Warren B. Powell Slide 59 Outline Features and capabilities

© 2008 Warren B. Powell Slide 60 Features and capabilities Modeling capabilities »Exogenous changes Hourly wind fluctuations Daily price fluctuations Seasonal rainfall and climate patterns Yearly changes in energy technology »Decisions Adjustments to mix of energy to meet hourly demand Weekly or monthly adjustments to coal and nuclear output Seasonal adjustments to hydroelectric power Yearly changes in energy resource capacity »Features Decisions will adapt to uncertainties such as the state of battery technology or carbon sequestration Capacity decisions will reflect future uncertainties Model can be guided by external policies to meet specific energy goals

© 2008 Warren B. Powell Slide 61 Features and capabilities Potential reports: »Likelihood of reaching energy goals given a particular policy » Usage patterns (hourly, daily, yearly) for different energy sources »Impact of different energy policies on usage patterns and energy goals »???

© 2008 Warren B. Powell Slide 62 Features and capabilities Planned features: »Spatially distributed energy sources and demands Our library already handles this, but we need spatially disaggregate data. »Hydroelectric storage Being developed as we speak. »Modular architecture Library is very flexible, but we have not made specific efforts to allow others to link in modules to model climate change, technology, … Plan is to make the overall architecture highly participatory.