1. Uwe A. Schneider www.fnu.zmaw.de
Dynamic Optimization Dynamic Optimization BP/LP (nur für das Wahlpflichtfach Umweltökonomie) 2st Di 10-12, Geomatikum
Uwe A. Schneider

2. Topics Review Integration, Simple Differential Equations
Calculus of Variation
Optimal Control Theory
Phase Diagrams / Stability Analysis
Dynamic Programming

3. Integration and Dynamic Optimization
Integration often needed for solving systems of differential equations
Euler's Equation of COV
Hamiltonian of Optimal Control
more difficult than differentiation
computers and look-up tables make life easier

4. Standard Integrals

5. Standard Integrals

6. Simple Integration Rules
Sums:
Powers:
Constant coefficients:
Log:
Exponentials:

7. Variable Separation
Derived from chain rule of differentiation

8. Integration by Parts
Derived from product rule of differentiation

9. Integration by Substitution
Derived from chain rule of differentiation

10. Integrating Factor
A function by which an ordinary differential equation is multiplied in order to make it integrable
Used to aid solving
linear first and higher order differential equations
bernoulli equations
non-exact equations
several others
Makes a differential equation look like a known antiderivative
A given differential equation may have zero, 1, or more integrating factors

11. Differential Equations, 1
Equations that involve dependent variables and their derivatives with respect to the independent variables are called differential equations
Ordinary Differential Equation: Differential equations that involve only ONE independent variable

12. Differential Equations, 2
Order: The order of a differential equation is the highest derivative that appears in the differential equation
Degree: The degree of a differential equation is the power of the highest derivative term.
… Second order of degree 3

13. Differential Equations, 2
Linear … if there are no multiplications among dependent variables and their derivatives … all coefficients are functions of independent variables
Non-linear … do not satisfy the linear condition
Quasi-linear … if there are no multiplications among all dependent variables and their derivatives in the highest derivative term

14. Differential Equations, 3
Homogeneous … if every single term contains the dependent variables or their derivatives.
Non-homogeneous … Otherwise
Autonomous … if independent variable does not appear in the equation
Exact vs. nonexact (see later slides)

15. Linear First Order Differential Equations (FODE)
Constant coefficients:
Variable coefficients:

16. General Solution to Linear FODE
Integrating
Factor

17. Particular Solution to Linear FODE

18. General Solution to Linear FODE
Integrating
Factor

19. Linear Second Order Differential Equations (FODE)
Constant coefficients:
Variable coefficients:

20. Bernoulli Equation
Fundamental equation of motion for neo-classical growth models with Cobb-Douglas technology
Use integrating factor: (1-a)y-a
Substitute k by z = y(1-a) yielding:
Solve this linear first order differential equation
Substitute z by y = z1/(1-a)

21. Exact Equations, 1

22. Exact Equations, 2

23. Exact Equations, 3 Integrate to find (y)
Note that (y) is a function of y only. Therefore, in the expression giving '(y) the variable, x, should disappear.
Write down the function F(x,y)
All solutions given by F(x,y) = k
Alternatively, one can integrate over y and use (x)

24. Conversion of Nonexact Equations

25. Calculus of Variation Find path x(t), which optimizes
F(.) is twice differentiable
Starting and end points are known

26. Derivation of First Order Conditions
Decompose x(t) into optimal path x*(t) + deviation a*h(t)
Setup
Evaluate g’(0) = 0 (Optimum is where g' = 0 and a = 0)

28. First Order Necessary Condition
= Euler Equation

29. Second Order Necessary Conditions
= Legendre condition
Maximization:
Minimization:

30. Sufficient Conditions
F is jointly concave in x and x' for a maximization
F is jointly convex in x and x' for a minimization

31. Production and Inventory Planning Example
Need to deliver B units at T
Production cost rise linearly with production rate
Cost of holding inventory is constant per unit of time
Zero inventory at beginning
Minimize total cost

32. Production and Inventory Planning

33. Problem Extensions Fixed starting point - free end value
Fixed starting point - free horizon
Transversality conditions
Salvage value
Several Functions

34. Free End Value

35. Free End Value

36. Free Starting Value, Solution
Free Starting and End Value, Solution

37. Free Horizon

39. Solution to Free Horizon Problem

40. Terminal Function

41. Solution to Terminal Function Problem

42. Terminal Function with Salvage Value

43. Terminal Function with Salvage Value Solution

44. Several Functions

45. Several Functions, Solution

46. Limitations Functional constraints Set-up Continuity Differentiability
Integrability (to solve for x(t))
Set-up
Continuous decisions
Simple functions

47. Optimal Control

48. Optimal Control (OC) Generalization of Calculus of Variation
Pontryagin, L.S. et al. late 1950's
State variable(s) … x(t)
Control variable(s) … u(t)
can deal with corner points
can have binary variables (0,1)

49. } OC – Simplest Problem Objective function State equation Boundary
Conditions

50. Relationship to COV letting u(t) = x'(t)
yields calculus of variation problem with free end value

51. OC – Functional restrictions
f and g need to be differentiable functions
u(t) piecewise continuous
x(t) continuous
u(t) t
x(t)

52. OC – Necessary Conditions, 1

53. OC – Necessary Conditions, 2
Define u*(t) as optimal path of control
Consider family of comparison curves:
u*(t) = u(t) + ah(t)
h(t) … some fixed function
a … parameter
u*(t) yields x*(t)
Let y(t,a) denote the state variable x(t)
Then, y(t,0) = x*(t) and y(t0,a) = x0

54. OC – Necessary Conditions, 3
The Maximum of J(a) is at a = 0. Hence, J'(0) = 0.

55. OC – Necessary Conditions, 4
Use Leibnitz Rule to obtain:

56. OC – Necessary Conditions, 5
dt1/da = dt0/da = 0 (for now)
dy(t0,0)/da = 0
Thus, we have

57. OC – Necessary Conditions, 6
Let (t) obey the differential equation
Multiplier Equation
(Costate, Adjoint, Auxiliary,
or Influence Equation)
This plus J'(0)=0 requires

58. OC – Necessary Conditions, 7
Above equation must hold for any h(t)
This implies the necessary condition
Optimality condition

59. OC – Summary If u*(t) and x*(t) maximize objective,
then there is a continuously differentiable function (t) which satisfies:
state equation
multiplier equation
optimality condition
for

60. Hamiltonian
Define
or in short notation:

61. Hamiltonian 1 = Optimality Condition 2 = Multiplier Equation
3 = State Equation

62. OC, Sufficient Conditions
f and g jointly concave in x and u (max) and   0
g is linear in x and u, f is concave in x and u,  can be of any sign
f is concave, g is convex,   0

63. OC vs. Standard Calculus
Standard calculus for static setting
Optimal control for dynamic setting
If time is not important, we have
u*(t) = constant and du/dt = u' = 0
x*(t) = constant and dx/dt = x' = 0
*(t) = constant and d/dt = ' = 0
Necessary conditions from OC reduce to:
Necessary conditions
for a constrained, two
variable Lagrange problem {

64. OC – Euler Equation {
COV problem written
as OC problem {
Hamiltonian

65. OC – Several Variables, 1

66. OC – Several Variables, 2

67. OC – Several Variables, 3

68. OC – Several Variables, 4
Conditions can easily be extended to n variables:

69. Various Endpoint Restrictions
Fixed end point
Free end value
Other terminal constraints
Positive (restricted) end value
Salvage value

72. Transversality Conditions

73. Discounting

74. Current Value Hamiltonian

75. Infinite Time Horizon
transversality condition is replaced by the assumption that the optimal solution approaches a steady state
Solution conditions form a system of differential equation´s, which can be checked for equilibrium states and their stability properties

76. Infinite Horizon, Example
n people, x of n people know the product, their purchases generate profits of P(x), x(t)u(t) people can learn through contact with knowledgeable people x(t), u(t) is the contact rate, which can be influenced by the firm at cost C(u), only 1-x/n people will be newly informed, people forget at rate bx(t)

79. OC – Interpretation
Economic optimization principle: marginal benefits = marginal cost
Dynamic optimization: marginal benefit today = marginal benefit tomorrow = marginal opportunity cost tomorrow

80. OC – Consumption / Investment
Consume more (less) today – invest less (more) today – have less (more) capital tomorrow – consume less (more) tomorrow
Dynamic Optimality: Marginal utility from consumption today = marginal (contribution of investment today to) utility from consumption tomorrow = marginal opportunity cost of consumption today

81. Dorfman – The problem

82. Dorfman – Optimality Condition
The choice variable at every instant should be selected so that the marginal immediate gains are in balance with the value of the marginal contribution to the accumulation of capital

83. Dorfman – Multiplier Condition
To an economist, it [ ] is the rate at which the capital is appreciating is therefore the rate at which a unit of capital depreciates at time t. … In other words, a unit of capital loses value or depreciates as time passes at the rate at which its potential contribution to profits becomes its past contribution. … [or] Each unit of the capital good is gradually decreasing in value at precisely the same rate at which it is giving rise to valuable outputs. We can also interpret as the loss that would be incurred if the acquisition of a unit of capital were postponed for a short time.

84. OC – Production / Storage
Produce more (less) today – have higher (lower) marginal production cost today – have higher (lower) storage cost between today and tomorrow – have less (more) production tomorrow
Dynamic Optimality: Marginal cost of producing an additional amount today plus storing this amount today = marginal cost of producing the additional amount tomorrow

85. Production / Storage, 2 Solving this we obtain
the following condition:
Marginal change in production cost = Marginal storage cost

86. Production / Storage, 3

87. OC – Mine Extraction
Mine more (less) today – Have higher (lower) extraction cost today – Have lower (higher) price today – Have less (more) extraction tomorrow – Have higher (lower) price tomorrow
Dynamic optimality: The marginal value of an extraction today = the marginal value of an extraction tomorrow (Hotelling's Rule)

88. Exhaustible Resource Dynamics
Introduced 1931 by Hotelling: "The Economics of Exhaustible Resources." The Journal of Political Economy 39(2):137-75
Very intuitive
Applies to any depletable asset
Optimal control was not discovered yet
Optimal control is very suitable to context

89. The basic Hotelling model
How might a depletable (exhaustible) resource be optimally used over time?
Stock of resource x(t)
Rate of depletion z(t)
Revenue = p(t)z(t)
Cost are ignored (initially)
Perfect competition

90. The basic Hotelling model
z(t)
p(t) D PS CS

91. The basic Hotelling model

92. Optimal price grows at discount rate

93. Hotelling Conclusions
Price of resource increases at the rate of interest
The marginal value of the (last) resource unit is constant across time

94. OC – Forest Management
Harvest (keep) stand today – have some (no) timber sales today – have no (more) timber sales from unharvested stand tomorrow – have some (no) timber value tomorrow from replanting today
Dynamic Optimality: Marginal net benefit of harvesting today = marginal net benefit from delaying harvest until tomorrow

95. OC – Fishery
Catch and sell more (less) today – Have less (more) fish tomorrow – Have lower (higher) fishing cost tomorrow
Dynamic optimality: The marginal value of catching a fish today = the marginal value of letting the fish grow and multiply today and catching it tomorrow
Externalities prevent many private fisheries from reaching the socially optimal dynamic path

96. Stability Analysis Phase Diagrams

97. Algebraic and graphical analysis of 2x2 system of autonomous differential equations

98. Phase Planes (Diagrams)
Graphical analysis tool for 2x2 system

99. Trajectories
Solutions to the system are functions x1 = x1(t) and x2 = x2(t) . If you consider this solution as parametric equations*, the graph in the x1x2-plane is called a trajectory and the x1x2 -plane is called the phase plane
*A parametric equation explicitly relates two or more variables in terms of one or more independent parameters

100. Critical Points points of an autonomous system such that x1' = x2' = 0
without disturbance, system is in equilibrium
stability of critical points depends on behavior in the neighborhood

101. Nature of Critical Points
Nodes
proper or improper
stable or unstable
unstable saddle points
Spiral points
stable Centers

102. Stable Improper Node

103. Unstable Saddle Point

104. Stable Proper Node

105. Unstable Spiral Point

106. Steps to Draw Phase Diagram
Solve for inequalities x1'  0 and x2'  0
Find the equilibriums x1' = 0 and x2' = 0
Graph the isoclines
Using the inequalities found in 1, sketch the trajectories for x1 and x2
Linearly approximate the system
Find Eigenvalues of that system and evaluate stability
Find general solution and compute trajectories

107. Linear Approximation based on Taylor's Theorem
ignore higher order terms

108. Linear Approximation

109. Solving the System, 1

110. Solving the System, 2

111. Solving the System, 3
 is called Eigenvalue or characteristic root, k is the associated eigenvector
Eigenvalues reflect and determine the stability of the system
 can be distinct real roots, identical real roots, or conjugate complex roots of the form

112. Solution with real roots
the general solution is given by
where c's are constants, 's are characteristic roots (eigenvalues) and k's are associated eigenvectors

113. Solution with complex roots

114. Stability Conclusions
Consider the 2x2 linear system
dx1/dt = a11x1 + a12x2
dx2/dt = a21x1 + a22x2
The characteristic equation is:
2- (a11+a22) + (a11a22- a12a21) = 0 or
2- (Trace A) + (det A) = 0
Let  = (Trace A)2 - 4(det A) denote the discriminant of the characteristic equations

115. Stability Conclusions
Characteristic roots
Characteristic Equation
Nature of Critical Point
Stability of the Critical Point
pure imaginary
<0, a11+a22=0
center
stable
complex but not pure imaginary
<0, a11+a220, real part of <0
spiral point
<0, a11+a220,
real part of >0
unstable

116. >0, det>0 improper node stable unstable >0, det<0
Characteristic roots
Characteristic Equation
Nature of Critical Point
Stability of the Critical Point
>0, det>0
improper node
stable
unstable
>0, det<0
saddle point
=0, a11=a22, a120, a210
=0, a11=a22, a12=a21=0
proper node

117. Stability Conditions Leonard and Van Long:
Stable if and only if its characteristic roots have negative real parts
A saddle point occurs if and only if the determinant of A is negative
A sufficient condition for instability is that the trace of A > 0

118. Computation of Trajectories
must have initial point (x10, x10) condition
substitute initial point condition in general solution
determine value of constants
substitute constants in general solution to get particular solution

119. Example Problem a) dx1/dt = 9x1 + 5x2 dx2/dt = –6x1 – 2x2
b) dx1/dt = x1 – 5x2
dx2/dt = x1 – 3x2
x1(0) = x2(0) = 1

120. Competing Species Problem
The nonlinear system is given by
dx/dt = x(e1-s1x-a1y) = f(x,y)
dy/dt = y(e2-s2y-a2x) = g(x,y)
Linearize about the critical points
f(x,y) = (e1-2s1x*-a1y*) (x-x*) - a1x*(y-y*)
g(x,y) = -a1x* (x-x*) + (e2-2s2y*-a2x*) (y-y*)
Find critical points and evaluate them
Draw phase diagram

121. Competing Species Example
The nonlinear system is given by
dx/dt = x(1.5-x-0.5y)
dy/dt = y(2-y-0.75x)
Determine critical points (equilibrium populations!
Identify the type and stability properties of each!
Sketch a phase diagram!

122. More on Species Models
Competing species problem also applies to predator prey relationships
Predator (x) benefits from prey (y), thus a1>0
Prey (y) suffers from predators (x), thus a2<0
Thus, coefficient signs of a's determine relationship (predator-prey, competing species, symbiotic species)

123. Market Stability Walrasian price adjustment dpx/dt=m+r11px +r12py
dpy/dt=m+r21px +r22py
Gross substitutability (r12>0, r21>0) plus law of demand (r11<0, r22<0) implies stability

124. Market Stability Examples
dpx/dt = px + 2py
dpy/dt= px - 4py
dpx/dt = px + 2py
dpy/dt= px - 6py

125. Dynamic Programming

126. Dynamic Programming Discrete time frame Multi-stage decision problem
Solves backwards

127. Dynamic Programming
All multistage decision problems can be formulated in terms of dynamic programming
Not all multistage decision processes can be solved by DP
Not all DP problems are multistage decision problems (may be 1 decision stage within dynamic problem)

128. Multistage Decision Process
... characterized by the task of finding a sequence of decisions (or path) which maximizes (or minimizes) an appropriately defined objective function

129. Stage
... the discrete point in time at which a decision can be made.
State
... Condition of the process at a particular stage ... Defined by the value of all state variables and other qualitative characteristics

130. State Variables (St)
... variables to describe the condition or state of the system at each stage
Usually the hardest part of a DP model to develop
Must describe the system completely enough to give “good” decision rules but remain small enough to have a manageable decision rule and computer program

131. Decision (Xt) Planning Horizon (T)
... variables which the decision maker controls at each stage – these variables control the state of the system in the next stage (state transition)
Planning Horizon (T)
... finite or infinte

132. Return Function Policy
... gives the immediate returns given the state, stage, and decision made
Policy
... defines the sequence of decisions to be made for a given state. In DP a decision is given for all possible combinations of state at each stage

133. Optimal Policy
The sequence of decision (policy) that optimizes (maximizes or minimizes) the objective function
If decisions are separated by large time intervals, future returns may be discounted

134. Bellman‘s Principle of Optimality
Fundamental concept forming the basis for DP formulation
An optimal policy has the property that, whatever the initial state and decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decisions

135. Markovian Requirement
An optimal policy starting in a given state depends only on the state of the process at that stage and not on the state at preceding stages. The path is of no consequence, only the present stage and state
The state variables fully describe the state of the system at each stage and capture all past decisions

136. Multi-Stage Decision Process
Return r1
Return r2
Return r3
Terminal
Value S1
Stage 1 S2
Stage 2
S2 ... ST
Stage 3
Decision x1
Decision x2
Decision xT

137. Traveling Salesman
Start 1
East 2 3 4 5 6 7 8 9
End 10
West

138. Insurance Policy Cost T O 2 3 4 5 6 7 8 9 10 F 1 R M

139. Minimize the cost for each run?
1 – 2 – 6 – 9 – 10
Total cost = 14
However,
1 – 4 – 6 – 9 – 10
Total cost = 12 !

140. Minimize the cost for each run?
Ignores basic tenet of dynamic optimization
Basic Tenet – by taking into account future consequences of present acts one is led to make choices though possibly sacrificing some present rewards will lead to a preferred sequence of events

141. Enumerate all possible routes?
18 in this example
Will give an optimal solution
Harder to as problem gets larger
Curse of dimensionality

142. Curse of Dimensionality
Computational Curse
Formulation Curse
Large Decision Rule Curse
Counterintuitive Decision Rule Curse
Acceptance Curse (by Analyst and Decision Maker)

143. Dynamic Programming
Reduce an n-period optimization process to n one period optimization processes
More efficient than total enumeration
Usually works backwards

144. Discrete – Time Markov Chains
Many real-world systems contain uncertainty and evolve over time.
Stochastic processes and Markov chains are probability models for such systems
A discrete-time stochastic process is a sequence of random variables x0, x1, x2, … typically denoted {xt}.

145. State Occupancy Probability Vector
Let π be a row vector. Denote πi to be the ith element of the vector with n elements. If π is a state occupancy probability vector, then πi(t) is the probability that a DTMC has value i (or is in state i) at time-step t

146. State Transition Probabilities

147. Transient Behavior of DTMC
π(t) = π(t-1)P
π(t-1)= π(t-2)P
π(t) = [π(t-2)P]P = π(t-2)P2
π(t-2)= π(t-3)P
π(t) = [π(t-3)P]P2 = π(t-3)P3
π(t) = π(0)Pt

148. Convergence t+1 = t P
State
...
Year t
Year t+1 1 50 2 75 3 83

149. Decision Rule Converging vs. non-converging Converging useful
Salvage value won‘t impact convergence property

150. Empirical Example
Soil carbon sequestration from land use will receive premium
Continuous application of a certain tillage system leads to specific soil carbon equilibrium (after few decades)
How to model optimal decision path?

151. Empirical Example Two tillage systems
Annual decisions over multi-decade horizon
Limited land availability
Carbon price

152. Tillage Effect on Soil Carbon
Zero Tillage
Soil Carbon
Intensive Tillage
Time

153. Land Use Decision Model
t … time index
r … region index
i … soil type index
u … tillage index
L … available land
vM … market profit
vC … market profit
 … discount factor

154. Soil Carbon Status Dynamics
t … time index
r … region index
i … soil type index
u … tillage index
o … soil carbon state

155. Transition ProbabilitiesII
III IV V
Case

156. Transition Probabilities