1. Reinforcement Learning
Slides for this part are adapted from those of Dan

3. Main Dimensions Model-based vs. Model-free Passive vs. Active
Model-based  Have/learn action models (i.e. transition probabilities)
Eg. Approximate DP
Model-free  Skip them and directly learn what action to do when (without necessarily finding out the exact model of the action)
E.g. Q-learning
Passive vs. Active
Passive: Assume the agent is already following a policy (so there is no action choice to be made; you just need to learn the state values and may be action model)
Active: Need to learn both the optimal policy and the state values (and may be action model)

4. Main Dimensions (contd)
Extent of Backup
Full DP
Adjust value based on values of all the neighbors (as predicted by the transition model)
Can only be done when transition model is present
Temporal difference
Adjust value based only on the actual transitions observed

6. Does self learning through simulator.
[Infants don’t get to “simulate” the
world since they neither have
T(.) nor R(.) of their world]

8. We are basically doing EMPIRICAL Policy Evaluation!
But we know this will be wasteful
(since it misses the correlation between values of neibhoring states!)
Do DP-based policy
evaluation!

14. 10/7

15. Learning/Planning/Acting

16. World ---- Model (and Simulator)
Add notes..

19. Relating TD and Monte Carlo
Both Monte Carlo and TD learn from samples (traces)
Monte Carlo waits until the trace hits a sink state, and then (discount) adds all the rewards of the trace
TD on the other hand considers the current state s, and the next experienced state s0
You can think of what TD is doing as “truncating” the experience and summarizing the aggregated reward of the entire trace starting from s0 in terms of the current value estimate of s0
Why truncate at the very first state s’? How about going from s s0s1s2..sk and truncate the remaining trace (by assuming that its aggregate reward is just the current value of sk)
(sort of like how deep down you go in game trees before applying evaluation function)
In this generalized view, TD corresponds to k=0 and Monte Carlo corresponds to k=infinity

20. Generalizing TD to TD(l)
Reason:
After Tth state the
remaining infinite
# states will all have
the same aggregated
Backup—but each
is discounted in l.
So, we have a
1/(1- l) factor that
Cancels out the (1- l)
TD(l) can be thought of as doing 1,2…k step predictions of the value of the state, and taking their weighted average
Weighting is done in terms of l such that
l=0 corresponds to TD
l=1 corresponds to Monte Carlo
Note that the last backup doesn’t have (1- l) factor…
No (1- l) factor!

26. 10/12

27. Full vs. Partial Backups

28. Dimensions of Reinforcement Learning.
Dimensions of Reinforcement Learning

33. --Factored TD and Q-learning --Policy search (has to be factored..)
10/14
--Factored TD and Q-learning
--Policy search (has to be factored..)

34. Large State Spaces
When a problem has a large state space we can not longer represent the V or Q functions as explicit tables
Even if we had enough memory
Never enough training data!
Learning takes too long
What to do??
[Slides from Alan Fern]

35. Function Approximation
Never enough training data!
Must generalize what is learned from one situation to other “similar” new situations
Idea:
Instead of using large table to represent V or Q, use a parameterized function
The number of parameters should be small compared to number of states (generally exponentially fewer parameters)
Learn parameters from experience
When we update the parameters based on observations in one state, then our V or Q estimate will also change for other similar states
I.e. the parameterization facilitates generalization of experience

36. Linear Function Approximation
Define a set of state features f1(s), …, fn(s)
The features are used as our representation of states
States with similar feature values will be considered to be similar
A common approximation is to represent V(s) as a weighted sum of the features (i.e. a linear approximation)
The approximation accuracy is fundamentally limited by the information provided by the features
Can we always define features that allow for a perfect linear approximation?
Yes. Assign each state an indicator feature. (I.e. i’th feature is 1 iff i’th state is present and i represents value of i’th state)
Of course this requires far to many features and gives no generalization.

37. Example
Consider grid problem with no obstacles, deterministic actions U/D/L/R (49 states)
Features for state s=(x,y): f1(s)=x, f2(s)=y (just 2 features)
V(s) = 0 + 1 x + 2 y
Is there a good linear approximation?
Yes.
0 =10, 1 = -1, 2 = -1
(note upper right is origin)
V(s) = 10 - x - y subtracts Manhattan dist. from goal reward 6 10 6

38. But What If We Change Reward …
V(s) = 0 + 1 x + 2 y
Is there a good linear approximation?
No. 10

39. But What If… V(s) = 0 + 1 x + 2 y + 3 z Include new feature z
z= |3-x| + |3-y|
z is dist. to goal location
Does this allow a good linear approx?
0 =10, 1 = 2 = 0, 0 = -1
V(s) = 0 + 1 x + 2 y 3 10 3
Feature Engineering….

42. Linear Function Approximation
Define a set of features f1(s), …, fn(s)
The features are used as our representation of states
States with similar feature values will be treated similarly
More complex functions require more complex features
Our goal is to learn good parameter values (i.e. feature weights) that approximate the value function well
How can we do this?
Use TD-based RL and somehow update parameters based on each experience.

43. TD-based RL for Linear Approximators
Start with initial parameter values
Take action according to an explore/exploit policy (should converge to greedy policy, i.e. GLIE)
Update estimated model
Perform TD update for each parameter
Goto 2
What is a “TD update” for a parameter?

44. Aside: Gradient Descent
Given a function f(1,…, n) of n real values =(1,…, n) suppose we want to minimize f with respect to 
A common approach to doing this is gradient descent
The gradient of f at point , denoted by  f(), is an n-dimensional vector that points in the direction where f increases most steeply at point 
Vector calculus tells us that  f() is just a vector of partial derivatives where
can decrease f by moving in negative gradient direction

45. Aside: Gradient Descent for Squared Error
Suppose that we have a sequence of states and target values for each state
E.g. produced by the TD-based RL loop
Our goal is to minimize the sum of squared errors between our estimated function and each target value:
After seeing j’th state the gradient descent rule tells us that we can decrease error by updating parameters by:
squared error of example j
target value for j’th state
our estimated value for j’th state
learning rate

46. Aside: continued depends on form of approximator
For a linear approximation function:
Thus the update becomes:
For linear functions this update is guaranteed to converge to best approximation for suitable learning rate schedule

47. TD-based RL for Linear Approximators
Start with initial parameter values
Take action according to an explore/exploit policy (should converge to greedy policy, i.e. GLIE) Transition from s to s’
Update estimated model
Perform TD update for each parameter
Goto 2
What should we use for “target value” v(s)?
Use the TD prediction based on the next state s’
this is the same as previous TD method only with approximation

48. TD-based RL for Linear Approximators
Start with initial parameter values
Take action according to an explore/exploit policy (should converge to greedy policy, i.e. GLIE)
Update estimated model
Perform TD update for each parameter
Goto 2
Step 2 requires a model to select greedy action
For applications such as Backgammon it is easy to get a simulation-based model
For others it is difficult to get a good model
But we can do the same thing for model-free Q-learning

49. Q-learning with Linear Approximators
Features are a function of states and actions.
Start with initial parameter values
Take action a according to an explore/exploit policy (should converge to greedy policy, i.e. GLIE) transitioning from s to s’
Perform TD update for each parameter
Goto 2
For both Q and V, these algorithms converge to the closest linear approximation to optimal Q or V.

50. Example: Tactical Battles in Wargus
Wargus is real-time strategy (RTS) game
Tactical battles are a key aspect of the game
RL Task: learn a policy to control n friendly agents in a battle against m enemy agents
Policy should be applicable to tasks with different sets and numbers of agents
5 vs. 5
10 vs. 10 50

57. Policy Gradient Ascent
Let () be the expected value of policy .
() is just the expected discounted total reward for a trajectory of .
For simplicity assume each trajectory starts at a single initial state.
Our objective is to find a  that maximizes ()
Policy gradient ascent tells us to iteratively update parameters via:
Problem: () is generally very complex and it is rare that we can compute a closed form for the gradient of ().
We will instead estimate the gradient based on experience

58. Gradient Estimation
Concern: Computing or estimating the gradient of discontinuous functions can be problematic.
For our example parametric policy
is () continuous?
No.
There are values of  where arbitrarily small changes, cause the policy to change.
Since different policies can have different values this means that changing  can cause discontinuous jump of ().

59. Example: Discontinous ()
Consider a problem with initial state s and two actions a1 and a2
a1 leads to a very large terminal reward R1
a2 leads to a very small terminal reward R2
Fixing 2 to a constant we can plot the ranking assigned to each action by Q and the corresponding value ()
Discontinuity in () when ordering of a1 and a2 change R1
() R2 1 1

60. Probabilistic Policies
We would like to avoid policies that drastically change with small parameter changes, leading to discontinuities
A probabilistic policy  takes a state as input and returns a distribution over actions
Given a state s (s,a) returns the probability that  selects action a in s
Note that () is still well defined for probabilistic policies
Now uncertainty of trajectories comes from environment and policy
Importantly if (s,a) is continuous relative to changing  then () is also continuous relative to changing 
A common form for probabilistic policies is the softmax function or Boltzmann exploration function

61. Empirical Gradient Estimation
Our first approach to estimating  () is to simply compute empirical gradient estimates
Recall that  = (1,…, n) and
so we can compute the gradient by empirically estimating each partial derivative
So for small  we can estimate the partial derivatives by
This requires estimating n+1 values:

62. Empirical Gradient Estimation
How do we estimate the quantities
For each set of parameters, simply execute the policy for N trials/episodes and average the values achieved across the trials
This requires a total of N(n+1) episodes to get gradient estimate
For stochastic environments and policies the value of N must be relatively large to get good estimates of the true value
Often we want to use a relatively large number of parameters
Often it is expensive to run episodes of the policy
So while this can work well in many situations, it is often not a practical approach computationally
Better approaches try to use the fact that the stochastic policy is differentiable.

64. Policy Gradient Recap Can be prone to finding local maxima
When policies have much simpler representations than the corresponding value functions, direct search in policy space can be a good idea
Allows us to design complex parametric controllers and optimize details of parameter settings
For baseline algorithm the gradient estimates are unbiased (i.e. they will converge to the right value) but have high variance
Can require a large N to get reliable estimates
OLPOMDP offers can trade-off bias and variance via the discount parameter [Baxter & Bartlett, 2000]
Can be prone to finding local maxima
Many ways of dealing with this, e.g. random restarts.

65. Gradient Estimation: Single Step Problems
For stochastic policies it is possible to estimate  () directly from trajectories of just the current policy 
Idea: take advantage of the fact that we know the functional form of the policy
First consider the simplified case where all trials have length 1
For simplicity assume each trajectory starts at a single initial state and reward only depends on action choice
() is just the expected reward of action selected by .
where s0 is the initial state and R(a) is reward of action a
The gradient of this becomes
How can we estimate this by just observing the execution of ?

66. Gradient Estimation: Single Step Problems
Rewriting
The gradient is just the expected value of g(s0,a)R(a) over execution trials of 
Can estimate by executing  for N trials and averaging samples
aj is action selected by policy on j’th episode
Only requires executing  for a number of trials that need not depend on the number of parameters
can get closed form g(s0,a)

67. Gradient Estimation: General Case
So for the case of a length 1 trajectories we got:
For the general case where trajectories have length greater than one and reward depends on state we can do some work and get:
sjt is t’th state of j’th episode, ajt is t’th action of epidode j
The derivation of this is straightforward but messy.
length of trajectory j
# of trajectories of current policy
Observed total reward in trajectory j from step t to end

68. How to interpret gradient expression?
So the overall gradient is a reward weighted combination of individual gradient directions
For large Rj(sj,t) will increase probability of aj,t in sj,t
For negative Rj(sj,t) will decrease probability of aj,t in sj,t
Intuitively this increases probability of taking actions that typically are followed by good reward sequences
Direction to move parameters in order to increase the probability that policy selects ajt in state sjt
Total reward observed after taking ajt in state sjt

69. Basic Policy Gradient Algorithm
Repeat until stopping condition
Execute  for N trajectories while storing the state, action, reward sequences
One disadvantage of this approach is the small number of updates per amount of experience
Also requires a notion of trajectory rather than an infinite sequence of experience
Online policy gradient algorithms perform updates after each step in environment (often learn faster)

70. Online Policy Gradient (OLPOMDP)
Repeat forever
Observe state s
Draw action a according to distribution (s)
Execute a and observe reward r
;; discounted sum of ;; gradient directions
Performs policy update at each time step and executes indefinitely
This is the OLPOMDP algorithm [Baxter & Bartlett, 2000]

71. Interpretation Repeat forever Observe state s
Draw action a according to distribution (s)
Execute a and observe reward r
;; discounted sum of ;; gradient directions
Step 4 computes an “eligibility trace” e
Discounted sum of gradients over previous state-action pairs
Points in direction of parameter space that increases probability of taking more recent actions in more recent states
For positive rewards step 5 will increase probability of recent actions and decrease for negative rewards.

72. Computing the Gradient of Policy
Both algorithms require computation of
For the Boltzmann distribution with linear approximation we have:
where
Here the partial derivatives needed for g(s,a) are:

73. Applications of Policy Gradient Search
Policy gradient techniques have been used to create controllers for difficult helicopter maneuvers
For example, inverted helicopter flight.
A planner called FPG also “won” the 2006 International Planning Competition
If you don’t count FF-Replan