1. Deep Reinforcement Learning
On the road to Skynet!
UW CSE Deep Learning – Felix Leeb

2. Overview Today Next time MDPs – formalizing decisions
Function Approximation
Value Function – DQN
Policy Gradients – REINFORCE, NPG
Actor Critic – A3C, DDPG
Model Based RL – forward/inverse
Planning – MCTS, MPPI
Imitation Learning – DAgger, GAIL
Advanced Topics – Exploration, MARL, Meta-learning, LMDPs…
UW CSE Deep Learning - Felix Leeb

3. Paradigm Objective Classification Regression Inference Generation
Supervised Learning
Unsupervised Learning
Reinforcement Learning
Paradigm
Objective
Classification
Regression
Inference
Generation
Prediction
Control
In reinforcement learning the goal is to learn a policy, which gives us the action given
Applications
UW CSE Deep Learning - Felix Leeb

4. 𝑥 𝑦 𝑥 𝑢 𝑦 Prediction Control
Prediction: finding the likely output given the input
Control is a little different, now we have to find a control given only the observation (and a reward signal)
So can we use any of the tricks we learned for prediction in control? 𝑦
UW CSE Deep Learning - Felix Leeb

5. Setting Environment Agent State/Observation Reward Action using policy
UW CSE Deep Learning - Felix Leeb

6. Markov Decision Processes
Transition function
Reward function
State space
Action space
Markov – only previous state matters
Decision – agent takes actions, and those decisions have consequences
Process – there is some transition function
Transition function is sometimes called the dynamics of the system
Reward function can in general depend on both the state and action, but often it’s only related to the state
Goal: maximize overall reward
UW CSE Deep Learning - Felix Leeb

7. Discount Factor Return: We want to be greedy but not impulsive
Implicitly takes uncertainty in dynamics into account
Mathematically: γ<1 allows infinite horizon returns
Return:
UW CSE Deep Learning - Felix Leeb

8. Solving an MDP
Objective:
Goal:
UW CSE Deep Learning - Felix Leeb

9. Value Functions Value = expected gain of a state
Q function – action specific value function
Advantage function – how much more valuable is an action
Value depends on future rewards  depends on policy
UW CSE Deep Learning - Felix Leeb

10. Tabular Solution: Policy Iteration
Policy Evaluation
Policy Update
UW CSE Deep Learning - Felix Leeb

11. Q Learning Without knowing the transition function
UW CSE Deep Learning - Felix Leeb

12. Function Approximation
Model:
Training data:
Allows continuous state spaces
Where’s the deep in “deep RL”
Whats with this other Q?
At the beginning of training our Q function will be really bad, so the updates will be bad, but each update is moving in the right direction, so overall we’re moving in the right direction
Take the derivative of loss wrt Q -> gives you q learning update -> shows the mse loss for params is equivalent to the tabular setting of updating the q values
Loss function:
where
UW CSE Deep Learning - Felix Leeb

13. Implementation Action-in Action-out Off-Policy Learning
The target depends in part on our model  old observations are still useful
Use a Replay Buffer of most recent transitions as dataset
Data: off policy data
UW CSE Deep Learning - Felix Leeb

14. Deep Q Networks (DQN) Mnih et al. (2015)
UW CSE Deep Learning - Felix Leeb

15. DQN Issues Convergence is not guaranteed – hope for deep magic!
Replay Buffer
Error Clipping
Using replicas
Reward scaling
Double Q Learning – decouple action selection and value estimation
Clipping errors
Scaling rewards
Use replay buffer – prioritizing recent actions
Double Q Learning – Using separate target and training Q networks
Sample complexity is not great – training deep CNN through RL
Continuous action spaces are essentially impossible
This is all really annoying
UW CSE Deep Learning - Felix Leeb

16. Policy Gradients
Parameterize policy and update those parameters directly
Enables new kinds of policies: stochastic, continuous action spaces
On policy learning  learn directly from your actions
Do we have to bother with a value function?
On policy learning – learn directly from actions
Any model that can be trained, could be a policy: Allows continuous action spaces, learning a stochastic policy
UW CSE Deep Learning - Felix Leeb

17. Policy Gradients Approximate expectation value from samples
Note: we’re going to be a little hand wavy with the notation
Essentially importance sampling
No guarantee of finding a global optimum
Approximate expectation value from samples
UW CSE Deep Learning - Felix Leeb

18. REINFORCE
Sutton et al. (2000)
UW CSE Deep Learning - Felix Leeb

19. Variance Reduction
Constant offsets make it harder to differentiate the right direction
Remove offset  a priori value of each state
Use baseline to reduce variance – the return of the value function
Turns out standard gradient descent is not necessarily the direction of steepest descent for stochastic function optimization – consider natural gradients
UW CSE Deep Learning - Felix Leeb

20. Advanced Policy Gradient Methods
For stochastic functions, the gradient is not the best direction
Consider the KL divergence
Approximating the Fisher information matrix
Computing gradients with KL constraint
Gradients with KL penalty
NPG 
TRPO 
PPO 
Natural Policy Gradients – use Fisher information matrix to choose gradient
TRPO – adjust gradient subject to KL divergence constraint
PPO – take a step directly related to KL divergence
UW CSE Deep Learning - Felix Leeb

21. Advanced Policy Gradient Methods
Natural Policy Gradients – use Fisher information matrix to choose gradient
TRPO – adjust gradient subject to KL divergence constraint
PPO – take a step directly related to KL divergence
Rajeswaran et al. (2017)
Heess et al. (2017)
UW CSE Deep Learning - Felix Leeb

22. Actor Critic Critic Actor Estimate Advantage Propose Actions
using Q learning update
Estimate Advantage
Propose Actions
Get the convergence of policy gradients, and the sample complexity of q learning
Actor
using policy gradient update
UW CSE Deep Learning - Felix Leeb

23. Async Advantage Actor-Critic (A3C)
Async – parallelizes updates
Uses advantage function
REINFORCE updates to policy
Mnih et al. (2016)
UW CSE Deep Learning - Felix Leeb

24. DDPG Off-policy learning – using deterministic policy gradients
Max Ferguson (2017)
Continuous control
Replay buffer
EMA between target and training networks for stability
Eps-greedy exploration
Batch normalization
UW CSE Deep Learning - Felix Leeb