1. Reinforcement Learning
Hien Van Nguyen
University of Houston
2/4/2019
Slides adopted from [1]
[2]

2. Deep Q-learning Deep Q-learning:
You don’t know the transitions T(s,a,s’)
You don’t know the rewards R(s,a,s’)
You choose the actions now
State space is large
Goal: learn the optimal policy / values
Idea: Represent Q-function by a deep network:
2/4/2019
Machine Learning

3. Deep Q-learning Represent Q-function by a deep network
Define objective function by mean-squared error in Q-values:
Take derivative:
Target
Train end-to-end via SGD
Can use raw data to represent state
2/4/2019
Machine Learning

4. Policy gradient for continuous actions
Challenge: Action space can be continuous and maximization of Q-function over action space is difficult.
2/4/2019
Machine Learning

5. Deterministic policy gradient
2/4/2019
Machine Learning

6. Deterministic actor-critic
2/4/2019
Machine Learning

7. Deterministic actor-critic learning rule
2/4/2019
Machine Learning

8. Stability issue with Deep RL
2/4/2019
Machine Learning

9. Strategies for improving stability
2/4/2019
Machine Learning

10. Experience replay
2/4/2019
Machine Learning

11. Fixed target Q-network
2/4/2019
Machine Learning

12. How much does DQN help?
2/4/2019
Machine Learning

13. Thank you for taking my class!
2/4/2019
Machine Learning