1. CHAPTER 10 Reinforcement Learning Utility Theory

2. QUESTION????

3. Recap: MDPs

4. Reinforcement Learning

5. Example: Animal Learning RL studied experimentally for more than 60 years in psychology Rewards: food, pain, hunger, drugs, etc. Mechanisms and sophistication debated Example: foraging Bees learn near-optimal foraging plan in field of artificial flowers with controlled nectar supplies Bees have a direct neural connection from nectar intake measurement to motor planning area

6. Example: Backgammon

7. Passive Learning

8. Example: Direct Estimation

9. Model-Based Learning Idea: Learn the model empirically (rather than values) Solve the MDP as if the learned model were correct Empirical model learning Simplest case: Count outcomes for each s,a Normalize to give estimate of T(s,a,s’) Discover R(s) the first time we enter s More complex learners are possible (e.g. if we know that all squares have related action outcomes “stationary noise”)

10. Example: Model-Based Learning

11. Model-Free Learning

12. (Greedy) Active Learning In general, want to learn the optimal policy Idea: –Learn an initial model of the environment: –Solve for the optimal policy for this model (value or policy iteration) Refine model through experience and repeat

13. Example: Greedy Active Learning

14. Q-Functions

15. Learning Q-Functions: MDPs

16. Q-Learning

17. Exploration / Exploitation

18. Exploration Functions

19. Function Approximation Problem: too slow to learn each state’s utility one by one Solution: what we learn about one state should generalize to similar states –Very much like supervised learning –If states are treated entirely independently, we can only learn on very small state spaces

20. Discretization

21. Linear Value Functions

22. TD Updates for Linear Values