1. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

2. Introduction Supervised Learning: Example Class Reinforcement Learning: Situation Reward …

3. Examples Playing chess: Reward comes at end of game Ping-pong: Reward on each point scored Animals: Hunger and pain - negative reward food intake – positive reward

4. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

5. Passive Learning We assume the policy Π is fixed. In state s we always execute action Π(s) Rewards are given.

7. Typical Trials (1,1) -0.04  (1,2) -0.04  (1,3) -0.04  (1,2) -0.04  (1,3) -0.04 …  (4,3) +1 Goal: Use rewards to learn the expected utility U Π (s)

8. Expected Utility U Π (s) = E [ Σ t=0 γ R(s t ) | Π, S 0 = s ] Expected sum of rewards when the policy is followed.

9. Example (1,1) -0.04  (1,2) -0.04  (1,3) -0.04  (2,3) -0.04  (3,3) -0.04  (4,3) +1 Total reward: (-0.04 x 5) + 1 = 0.80

10. Direct Utility Estimation Convert the problem to a supervised learning problem: (1,1)  U = 0.72 (2,1)  U = 0.68 … Learn to map states to utilities. But utilities are not independent of each other!

11. Bellman Equations Utility values obey the following equations: U Π (s) = R(s) + γ Σ s’ T(s,s’) U Π (s’) Can be solved using dynamic programming. Assumes knowledge of model.

12. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

13. Temporal Difference Learning Use the following update rule: U Π (s)  U Π (s) + α [ R(s) + γ U Π (s’) - U Π (s) ] α is the learning rate Temporal difference equation. No model assumption.

14. Example U(1,3) = 0.84 U(2,3) = 0.92 We hope to see that: U(1,3) = 0.84 + -0.04 + [U(2,3) – U(1,3)] U(1,3) = 0.84 + -0.04 + (0.92 – 0.84) The value is 0.88. Current value is a bit low and we must increase it.

17. Considerations Update values toward the equilibrium equation. Update includes the successor only. Over many trials the updates converge toward optimal values.

18. Other heuristics Prioritized Sweeping: Make adjustments to states where the most probable successors have undergone a large adjustment in terms of utility estimates.

19. Richard Sutton Author of classic textbook: “Reinforcement Learning” by Sutton and Barto, MIT Press, 1998. Dept. of Computer Science University of Alberta

20. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

21. Active Reinforcement Learning Now we must decide what actions to take. Optimal policy: Choose action with highest utility value. Is that the right thing to do?

22. Active Reinforcement Learning No! Sometimes we may get stuck in suboptimal solutions. Exploration vs Exploitation Tradeoff Why is this important? The learned model is not the same as the true environment.

23. Explore vs Exploit Exploitation: Maximize its reward vs Exploration: Maximize long-term well being.

24. Bandit Problem An n-armed bandit has n levers. Which lever to play to maximize reward? In genetic algorithms the selection strategy is to allocate coins optimally given appropriate set of assumptions.

25. Solution U + (s)  R(s) + γ max a f(u,N(a,s)) U + (s) : optimistic estimate of utility N(a,s): number of times action a has been tried. f(u,n): exploration function. Increasing in u (exploitation) Decreasing in n (exploration)

26. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

27. Applications Game Playing Checker playing program by Arthur Samuel (IBM) Update rules: change weights by difference between current states and backed-up value generating full look-ahead tree

28. Applications Robot Control Cart-pole balancing problem. Control the position of x so that the pole stays roughly upright.

31. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

32. Goal is to learn utility values and an optimal mapping from states to actions. Direct Utility Estimation ignores dependencies among states. We must follow Bellman Equations. Temporal difference updates values to match those of successor states. Active reinforcement learning learns optimal mapping from states to actions.

33. Video http://www.youtube.com/watch?v=YQIMGV5vtd4