1. Chapter 8: Generalization and Function Approximation pLook at how experience with a limited part of the state set be used to produce good behavior over a much larger part. pOverview of function approximation (FA) methods and how they can be adapted to RL Objectives of this chapter:

2. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 2 Any FA Method? pIn principle, yes: artificial neural networks decision trees multivariate regression methods etc. pBut RL has some special requirements: usually want to learn while interacting ability to handle nonstationarity other?

3. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 3 Gradient Descent Methods transpose

4. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 4 Performance Measures pMany are applicable but… pa common and simple one is the mean-squared error (MSE) over a distribution P : pWhy P ? pWhy minimize MSE? pLet us assume that P is always the distribution of states at which backups are done. pThe on-policy distribution: the distribution created while following the policy being evaluated. Stronger results are available for this distribution.

5. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 5 Gradient Descent Iteratively move down the gradient:

6. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 6 On-Line Gradient-Descent TD( )

7. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 7 Linear Methods

8. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 8 Nice Properties of Linear FA Methods pThe gradient is very simple: pFor MSE, the error surface is simple: quadratic surface with a single minumum.  Linear gradient descent TD( ) converges: Step size decreases appropriately On-line sampling (states sampled from the on-policy distribution) Converges to parameter vector with property: best parameter vector (Tsitsiklis & Van Roy, 1997)

9. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 9 Coarse Coding

10. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 10 Shaping Generalization in Coarse Coding

11. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 11 Learning and Coarse Coding

12. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 12 Tile Coding pBinary feature for each tile pNumber of features present at any one time is constant pBinary features means weighted sum easy to compute pEasy to compute indices of the freatures present

13. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 13 Tile Coding Cont. Irregular tilings Hashing CMAC “Cerebellar model arithmetic computer” Albus 1971

14. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 14 Radial Basis Functions (RBFs) e.g., Gaussians

15. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 15 Can you beat the “curse of dimensionality”? pCan you keep the number of features from going up exponentially with the dimension? pFunction complexity, not dimensionality, is the problem. pKanerva coding: Select a bunch of binary prototypes Use hamming distance as distance measure Dimensionality is no longer a problem, only complexity p“Lazy learning” schemes: Remember all the data To get new value, find nearest neighbors and interpolate e.g., locally-weighted regression

16. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 16 pLearning state-action values pThe general gradient-descent rule:  Gradient-descent Sarsa( ) (backward view): Control with FA Training examples of the form:

17. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 17 GPI Linear Gradient Descent Watkins’ Q( )

18. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 18 GPI with Linear Gradient Descent Sarsa( )

19. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 19 Mountain-Car Task

20. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 20 Mountain Car with Radial Basis Functions

21. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 21 Mountain-Car Results

22. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 22 Baird’s Counterexample

23. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 23 Baird’s Counterexample Cont.

24. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 24 Should We Bootstrap?

25. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 25 Summary pGeneralization pAdapting supervised-learning function approximation methods pGradient-descent methods pLinear gradient-descent methods Radial basis functions Tile coding Kanerva coding pNonlinear gradient-descent methods? Backpropation? pSubtleties involving function approximation, bootstrapping and the on-policy/off-policy distinction

26. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 26 Value Prediction with FA As usual: Policy Evaluation (the prediction problem): for a given policy , compute the state-value function In earlier chapters, value functions were stored in lookup tables.

27. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 27 Adapt Supervised Learning Algorithms Supervised Learning System Inputs Outputs Training Info = desired (target) outputs Error = (target output – actual output) Training example = {input, target output}

28. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 28 Backups as Training Examples As a training example: inputtarget output

29. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 29 Gradient Descent Cont. For the MSE given above and using the chain rule:

30. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 30 Gradient Descent Cont. Use just the sample gradient instead: Since each sample gradient is an unbiased estimate of the true gradient, this converges to a local minimum of the MSE if  decreases appropriately with t.

31. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 31 But We Don’t have these Targets

32. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 32 What about TD( ) Targets?