1. An Introduction to COMPUTATIONAL REINFORCEMENT LEARING Andrew G. Barto Department of Computer Science University of Massachusetts – Amherst Lecture 3 Autonomous Learning Laboratory – Department of Computer Science

2. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 2 The Overall Plan pLecture 1: What is Computational Reinforcement Learning? Learning from evaluative feedback Markov decision processes pLecture 2: Dynamic Programming Basic Monte Carlo methods Temporal Difference methods A unified perspective Connections to neuroscience pLecture 3: Function approximation Model-based methods Dimensions of Reinforcement Learning

3. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 3 The Overall Plan pLecture 1: What is Computational Reinforcement Learning? Learning from evaluative feedback Markov decision processes pLecture 2: Dynamic Programming Basic Monte Carlo methods Temporal Difference methods A unified perspective Connections to neuroscience pLecture 3: Function approximation Model-based methods Dimensions of Reinforcement Learning

4. Lecture 3, Part 1: Generalization and Function Approximation pLook at how experience with a limited part of the state set be used to produce good behavior over a much larger part pOverview of function approximation (FA) methods and how they can be adapted to RL Objectives of this part:

5. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 5 Value Prediction with FA As usual: Policy Evaluation (the prediction problem): for a given policy , compute the state-value function In earlier chapters, value functions were stored in lookup tables.

6. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 6 Adapt Supervised Learning Algorithms Supervised Learning System Inputs Outputs Training Info = desired (target) outputs Error = (target output – actual output) Training example = {input, target output}

7. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 7 Backups as Training Examples As a training example: inputtarget output

8. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 8 Any FA Method? pIn principle, yes: artificial neural networks decision trees multivariate regression methods etc. pBut RL has some special requirements: usually want to learn while interacting ability to handle nonstationarity other?

9. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 9 Gradient Descent Methods transpose

10. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 10 Performance Measures pMany are applicable but… pa common and simple one is the mean-squared error (MSE) over a distribution P : pWhy P ? pWhy minimize MSE? pLet us assume that P is always the distribution of states with which backups are done. pThe on-policy distribution: the distribution created while following the policy being evaluated. Stronger results are available for this distribution.

11. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 11 Gradient Descent Iteratively move down the gradient:

12. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 12 Gradient Descent Cont. For the MSE given above and using the chain rule:

13. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 13 Gradient Descent Cont. Use just the sample gradient instead: Since each sample gradient is an unbiased estimate of the true gradient, this converges to a local minimum of the MSE if  decreases appropriately with t.

14. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 14 But We Don’t have these Targets

15. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 15 What about TD( ) Targets?

16. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 16 On-Line Gradient-Descent TD( )

17. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 17 Linear Methods

18. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 18 Nice Properties of Linear FA Methods pThe gradient is very simple: pFor MSE, the error surface is simple: quadratic surface with a single minumum.  Linear gradient descent TD( ) converges: Step size decreases appropriately On-line sampling (states sampled from the on-policy distribution) Converges to parameter vector with property: best parameter vector (Tsitsiklis & Van Roy, 1997)

19. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 19 Coarse Coding

20. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 20 Learning and Coarse Coding

21. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 21 Tile Coding pBinary feature for each tile pNumber of features present at any one time is constant pBinary features means weighted sum easy to compute pEasy to compute indices of the freatures present

22. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 22 Tile Coding Cont. Irregular tilings Hashing CMAC “Cerebellar Model Arithmetic Computer” Albus 1971

23. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 23 Radial Basis Functions (RBFs) e.g., Gaussians

24. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 24 Can you beat the “curse of dimensionality”? pCan you keep the number of features from going up exponentially with the dimension? pFunction complexity, not dimensionality, is the problem. pKanerva coding: Select a bunch of binary prototypes Use hamming distance as distance measure Dimensionality is no longer a problem, only complexity p“Lazy learning” schemes: Remember all the data To get new value, find nearest neighbors and interpolate e.g., locally-weighted regression

25. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 25 Control with FA Training examples of the form: pLearning state-action values pThe general gradient-descent rule:  Gradient-descent Sarsa( ) (backward view):

26. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 26 Linear Gradient Descent Sarsa( )

27. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 27 GPI Linear Gradient Descent Watkins’ Q( )

28. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 28 Mountain-Car Task

29. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 29 Mountain-Car Results

30. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 30 Baird’s Counterexample

31. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 31 Baird’s Counterexample Cont.

32. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 32 Should We Bootstrap?

33. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 33 Summary pGeneralization pAdapting supervised-learning function approximation methods pGradient-descent methods pLinear gradient-descent methods Radial basis functions Tile coding Kanerva coding pNonlinear gradient-descent methods? Backpropation? pSubleties involving function approximation, bootstrapping and the on-policy/off-policy distinction

34. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 34 The Overall Plan pLecture 1: What is Computational Reinforcement Learning? Learning from evaluative feedback Markov decision processes pLecture 2: Dynamic Programming Basic Monte Carlo methods Temporal Difference methods A unified perspective Connections to neuroscience pLecture 3: Function approximation Model-based methods Dimensions of Reinforcement Learning

35. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 35 Lecture 3, Part 2: Model-Based Methods pUse of environment models pIntegration of planning and learning methods Objectives of this part:

36. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 36 Models pModel: anything the agent can use to predict how the environment will respond to its actions pDistribution model: description of all possibilities and their probabilities e.g., pSample model: produces sample experiences e.g., a simulation model pBoth types of models can be used to produce simulated experience pOften sample models are much easier to come by

37. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 37 Planning pPlanning: any computational process that uses a model to create or improve a policy pPlanning in AI: state-space planning plan-space planning (e.g., partial-order planner) pWe take the following (unusual) view: all state-space planning methods involve computing value functions, either explicitly or implicitly they all apply backups to simulated experience

38. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 38 Planning Cont. Random-Sample One-Step Tabular Q-Planning pClassical DP methods are state-space planning methods pHeuristic search methods are state-space planning methods pA planning method based on Q-learning:

39. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 39 Learning, Planning, and Acting pTwo uses of real experience: model learning: to improve the model direct RL: to directly improve the value function and policy pImproving value function and/or policy via a model is sometimes called indirect RL or model-based RL. Here, we call it planning.

40. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 40 Direct vs. Indirect RL pIndirect (model-based) methods: make fuller use of experience: get better policy with fewer environment interactions pDirect methods simpler not affected by bad models But they are very closely related and can be usefully combined: planning, acting, model learning, and direct RL can occur simultaneously and in parallel

41. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 41 The Dyna Architecture (Sutton 1990)

42. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 42 The Dyna-Q Algorithm direct RL model learning planning

43. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 43 Dyna-Q on a Simple Maze rewards = 0 until goal, when =1

44. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 44 Dyna-Q Snapshots: Midway in 2nd Episode

45. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 45 When the Model is Wrong: Blocking Maze The changed envirnoment is harder

46. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 46 Shortcut Maze The changed environment is easier

47. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 47 What is Dyna-Q ? pUses an “exploration bonus”: Keeps track of time since each state-action pair was tried for real An extra reward is added for transitions caused by state-action pairs related to how long ago they were tried: the longer unvisited, the more reward for visiting The agent actually “plans” how to visit long unvisited states +

48. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 48 Prioritized Sweeping pWhich states or state-action pairs should be generated during planning? pWork backwards from states whose values have just changed: Maintain a queue of state-action pairs whose values would change a lot if backed up, prioritized by the size of the change When a new backup occurs, insert predecessors according to their priorities Always perform backups from first in queue pMoore and Atkeson 1993; Peng and Williams, 1993

49. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 49 Prioritized Sweeping

50. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 50 Prioritized Sweeping vs. Dyna-Q Both use N=5 backups per environmental interaction

51. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 51 Rod Maneuvering (Moore and Atkeson 1993)

52. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 52 Full and Sample (One-Step) Backups

53. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 53 Full vs. Sample Backups b successor states, equally likely; initial error = 1; assume all next states’ values are correct

54. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 54 Trajectory Sampling pTrajectory sampling: perform backups along simulated trajectories pThis samples from the on-policy distribution pAdvantages when function approximation is used pFocusing of computation: can cause vast uninteresting parts of the state space to be (usefully) ignored: Initial states Reachable under optimal control Irrelevant states

55. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 55 Trajectory Sampling Experiment pone-step full tabular backups puniform: cycled through all state- action pairs pon-policy: backed up along simulated trajectories p200 randomly generated undiscounted episodic tasks p2 actions for each state, each with b equally likely next states p.1 prob of transition to terminal state pexpected reward on each transition selected from mean 0 variance 1 Gaussian

56. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 56 Heuristic Search pUsed for action selection, not for changing a value function (=heuristic evaluation function) pBacked-up values are computed, but typically discarded pExtension of the idea of a greedy policy — only deeper pAlso suggests ways to select states to backup: smart focusing:

57. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 57 Summary pEmphasized close relationship between planning and learning pImportant distinction between distribution models and sample models pLooked at some ways to integrate planning and learning synergy among planning, acting, model learning pDistribution of backups: focus of the computation trajectory sampling: backup along trajectories prioritized sweeping heuristic search pSize of backups: full vs. sample; deep vs. shallow

58. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 58 The Overall Plan pLecture 1: What is Computational Reinforcement Learning? Learning from evaluative feedback Markov decision processes pLecture 2: Dynamic Programming Basic Monte Carlo methods Temporal Difference methods A unified perspective Connections to neuroscience pLecture 3: Function approximation Model-based methods Dimensions of Reinforcement Learning

59. Lecture 3, part 3: Dimensions of Reinforcement Learning pReview the treatment of RL taken in this course pWhat have left out? pWhat are the hot research areas? Objectives of this part:

60. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 60 Three Common Ideas pEstimation of value functions pBacking up values along real or simulated trajectories pGeneralized Policy Iteration: maintain an approximate optimal value function and approximate optimal policy, use each to improve the other

61. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 61 Backup Dimensions

62. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 62 Other Dimensions pFunction approximation tables aggregation other linear methods many nonlinear methods pOn-policy/Off-policy On-policy: learn the value function of the policy being followed Off-policy: try learn the value function for the best policy, irrespective of what policy is being followed

63. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 63 Still More Dimensions pDefinition of return: episodic, continuing, discounted, etc. pAction values vs. state values vs. afterstate values  Action selection/exploration:  -greed, softmax, more sophisticated methods pSynchronous vs. asynchronous pReplacing vs. accumulating traces pReal vs. simulated experience pLocation of backups (search control) pTiming of backups: part of selecting actions or only afterward? pMemory for backups: how long should backed up values be retained?

64. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 64 Frontier Dimensions pProve convergence for bootstrapping control methods. pTrajectory sampling pNon-Markov case: Partially Observable MDPs (POMDPs) –Bayesian approach: belief states –construct state from sequence of observations Try to do the best you can with non-Markov states pModularity and hierarchies Learning and planning at several different levels –Theory of options

65. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 65 More Frontier Dimensions pUsing more structure factored state spaces: dynamic Bayes nets factored action spaces

66. A. G. Barto, Barcelona Lectures, April 2006. Based on R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction, MIT Press, 1998 66 Still More Frontier Dimensions pIncorporating prior knowledge advice and hints trainers and teachers shaping Lyapunov functions etc.