1. CS 188: Artificial Intelligence Spring 2007 Lecture 21:Reinforcement Learning: II MDP 4/12/2007 Srini Narayanan – ICSI and UC Berkeley

2. Announcements  Othello tournament signup  Please send email to cs188@imail.berkeley.edu cs188@imail.berkeley.edu  HW on classification out  Due 4/23  Can work in pairs

3. Types of Learning: Recap  Past (learning to classify the world)  Supervised  Naïve Bayes, Perceptron  Unsupervised  Clustering k-means  Next (learning to act in the world)  Reinforcement Learning

4. Reinforcement Learning  Basic idea:  Receive feedback in the form of rewards  Agent’s utility is defined by the reward function  Must learn to act so as to maximize expected utility  Change the rewards, change the behavior  Examples:  Learning optimal paths  Playing a game, reward at the end for winning / losing  Vacuuming a house, reward for each piece of dirt picked up  Automated taxi, reward for each passenger delivered

5. Recap: MDPs  Markov decision processes:  States S  Actions A  Transitions P(s’|s,a) (or T(s,a,s’))  Rewards R(s,a,s’)  Start state s 0  Examples:  Gridworld, High-Low, N-Armed Bandit  Any process where the result of your action is stochastic  Goal: find the “best” policy   Policies are maps from states to actions  What do we mean by “best”?  This is like search – it’s planning using a model, not actually interacting with the environment

6. MDP Solutions  In deterministic single-agent search, want an optimal sequence of actions from start to a goal  In an MDP, like expectimax, want an optimal policy  (s)  A policy gives an action for each state  Optimal policy maximizes expected utility (i.e. expected rewards) if followed  Defines a reflex agent Optimal policy when R(s, a, s’) = -0.04 for all non-terminals s

7. Example Optimal Policies R(s) = -2.0 R(s) = -0.4 R(s) = -0.03R(s) = -0.01

8. Stationarity  In order to formalize optimality of a policy, need to understand utilities of reward sequences  Typically consider stationary preferences:  Theorem: only two ways to define stationary utilities  Additive utility:  Discounted utility:

9. Infinite Utilities?!  Problem: infinite state sequences with infinite rewards  Solutions:  Finite horizon:  Terminate after a fixed T steps  Gives nonstationary policy (  depends on time left)  Absorbing state(s): guarantee that for every policy, agent will eventually “die” (like “done” for High-Low)  Discounting: for 0 <  < 1  Smaller  means smaller horizon

10. How (Not) to Solve an MDP  The inefficient way:  Enumerate policies  For each one, calculate the expected utility (discounted rewards) from the start state  E.g. by simulating a bunch of runs  Choose the best policy  We’ll return to a (better) idea like this later

11. Utility of a State  Define the utility of a state under a policy: V  (s) = expected total (discounted) rewards starting in s and following   Recursive definition (one-step look-ahead):

12. Policy Evaluation  Idea one: turn recursive equations into updates  Idea two: it’s just a linear system, solve with Matlab (or Mosek, or Cplex)

13. Example: High-Low  Policy: always say “high”  Iterative updates:

14. Optimal Utilities  Goal: calculate the optimal utility of each state V*(s) = expected (discounted) rewards with optimal actions  Why: Given optimal utilities, MEU tells us the optimal policy

15. Bellman’s Equation for Selecting actions  Definition of utility leads to a simple relationship amongst optimal utility values: Optimal rewards = maximize over first action and then follow optimal policy Formally: Bellman’s Equation That’s my equation!

16. Practice: Computing Actions  Which action should we chose from state s:  Given optimal q-values Q?  Given optimal values V?

17. Example: GridWorld

18. Value Iteration  Idea:  Start with bad guesses at all utility values (e.g. V 0 (s) = 0)  Update all values simultaneously using the Bellman equation (called a value update or Bellman update):  Repeat until convergence  Theorem: will converge to unique optimal values  Basic idea: bad guesses get refined towards optimal values  Policy may converge long before values do

19. Example: Bellman Updates

20. Example: Value Iteration  Information propagates outward from terminal states and eventually all states have correct value estimates [DEMO]

21. Convergence*  Define the max-norm:  Theorem: For any two approximations U and V (any two utility vectors)  I.e. any distinct approximations must get closer to each other (after the Bellman update), so, in particular, any approximation must get closer to the true U (Bellman update is U) and value iteration converges to a unique, stable, optimal solution  Theorem:  I.e. once the change in our approximation is small, it must also be close to correct

22. Policy Iteration  Alternate approach:  Policy evaluation: calculate utilities for a fixed policy until convergence (remember the beginning of lecture)  Policy improvement: update policy based on resulting converged utilities  Repeat until policy converges  This is policy iteration  Can converge faster under some conditions

23. Policy Iteration  If we have a fixed policy , use simplified Bellman equation to calculate utilities:  For fixed utilities, easy to find the best action according to one-step look-ahead

24. Comparison  In value iteration:  Every pass (or “backup”) updates both utilities (explicitly, based on current utilities) and policy (possibly implicitly, based on current policy)  In policy iteration:  Several passes to update utilities with frozen policy  Occasional passes to update policies  Hybrid approaches (asynchronous policy iteration):  Any sequences of partial updates to either policy entries or utilities will converge if every state is visited infinitely often

25. Reinforcement Learning  Reinforcement learning:  Still have an MDP:  A set of states s  S  A model T(s,a,s’)  A reward function R(s)  Still looking for a policy  (s)  New twist: don’t know T or R  I.e. don’t know which states are good or what the actions do  Must actually try actions and states out to learn

26. Example: Animal Learning  RL studied experimentally for more than 60 years in psychology  Rewards: food, pain, hunger, drugs, etc.  Mechanisms and sophistication debated  Example: foraging  Bees learn near-optimal foraging plan in field of artificial flowers with controlled nectar supplies  Bees have a direct neural connection from nectar intake measurement to motor planning area

27. Example: Backgammon  Reward only for win / loss in terminal states, zero otherwise  TD-Gammon learns a function approximation to U(s) using a neural network  Combined with depth 3 search, one of the top 3 players in the world

28. Passive Learning  Simplified task  You don’t know the transitions T(s,a,s’)  You don’t know the rewards R(s,a,s’)  You are given a policy  (s)  Goal: learn the state values (and maybe the model)  In this case:  No choice about what actions to take  Just execute the policy and learn from experience  We’ll get to the general case soon

29. Example: Direct Estimation  Episodes: x y (1,1) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) up -1 (3,3) right +100 (done) (1,1) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) right -1 (4,2) right -100 (done) U(1,1) ~ (93 + -105) / 2 = -6 U(3,3) ~ (100 + 98 + -101) / 3 = 32.3  = 1, R = -1 +100 -100

30. Model-Based Learning  Idea:  Learn the model empirically (rather than values)  Solve the MDP as if the learned model were correct  Empirical model learning  Simplest case:  Count outcomes for each s,a  Normalize to give estimate of T(s,a,s’)  Discover R(s,a,s’) the first time we experience (s,a,s’)  More complex learners are possible (e.g. if we know that all squares have related action outcomes, e.g. “stationary noise”)

31. Example: Model-Based Learning  Episodes: x y T(, right, ) = 1 / 3 T(, right, ) = 2 / 2 +100 -100  = 1 (1,1) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) up -1 (3,3) right +100 (done) (1,1) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) right -1 (4,2) right -100 (done)

32. Model-Based Learning  In general, want to learn the optimal policy, not evaluate a fixed policy  Idea: adaptive dynamic programming  Learn an initial model of the environment:  Solve for the optimal policy for this model (value or policy iteration)  Refine model through experience and repeat  Crucial: we have to make sure we actually learn about all of the model