1. Reinforcement Learning (RL) Consider an “agent” embedded in an environmentConsider an “agent” embedded in an environment Task of the agentTask of the agent Repeat forever: 1)sense world 2)reason 3)choose an action to perform

2. Definition of RL Assume the world (ie, environment) periodically provides rewards or punishments (“reinforcements”)Assume the world (ie, environment) periodically provides rewards or punishments (“reinforcements”) Based on reinforcements received, learn how to better choose actionsBased on reinforcements received, learn how to better choose actions

3. Sequential Decision Problems Courtesy of A.G. Barto, April 2000 Decisions are made in stagesDecisions are made in stages The outcome of each decision is not fully predictable but can be observed before the next decision is madeThe outcome of each decision is not fully predictable but can be observed before the next decision is made The objective is to maximize a numerical measure of total reward (or equivalently, to minimize a measure of total cost)The objective is to maximize a numerical measure of total reward (or equivalently, to minimize a measure of total cost) Decisions cannot be viewed in isolation:Decisions cannot be viewed in isolation: need to balance desire for immediate reward with possibility of high future reward

4. Reinforcement Learning vs Supervised Learning How would we use SL to train an agent in an environment?How would we use SL to train an agent in an environment? Show action to choose in sample of world states – “I/O pairs”Show action to choose in sample of world states – “I/O pairs” RL requires much less of teacherRL requires much less of teacher Must set up “reward structure”Must set up “reward structure” Learner “works out the details” – i.e. writes a program to maximize rewards receivedLearner “works out the details” – i.e. writes a program to maximize rewards received

5. Embedded Learning Systems: Formalization S E = the set of states of the worldS E = the set of states of the world e.g., an N -dimensional vectore.g., an N -dimensional vector “sensors”“sensors” A E = the set of possible actions an agent can performA E = the set of possible actions an agent can perform “effectors”“effectors” W = the worldW = the world R = the immediate reward structureR = the immediate reward structure W and R are the environment, can be probabilistic functions

6. Embedded learning Systems (formalization) W: S E x A E  S E The world maps a state and an action and produces a new state R: S E x A E  “reals” Provides rewards (a number) as a function of state and action (as in textbook). Can equivalently formalize as a function of state (next state) alone.

7. A Graphical View of RL Note that both the world and the agent can be probabilistic, so W and R could produce probability distributions.Note that both the world and the agent can be probabilistic, so W and R could produce probability distributions. For now, assume deterministic problemsFor now, assume deterministic problems The real world, W The Agent an action sensory info R, reward (a scalar) - indirect teacher

8. Common Confusion State need not be solely the current sensor readings Markov AssumptionMarkov Assumption Value of state is independent of path taken to reach that state Can have memory of the pastCan have memory of the past Can always create Markovian task by remembering entire past history

9. Need for Memory: Simple Example “out of sight, but not out of mind” T=1 learning agent W A L L opponent T=2 learning agent W A L L opponent Seems reasonable to remember opponent recently seen

10. State vs. Current Sensor Readings Remember state is what is in one’s head (past memories, etc) not ONLY what one currently sees/hears/smells/etc

11. Policies The agent needs to learn a policy  E : S E  A E Given a world state, S E, which action, A E, should be chosen? The policy,  E, function Remember: The agent’s task is to maximize the total reward received during its lifetime

12. Policies (cont.) To construct  E, we will assign a utility (U) (a number) to each state. -  is a positive constant < 1 -R(s,  E, t) is the reward received at time t, assuming the agent follows policy  E and starts in state s at t=0 -Note: future rewards are discounted by  t-1

13. The Action-Value Function We want to choose the “best” action in the current state So, pick the one that leads to the best next state (and include any immediate reward) Let immediate reward received for going to state W(s,a) Future reward from further actions (discounted due to 1- step delay)

14. The Action-Value Function (cont. ) If we can accurately learn Q (the action- value function), choosing actions is easy Choose a, where

15. Q vs. U Visually U(1) U(3) U(2) U(4) U(5) U(6) state Q(1,i) Q(1,ii) Q(1,iii) state action Key states actions U’s “stored” on states Q’s “stored” on arcs

16. Q-Learning (Watkins PhD, 1989) Let Q t be our current estimate of the correct Q Our current policy is such that Our current utility-function estimate is - hence, the U table is embedded in the Q table and we don’t need to store both

17. Q-Learning (cont.) Assume we are in state S t “Run the program” (1) for awhile (n steps) Determine actual reward and compare to predicted reward Adjust prediction to reduce error (1 ) I.e., follow the current policy

18. How Many Actions Should We Take Before Updating Q ? Why not do so after each action? “1 – Step Q learning”“1 – Step Q learning” Most common approachMost common approach

19. Exploration vs. Exploitation In order to learn about better alternatives, we can’t always follow the current policy (“exploitation”) Sometimes, need to try “random” moves (“exploration”)

20. Exploration vs. Exploitation (cont) Approaches 1) p percent of the time, make a random move; could let 2) Prob(picking action A in state S ) A in state S ) Exponentia- ting gets rid of negative values

21. 0.S  initial state 1. If random #  P then a = random choice Else a =  t (S) 2. S new  W(S, a) R immed  R(S new ) 3.Q(S, a)  R immed +  max a’ Q(S new, a’) 4. S  S new Go to 1Go to 1 One-Step Q-Learning Algo Act on world and get reward

22. 3.Q(S, a)  R immed +  max a’ Q(S new, a’) To: Q(S, a)  α [R immed +  max a’ Q(S new, a’)] + (1-α) Q(S, a) (1-α) Q(S, a) In Stochastic World, Don’t Trash Current Q Entirely… Change Line 3:

23. A Simple Example (of Q-learning - with updates after each step, ie N =1) Repeat (deterministic world, so α =1) Algo: Pick State +Action S 0 R = 0 S 1 R = 1 S 3 R = 0 S 2 R = -1 S 4 R = 3 Let  = 2/3 Q = 0

24. A Simple Example (Step 1) S 0  S 2 Repeat (deterministic world, so α =1) Algo: Pick State +Action S 0 R = 0 S 1 R = 1 S 3 R = 0 S 2 R = -1 S 4 R = 3 Let  = 2/3 Q = 0 Q = -1

25. A Simple Example (Step 2) S 2  S 4 Repeat (deterministic world, so α =1) Algo: Pick State +Action S 0 R = 0 S 1 R = 1 S 3 R = 0 S 2 R = -1 S 4 R = 3 Let  = 2/3 Q = 0 Q = 3 Q = 0 Q = -1

26. A Simple Example (Step  ) Repeat (deterministic world, so α =1) Algo: Pick State +Action S 0 R = 0 S 1 R = 1 S 3 R = 0 S 2 R = -1 S 4 R = 3 Let  = 2/3 Q = 1 Q = 3 Q = 0 Q = 1

27. Q-Learning: Implementation Details Remember, conceptually we are filling in a huge table S0 S1 S2... Sn abc...zabc...z... Q(S2, c)...... Tables are a very verbose representation of a function States Actions Actions

28. Q-Learning: Convergence Proof Applies to Q tables and deterministic, Markovian worlds. Initialize Q’s 0 or random finite.Applies to Q tables and deterministic, Markovian worlds. Initialize Q’s 0 or random finite. Theorem: if every state-action pair visited infinitely often, 0≤  <1, and |rewards| ≤ C (some constant), thenTheorem: if every state-action pair visited infinitely often, 0≤  <1, and |rewards| ≤ C (some constant), then  s, a the approx. Q table (Q) the true Q table (Q) ^

29. Q-Learning Convergence Proof (cont.) Consider the max error in the approx. Q-table at step t :Consider the max error in the approx. Q-table at step t : The max is finite since |r| ≤ C, so max | | +C+CγThe max is finite since |r| ≤ C, so max | | +C+Cγ Since finite, we have finite, i.e. initial max error is finiteSince finite, we have finite, i.e. initial max error is finite

30. Q-Learning Convergence Proof (cont.) Let s’ be the state that results from doing action a in state s. Consider what happens when we visit s and do a at step t + 1: Current state Next state By Q-learning rule (one step) By def’n of Q (notice best a in s’ might be different)

31. Q-Learning Convergence Proof (cont.) By algebra Since max of all differences is as big as a particular one Max at s’ ≤ max at any s Plugging in defn of Δ t =  | max a’ Q t (s’, a’) – max a’’ Q(s’, a’’) | =  Δ t ≤  max a’’’ | Q t (s’, a’’’) – Q(s’, a’’’) | ≤  max s’’,a’’’ | Q t (s’’, a’’’) – Q(s’’, a’’’) | ^ ^ ^

32. Q-Learning Convergence Proof (cont.) Hence, every time, after t, we visit an, its Q value differs from the correct answer by no more than  Δ tHence, every time, after t, we visit an, its Q value differs from the correct answer by no more than  Δ t Let T o =t o (i.e. the start) and T N be the first time since T N-1 where every visited at least onceLet T o =t o (i.e. the start) and T N be the first time since T N-1 where every visited at least once Call the time between T N-1 and T N, a complete intervalCall the time between T N-1 and T N, a complete interval Clearly Δ T N Δ T N-1 Clearly Δ T N ≤  Δ T N-1

33. Q-Learning Convergence Proof (concluded) That is, every complete interval, Δ t is reduced by at least That is, every complete interval, Δ t is reduced by at least  Since we assumed every pair visited infinitely often, we will have an infinite number of complete intervalsSince we assumed every pair visited infinitely often, we will have an infinite number of complete intervals Hence, lim Δ t = 0 t   t  

34. ......... Q (S, a) Q (S, b) Q (S, z) Representing Q Functions More Compactly We can use some other function representation (eg, neural net) to compactly encode this big table An encoding of the state (S) Second argument is a constant Or could have one net for each possible action Each input unit encodes a property of the state (eg, a sensor value)

35. Q Tables vs Q Nets Given: 100 Boolean-valued features 10 possible actions 10 possible actions Size of Q table 10 * 2 to the power of 100 Size of Q net (100 HU’s) 100 * 100 + 100 * 10 = 11,000 # of possible states Weights between inputs and HU’s Weights between HU’s and outputs

36. Why Use a Compact Q-Function? 1.Full Q table may not fit in memory for realistic problems 2.Can generalize across states, thereby speeding up convergence i.e., one example “fills” many cells in the Q table

37. SARSA vs. Q-Learning Exploring can be hazardous! Should we learn to consider its impact? (1994, 1996) (1989) The Cliff-Walking Task (pg 150 of Sutton + Barto RL Text) Start The Cliff (R = -100) Goal R=0 R=-1 for all of these Safe route Optimal path -if no exploration! What would Q-Learning learn?

38. SARSA = State Action Reward State Action SARSA Standard Q-Learning Q(s,a)  Q(s,a) +  [ R +  Q(s’, a’) – Q(s,a) ] Q(s,a)  Q(s,a) +  [ R +  max Q(s’, a’’) – Q(s,a) ] SARSA uses actual next action (still chosen via explore-exploit strategy, e.g. soft-max or “coin-flip”) - SARSA also converges Notice that in Q learning, (currently) non-optimal moves do not impact the Q function a’ s s’ a’’ a’ R actual best

39. Sample Results: Cliff-Walking Task optimal SARSA Q-Learning Total Reward Until Goal Episodes (ie from Start to Goal) (prob of random move = 0.1) SARSA learns to avoid ‘pitfalls’ while in exploration phase (always need to explore if in a real-world situation?)