1. Reinforcement Learning
Unsupervised learning
Reinforcement Learning

2. So far ….
Supervised machine learning: given a set of annotated istances and a set of categories, find a model to automatically categorize unseen istances
Unsupervided learning: no examples are available
Previous lesson: clustering
Given a set of instances described by feature vectors, group them in clusters such as intra-cluster similarity is maximized and inter-cluster dissimilarity is maximized

3. Reinforcement learning
Agent receives no examples and starts with no model of the environment.
Agent gets feedback through rewards, or reinforcement.

4. Learning by reinforcement
Examples:
Learning to play Backgammon
Robot learning to move in an environment
Characteristics:
No direct training examples – (delayed) rewards instead
Need for exploration of environment & exploitation
The environment might be stochastic and/or unknown
The actions of the learner affect future rewards

5. Example: Robot moving in an environment

6. Example: playing a game (e.g. backgammon, chess)

7. Reinforcement Learning
Supervised Learning
Input is some “goal” output
Is a sequence of actions to
meet the goal
Input is an istance, output is a
classification of the istance

8. Elements of RL
In RL, the “model” to be learned is a policy to meet “at best”
some given goal (e.g. win a game)
Agent
Policy
State
Reward
Action
Environment
Transition model, how actions A influence states S
Reward R, immediate value of state-action transition
Policy , maps states to actions
By a model of the environment we mean anything that an agent can use to predict how the environment will respond to its actions. Given a state and an action, a model produces a prediction of the resultant next state and next reward. If the model is stochastic, then there are several possible next states and next rewards, each with some probability of occurring

9. Markov Decision Process (MDP)
MDP is a formal model of the RL problem
At each discrete time point
Agent observes state st and chooses action at (according to some probability distribution)
Receives reward rt from the environment and the state changes to st+1
Markov assumption:
rt=r(st,at) st+1=(st,at)
i.e. rt and st+1 depend only on the current state and action
In general, the functions r and  may not be deterministic (i.e. they are stochastic) and are not necessarily known to the agent

10. Agent’s Learning Task
Execute actions in environment, observe results and
Learn action policy that maximises expected cumulative reward
from any starting state in S.
Here is the discount factor for future rewards
Note:
Target function is
There are no training examples of the form (s,a) but only of the form ((s,a),r)

11. Example: TD-Gammon Immediate reward:
+100 if win
-100 if lose
0 for all other states
Trained by playing 1.5 million games against itself
Now approximately equal to the best human player

12. Example: Mountain-Car
States: position and velocity
Actions: accelerate forward, accelerate backward, coast
Rewards
Reward=-1for every step, until the car reaches the top
Reward=1 at the top, 0 otherwise, <1
The eventual reward will be maximised by minimising the number of steps to the top of the hill

13. Value function We will consider deterministic worlds first
In a deterministic environment, rewards are known and the environment is known
Given a policy π (adopted by the agent), define an evaluation function over states:

14. Example: robot in a grid environment
Grid world environment
Six possible states
Arrows represent possible actions
G: goal state
r(state, action)
immediate reward values
100 G
r( (2,3), UP)=100

15. Example: robot in a grid environment
The “value” of being
in (1,2) according to
some known π is
100
Value function: maps states to state values, acording to some policy π
r(state, action)
immediate reward values
100 G G 90
100 81 G 90
100 81 G 90
100 81
V*(state) values

16. Computing V(s) (if policy is given)
Suppose the following is an
Optimal policy – denoted * :
* tells us what is the best thing to do when in each state.
According to * , we can compute the values of the states for this policy – denoted V*, or V*

17. r(s,a) (immediate reward) values: V*(s) values, with =0.9
one optimal policy:
V*(s6) = *0 = 100
V*(s5) = *100 = 90
V*(s4) = *90 = 81
However, * is usually unknown and the task is to learn the optimal policy

18. Learning * Target function is  : state  action However…
We have no training examples of form <state, action>
We only know:
<<state, action>, reward>
Reward might not be known for all states!

19. Utility-based agents Try to learn V * (abbreviated V*)
Perform look ahead search to choose best action from any state s
Works well if agent knows
 : state  action  state
r : state  action  R
When agent doesn’t know  and r, cannot choose actions this way

20. Q-learning: an utility-based learner in deterministic environments
Q-values
Define new function very similar to V*
If agent learns Q, it can choose optimal action even without knowing  or R
Using Q

21. Learning the Q-value Note: Q and V* closely related
Allows us to write Q recursively as:
This policy is also called Temporal Difference learning
V(s(t+1))

22. Learning the Q-value FOR each <s, a> DO Observe current state s
Initialize table entry:
Observe current state s
WHILE (true) DO
Select action a and execute it
Receive immediate reward r
Observe new state s’
Update table entry for as follows
Move: record transition from s to s’

23. Q-Learning: Example (=0.9)
actions 1 s1 s2 s3 s4 s5 s6 N S W E s1 s2 s3 s4
0+0.9*0 =0 s5 s6 1
s=s4, s’=s5
Select a possible action in s and execute
Obtain the reward r
Observe new state s’ reached when executing a from s
Update estimate as follows:
s  s’
Q-values table
The “max Q” function search for the “most promising action” from s’
But in s5: therefore:

24. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2 s3 s4 s5
0+0.9*0 =0 s6 1
s=s5, s’=s6
Select a possible action in s and execute
Obtain the reward r
Observe new state s’ reached when executing a from s
Update estimate as follows:
s  s’

25. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2 s3 s4 s5 s6
1+0.9*0 =1 1
s=s6, s’=s3
In s3 r=1 so Q is updated to 1
Select a possible action in s and execute
Obtain the reward r
Observe new state s’ reached when executing a from s
Update estimate as follows:
s  s’
GOAL REACHED: END OF FIRST EPISODE

26. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2 s3 s4
0+0.9*0 =0 s5 s6 1 1
s=s4, s’=s1
Select a possible action in s and execute
Obtain the reward r
Observe new state s’ reached when executing a from s
Update estimate as follows:
s  s’

27. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1
0+0.9*0 =0 s2 s3 s4 s5 s6 1 1
s=s1, s’=s2
Select a possible action in s and execute
Obtain the reward r
Observe new state s’ reached when executing a from s
Update estimate as follows:
s  s’

28. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2
1+0.9*0 =1 s3 s4 s5 s6 1 1
s=s2, s’=s3
Select a possible action in s and execute
Obtain the reward r
Observe new state s’ reached when executing a from s
Update estimate as follows:
s  s’
GOAL REACHED: END OF 2nd EPISODE

29. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2 1 s3 s4
0+0 s5 s6 1
s=s4, s’=s5

30. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2 1 s3 s4 s5
0+0.9*1
=0.9 s6 1
s=s5, s’=s2

31. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2
1+0 s3 s4 s5
0.9 s6 1 1
s=s2, s’=s3
GOAL REACHED: END OF 3rd EPISODE

32. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2 1 s3 s4
0+0.9*0.9
=0.81 s5
0.9 s6 1
s=s4, s’=s5

33. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2 1 s3 s4
0.81 s5
0+0.9*1
=0.9 s6 1
s=s5, s’=s2

34. Q-Learning: Esempio (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2
1+0 s3 s4
0.81 s5
0.9 s6 1 1
s=s2, s’=s3
GOAL REACHED: END OF 4th EPISODE

35. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1 s2 1 s3 s4
0+0
0.81 s5
0.9 s6 1
s=s4, s’=s1

36. Q-Learning: Example (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1
0+0.9*1
=0.9 s2 1 s3 s4
0.81 s5
0.9 s6 1
s=s1, s’=s2

37. Q-Learning: Esempio (=0.9)1 s1 s2 s3 s4 s5 s6 N S O E s1
0.9 s2
1+0 s3 s4
0.81 s5 s6 1 1
s=s2, s’=s3
GOAL REACHED: END OF 5th EPISODE

38. Q-Learning: Example(=0.9)
After several iterations, the algorithm converges to the following table: N S O E s1
0.72
0.9 s2
0.81 1 s3 s4 s5 s6 s1 s2 s3 s4 s5 s6
0.9 1
0.81
0.9
0.72
0.81
0.81 1
0.81
0.9
0.72
0.81

39. A crawling robot using Q-Learning
An example of Q-learning using 16 states
(4 positions for each servo's) and 4 possible actions
(moving 1 position up or down for on of the servos).
The robot is build with lego components, two servo's, a wheel
encoder and a dwengo-board with a microcontroller.
The robot learns using the Q-learning algorithm
and its reward is related with the moved distance.

40. The non-deterministic case
What if the reward and the state transition are not deterministic? – e.g. in Backgammon learning and playing depends on rolls of dice!
Then V and Q needs redefined by taking expected values
Similar reasoning and convergent update iteration will apply

41. Summary
Reinforcement learning is suitable for learning in uncertain environments where rewards may be delayed and subject to chance
The goal of a reinforcement learning program is to maximise the eventual reward
Q-learning is a form of reinforcement learning that doesn’t require that the learner has prior knowledge of how its actions affect the environment