Presentation is loading. Please wait.

Presentation is loading. Please wait.

RL2: Q(s,a) easily gets us U(s) and pi(s) Leftarrow with alpha above = move towards target, weighted by learning rate Sum of alphas is infinity, sum of.

Published byGilbert Cunningham Modified over 9 years ago

Similar presentations

Presentation on theme: "RL2: Q(s,a) easily gets us U(s) and pi(s) Leftarrow with alpha above = move towards target, weighted by learning rate Sum of alphas is infinity, sum of."— Presentation transcript:

1 RL2: Q(s,a) easily gets us U(s) and pi(s) Leftarrow with alpha above = move towards target, weighted by learning rate Sum of alphas is infinity, sum of squared alphas is less than infinity Simulated annealing -> epsilon exploration – Combine random with greedy: exploration + exploitation GLIE: Greedy limit + infinite exploration

2

3

4

5

6 t, r -> planner -> policyDyn. Prog. s, a, s’, r -> learner -> policyMonte Carlo s, a, s’, r -> modeler -> t,r model -> simulate -> s, a, s’, r Q-Learning ? TD ?

7

8

9

10

11 DP, MC, DP Model, online/offline update, bootstrapping, on-policy/off-policy Batch updating vs. online updating

12 Ex. 6.4 V(B) = ¾

13 Ex. 6.4 V(B) = ¾ V(A) = 0? Or ¾?

14

15 4 is terminal state V(3) = 0.5 TD(0) here is better than MC for V(2). Why?

16 4 is terminal state V(3) = 0.5 TD(0) here is better than MC. Why? Visit 2 on the kth time, state 3 visited 10k times Variance for MC will be much higher than TD(0) because of bootstrapping

17 4 is terminal state V(3) = 0.5 Change so that R(3,a,4) was deterministic Now, MC would be faster

18 What happened in the 1 st episode?

19 Ex 6.4, Figure 6.7. MC vs TD? TD RMS goes down and up again with high learning rates?

20 Ex 6.4, Figure 6.7. RMS goes down and up again with high learning rates. Would things change if you had a -1 on each step?

21 Q-Learning

22

23 Decaying learning rate? Decaying exploration rate?

Download ppt "RL2: Q(s,a) easily gets us U(s) and pi(s) Leftarrow with alpha above = move towards target, weighted by learning rate Sum of alphas is infinity, sum of."

Similar presentations

Lecture 18: Temporal-Difference Learning

Lecture 18: Temporal-Difference Learning
Programming exercises: Angel – lms.wsu.edu – Submit via zip or tar – Write-up, Results, Code Doodle: class presentations Student Responses First visit.

Programming exercises: Angel – lms.wsu.edu – Submit via zip or tar – Write-up, Results, Code Doodle: class presentations Student Responses First visit.
RL for Large State Spaces: Value Function Approximation

RL for Large State Spaces: Value Function Approximation
TEMPORAL DIFFERENCE LEARNING Mark Romero – 11/03/2011.

TEMPORAL DIFFERENCE LEARNING Mark Romero – 11/03/2011.
Monte-Carlo Methods Learning methods averaging complete episodic returns Slides based on [Sutton & Barto: Reinforcement Learning: An Introduction, 1998]

Monte-Carlo Methods Learning methods averaging complete episodic returns Slides based on [Sutton & Barto: Reinforcement Learning: An Introduction, 1998]
INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, 2014 Lecture.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.
Reinforcement learning (Chapter 21)

Reinforcement learning (Chapter 21)
1 Monte Carlo Methods Week #5. 2 Introduction Monte Carlo (MC) Methods –do not assume complete knowledge of environment (unlike DP methods which assume.

1 Monte Carlo Methods Week #5. 2 Introduction Monte Carlo (MC) Methods –do not assume complete knowledge of environment (unlike DP methods which assume.
1 Reinforcement Learning Introduction & Passive Learning Alan Fern * Based in part on slides by Daniel Weld.

1 Reinforcement Learning Introduction & Passive Learning Alan Fern * Based in part on slides by Daniel Weld.
1 Temporal-Difference Learning Week #6. 2 Introduction Temporal-Difference (TD) Learning –a combination of DP and MC methods updates estimates based on.

1 Temporal-Difference Learning Week #6. 2 Introduction Temporal-Difference (TD) Learning –a combination of DP and MC methods updates estimates based on.
1 Eligibility Traces (ETs) Week #7. 2 Introduction A basic mechanism of RL. λ in TD(λ) refers to an eligibility trace. TD methods such as Sarsa and Q-learning.

1 Eligibility Traces (ETs) Week #7. 2 Introduction A basic mechanism of RL. λ in TD(λ) refers to an eligibility trace. TD methods such as Sarsa and Q-learning.
Adapted from R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction From Sutton & Barto Reinforcement Learning An Introduction.

Adapted from R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction From Sutton & Barto Reinforcement Learning An Introduction.
Reinforcement Learning

Reinforcement Learning
Reinforcement Learning

Reinforcement Learning
Reinforcement learning

Reinforcement learning
Università di Milano-Bicocca Laurea Magistrale in Informatica Corso di APPRENDIMENTO E APPROSSIMAZIONE Lezione 6 - Reinforcement Learning Prof. Giancarlo.

Università di Milano-Bicocca Laurea Magistrale in Informatica Corso di APPRENDIMENTO E APPROSSIMAZIONE Lezione 6 - Reinforcement Learning Prof. Giancarlo.
Reinforcement Learning Tutorial

Reinforcement Learning Tutorial
Reinforcement Learning

Reinforcement Learning
CS 182/CogSci110/Ling109 Spring 2008 Reinforcement Learning: Algorithms 4/1/2008 Srini Narayanan – ICSI and UC Berkeley.

CS 182/CogSci110/Ling109 Spring 2008 Reinforcement Learning: Algorithms 4/1/2008 Srini Narayanan – ICSI and UC Berkeley.
Blackjack as a Test Bed for Learning Strategies in Neural Networks A. Perez-Uribe and E. Sanchez Swiss Federal Institute of Technology IEEE IJCNN'98.

Blackjack as a Test Bed for Learning Strategies in Neural Networks A. Perez-Uribe and E. Sanchez Swiss Federal Institute of Technology IEEE IJCNN'98.

Similar presentations

Ads by Google