Presentation is loading. Please wait.

Presentation is loading. Please wait.

RL 2 It’s 2:00 AM. Do you know where your mouse is?

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "RL 2 It’s 2:00 AM. Do you know where your mouse is?"— Presentation transcript:

1 RL 2 It’s 2:00 AM. Do you know where your mouse is?

2 First up: Vote! Albuquerque Municipal Election today (Oct 4) Not all of you are eligible to vote, I know...... But if you are, you should. Educate yourself first! Mayor City councilors Bonds (what will ABQ spend its money on?) Propositions (election finance, min wage, voter ID) Polls close at 7:00 PM today...

3 Voting resources City of Albuquerque web site: www.cabq.govwww.cabq.gov League of Women Voters web site: http://www.lwvabc.org/elections/2005VG_Engli sh.html http://www.lwvabc.org/elections/2005VG_Engli sh.html

4 News o’ the day Wall Street Journal reports: “Microsoft Windows Officially Broken” In 2004, MS Longhorn (successor to XP) bogged down Whole code base had to be scrapped & started afresh ⇒ Vista Point: not MS bashing (much) Importance of software process MS moved to a more agile process for Vista Test first Rigorous regression testing Better coding infrastructure

5 Administrivia Grading: P1 rollout finished grading I will send grade reports this afternoon & tomorrow morning Prof Lane out of town Oct 11 Andree Jacobsen will cover Stefano Markidis out of town Oct 19 Will announce new office hours presently

6 Your place in History Last time: Q2 Introduction to Reinforcement Learning (RL)

7 Your place in History This time: ✓ P2M1 due ✓ Voting ✓ News ✓ Administriva ✓ Q&A More on RL Design exercise: WorldSimulator and Terrains

8 Recall: Mack & his maze Mack lives a hard life as a psychology test subject Has to run around mazes all day, finding food and avoiding electric shocks Needs to know how to find cheese quickly, while getting shocked as little as possible Q: How can Mack learn to find his way around? ?

9 Reward over time s1s1 s2s2 s3s3 s4s4 s5s5 s6s6 s4s4 s2s2 s7s7 s 11 s8s8 s9s9 s 10

10 Reward over time s1s1 s2s2 s3s3 s4s4 s5s5 s6s6 s4s4 s2s2 s7s7 s 11 s8s8 s9s9 s 10 V(s 1 )=R(s 1 )+R(s 4 )+R(s 11 )+R(s 10 )+...

11 Reward over time s1s1 s2s2 s3s3 s4s4 s5s5 s6s6 s4s4 s2s2 s7s7 s 11 s8s8 s9s9 s 10 V(s 1 )=R(s 1 )+R(s 2 )+R(s 6 )+...

12 Where can you go? Definition: Complete set of all states agent could be in is called the state space: S Could be discrete or continuous For Project 2: states are discrete Q: what is the state space for P2? Size of state space: | S | Q: How big is the state space for P2?

13 Where can you go? Definition: Complete set of actions an agent could take is called the action space: A Again, discrete or cont. Again, P2: A is discrete Q: What is A for P2? Size? Again, size: | A |

14 What is it worth to you? Idea of “good” and “bad” places to go Quantified as “rewards” (This is where term “reinforcement learning” comes from. Originated in psychology.) Formally: R : S → Reals R(s) == reward for getting to state s How good or bad it is to reach state s Larger (more positive) is better Agent “wants” to get more positive rwd

15 How does it happen? Dynamics of agent defined by transition function T: S x A x S → [0,1] T(s,a,s’)==Pr[next state is s’ | curr state is s, act a] Examples from P2?

16 How does it happen? Dynamics of agent defined by transition function T: S x A x S → [0,1] T(s,a,s’)==Pr[next state is s’ | curr state is s, act a] Examples from P2? In practice: Don’t write T down explicitly. Encoded by WorldSimulator and Terrain/agent interactions.

17 The MDP Entire RL environment defined by a Markov decision process: M = 〈 S, A,T,R 〉 S : state space A : action space T : transition function R : reward function Q: What modules represent these in P2?

18 Policies Total accumulated reward (value, V ) depends on Where agent starts What agent does at each step (duh)

19 Policies Total accumulated reward (value, V ) depends on Where agent starts What agent does at each step (duh) Plan of action is called a policy, π Policy defines what action to take in every state of the system:

20 Experience & histories Fundamental unit of experience in RL: At time t in some state s i, take action a j, get reward r t, end up in state s k Called an experience tuple or SARSA tuple Set of all experience during a single episode up to time T is a history or trajectory:

21 How good is a policy? Value is a function of start state and policy: Value measures: How good is policy π, averaged over all time, if agent starts at state s 1 and runs forever?

22 The goal of RL Agent’s goal: Find the best possible policy: π* Find policy, π*, that maximizes V π (s) for all s

23 Design Exercise: WorldSimulator & Friends

24 Design exercise Q1: Design the act() method in WorldSimulator What objects does it need to access? How can it take different terrains/agents into account? Q2: GridWorld2d could be really large Most of the terrain tiles are the same everywhere How can you avoid millions of copies of same tile?

Download ppt "RL 2 It’s 2:00 AM. Do you know where your mouse is?"

Similar presentations

1 Reinforcement Learning Problem Week #3. Figure reproduced from the figure on page 52 in reference [1] 2 Reinforcement Learning Loop state Agent Environment.

1 Reinforcement Learning Problem Week #3. Figure reproduced from the figure on page 52 in reference [1] 2 Reinforcement Learning Loop state Agent Environment.
COSC 878 Seminar on Large Scale Statistical Machine Learning 1.

COSC 878 Seminar on Large Scale Statistical Machine Learning 1.
Reinforcement learning

Reinforcement learning
Announcements Homework 3: Games Project 2: Multi-Agent Pacman

Announcements Homework 3: Games Project 2: Multi-Agent Pacman
RL at Last! Q- learning and buddies. Administrivia R3 due today Class discussion Project proposals back (mostly) Only if you gave me paper; e-copies yet.

RL at Last! Q- learning and buddies. Administrivia R3 due today Class discussion Project proposals back (mostly) Only if you gave me paper; e-copies yet.
RL Cont’d. Policies Total accumulated reward (value, V ) depends on Where agent starts What agent does at each step (duh) Plan of action is called a policy,

RL Cont’d. Policies Total accumulated reward (value, V ) depends on Where agent starts What agent does at each step (duh) Plan of action is called a policy,
Model-Free vs. Model- Based RL: Q, SARSA, & E 3. Administrivia Reminder: Office hours tomorrow truncated 9:00-10:15 AM Can schedule other times if necessary.

Model-Free vs. Model- Based RL: Q, SARSA, & E 3. Administrivia Reminder: Office hours tomorrow truncated 9:00-10:15 AM Can schedule other times if necessary.
Intro to Reinforcement Learning Learning how to get what you want...

Intro to Reinforcement Learning Learning how to get what you want...
Bayesian Reinforcement Learning with Gaussian Processes Huanren Zhang Electrical and Computer Engineering Purdue University.

Bayesian Reinforcement Learning with Gaussian Processes Huanren Zhang Electrical and Computer Engineering Purdue University.
Nov 14 th  Homework 4 due  Project 4 due 11/26.

Nov 14 th  Homework 4 due  Project 4 due 11/26.
Reinforcement Learning (for real this time). Administrivia No noose is good noose (Crazy weather?)

Reinforcement Learning (for real this time). Administrivia No noose is good noose (Crazy weather?)
Reinforcement Learning, Cont’d Useful refs: Sutton & Barto, Reinforcement Learning: An Introduction, MIT Press 1998.

Reinforcement Learning, Cont’d Useful refs: Sutton & Barto, Reinforcement Learning: An Introduction, MIT Press 1998.
Planning in MDPs S&B: Sec 3.6; Ch. 4. Administrivia Reminder: Final project proposal due this Friday If you haven’t talked to me yet, you still have the.

Planning in MDPs S&B: Sec 3.6; Ch. 4. Administrivia Reminder: Final project proposal due this Friday If you haven’t talked to me yet, you still have the.
Reinforcement Learning Reinforced. Administrivia I’m out of town next Tues and Wed Class cancelled Apr 4 -- work on projects! No office hrs Apr 4 or 5.

Reinforcement Learning Reinforced. Administrivia I’m out of town next Tues and Wed Class cancelled Apr 4 -- work on projects! No office hrs Apr 4 or 5.
Machine Learning Lecture 11: Reinforcement Learning

Machine Learning Lecture 11: Reinforcement Learning
Reinforcement Learning: Learning to get what you want... Sutton & Barto, Reinforcement Learning: An Introduction, MIT Press 1998.

Reinforcement Learning: Learning to get what you want... Sutton & Barto, Reinforcement Learning: An Introduction, MIT Press 1998.
Planning to learn. Progress report Last time: Transition functions & stochastic outcomes Markov chains MDPs defined Today: Exercise completed Value functions.

Planning to learn. Progress report Last time: Transition functions & stochastic outcomes Markov chains MDPs defined Today: Exercise completed Value functions.
RL: Algorithms time. Happy Full Moon! Administrivia Reminder: Midterm exam, this Thurs (Oct 20) Spec v 0.98 released today (after class) Check class.

RL: Algorithms time. Happy Full Moon! Administrivia Reminder: Midterm exam, this Thurs (Oct 20) Spec v 0.98 released today (after class) Check class.
More RL. MDPs defined A Markov decision process (MDP), M, is a model of a stochastic, dynamic, controllable, rewarding process given by: M = 〈 S, A,T,R.

More RL. MDPs defined A Markov decision process (MDP), M, is a model of a stochastic, dynamic, controllable, rewarding process given by: M = 〈 S, A,T,R.
The People Have Spoken.... Administrivia Final Project proposal due today Undergrad credit: please see me in office hours Dissertation defense announcements.

The People Have Spoken.... Administrivia Final Project proposal due today Undergrad credit: please see me in office hours Dissertation defense announcements.

Similar presentations

Ads by Google