CS 188: Artificial Intelligence Fall 2006 Lecture 13: Advanced Reinforcement Learning 10/12/2006 Dan Klein – UC Berkeley

Midterms Exams are graded, will be in glookup today, returned and gone over in section Monday My impressions: long and fairly hard exam, class generally did fine You should expect that the final will be equally hard, but seem much less long We added 25 points and took it out of 100

Midcourse Reviews You liked: You wanted: Projects (19) Lectures (19) Visual / demo presentations (14) Newsgroup (4) Pacman (3) You wanted: Debugging help / coding advice / see staff code (8) More time between projects (6) More problem sets (4) A webcast or podcast (4) More coding or more projects (4) Slides / reading earlier (3) Class later in the day (3) Lecture to be less fast / dense / technical / confusing (3)

Midcourse Reviews II Difficult / workload is: Dan’s office hours Hard (15) Medium (9) Easy (7) Dan’s office hours Thursday (7) Not Thursday (9)

Midcourse Reviews I propose: Other: I’ll hang out after class on Tuesdays, we can walk to my office if there are more questions I’ll add Thursday OH for a few weeks, and keep if attended We’ll spend more section time devoted to projects I’ll link to last term’s slides so you can get a preview I’ll keep coding demos for you There will be a (slight) shift from projects to written questions Rough “midterm grades” soon, to let you know where you stand (will incorporate at least first two projects) Other: I’ve asked about webcasting / podcasting, but it seems very unlikely this term Limits to how early I can get new slides up, since I do revise extensively from last term Can’t really change: programming language or time of day

Midcourse Reviews - Anonymous

Today How advanced reinforcement learning works for large problems Some previews of fundamental ideas we’ll see throughout the rest of the term Next class we’ll start on probabilistic reasoning and reasoning about beliefs

Recap: Q-Learning Learn Q*(s,a) values from samples Receive a sample (s,a,s’,r) On one hand: old estimate of return: But now we have a new estimate for this sample: Nudge the old estimate towards the new sample: Equivalently, average samples over time:

Q-Learning Q-learning produces tables of q-values:

Q-Learning In realistic situations, we cannot possibly learn about every single state! Too many states to visit them all in training Too many states to even hold the q-tables in memory Instead, we want to generalize: Learn about some small number of training states from experience Generalize that experience to new, similar states This is a fundamental idea in machine learning, and we’ll see it over and over again

Example: Pacman Let’s say we discover through experience that this state is bad: In naïve q-learning, we know nothing about this state or its q-states: Or even this one!

Feature-Based Representations Solution: describe a state using a vector of features Features are functions from states to real numbers (often 0/1) that capture important properties of the state Example features: Distance to closest ghost Distance to closest dot Number of ghosts 1 / (dist to dot)2 Is Pacman in a tunnel? (0/1) …… etc. Can also describe a q-state (s, a) with features (e.g. action moves closer to food)

Linear Feature Functions Using a feature representation, we can write a q-function (or value function) for any state using a few weights: Advantage: our experience is summed up in a few powerful numbers Disadvantage: states may share features but be very different in value!

Function Approximation Q-learning with linear q-functions: Intuitive interpretation: Adjust weights of active features E.g. if something unexpectedly bad happens, disprefer all states with that state’s features Formal justification: online least squares (much later)

Example: Q-Pacman

Hierarchical Learning

Hierarchical RL Stratagus: Example of a large RL task, from Bhaskara Marthi’s thesis (w/ Stuart Russell) Stratagus is hard for reinforcement learning algorithms > 10100 states > 1030 actions at each point Time horizon ≈ 104 steps Stratagus is hard for human programmers Typically takes several person-months for game companies to write computer opponent Still, no match for experienced human players Programming involves much trial and error Hierarchical RL Humans supply high-level prior knowledge using partial program Learning algorithm fills in the details

Partial “Alisp” Program (defun top () (loop (choose (gather-wood) (gather-gold)))) (defun gather-wood () (with-choice (dest *forest-list*) (nav dest) (action ‘get-wood) (nav *base-loc*) (action ‘dropoff))) (defun gather-gold () (with-choice (dest *goldmine-list*) (nav dest)) (action ‘get-gold) (nav *base-loc*)) (action ‘dropoff))) (defun nav (dest) (until (= (pos (get-state)) dest) (move ‘(N S E W NOOP)) (action move)))) Animation, fade outs… Program state Motivate Q(\omega,…) here Before walking through program, Alisp is an extension of Lisp that adds a choose operator Mention nav-choice wd have to be path-planning alg in complete program

Hierarchical RL They then define a hierarchical Q-function which learns a linear feature-based mini-Q-function at each choice point Very good at balancing resources and directing rewards to the right region Still not very good at the strategic elements of these kinds of games (i.e. the Markov game aspect) [DEMO]

Policy Search

Policy Search Problem: often the feature-based policies that work well aren’t the ones that approximate V / Q best E.g. your value functions from 1.3 were probably horrible estimates of future rewards, but they still produce good decisions We’ll see this distinction between modeling and prediction again later in the course Solution: learn the policy that maximizes rewards rather than the value that predicts rewards This is the idea behind policy search, such as what controlled the upside-down helicopter

Policy Search Simplest policy search: Problems: Start with an initial linear value function or q-function Nudge each feature weight up and down and see if your policy is better than before Problems: How do we tell the policy got better? Need to run many sample episodes! If there are a lot of features, this can be impractical

Policy Search* Advanced policy search: Write a stochastic (soft) policy: Turns out you can efficiently approximate the derivative of the returns with respect to the parameters w (details in the book, but you don’t have to know them) Take uphill steps, recalculate derivatives, etc.

Take a Deep Breath… We’re done with search and planning! Next, we’ll look at how to reason with probabilities Diagnosis Tracking objects Speech recognition Robot mapping … lots more! Last part of course: machine learning

Digression / Preview

Linear regression Given examples Predict given a new point Temperature 40 26 24 Temperature 20 22 20 30 40 20 30 10 20 10 20 10 Figure 1: scatter(1:20,10+(1:20)+2*randn(1,20),'k','filled'); a=axis; a(3)=0; axis(a); Given examples Predict given a new point

Linear regression Prediction Prediction Temperature 10 20 30 40 22 24 26 40 Temperature 20 20 Figure 1: scatter(1:20,10+(1:20)+2*randn(1,20),'k','filled'); a=axis; a(3)=0; axis(a); Prediction Prediction

Ordinary Least Squares (OLS) Error or “residual” Observation Prediction Figure 1: scatter(1:20,10+(1:20)+2*randn(1,20),'k','filled'); a=axis; a(3)=0; axis(a); 20

Overfitting Degree 15 polynomial [DEMO] 30 25 20 15 10 5 -5 -10 -15 2 -5 -10 -15 2 4 6 8 10 12 14 16 18 20 [DEMO]