Quiz 5: Expectimax/Prob review Expectimax assumes the worst case scenario. False Expectimax assumes the most likely scenario. False P(X,Y) is the conditional probability of X given Y. False P(X,Y,Z)=P(Z|X,Y)P(X|Y)P(Y). True Bayes Rule computes P(Y|X) from P(X|Y) and P(Y). True* Bayes Rule only applies in Bayesian Statistics. False FALSE Bayes Rule computes P(Y|X) from P(X|Y) and P(Y). (yeah, it requires P(X) too…) Final Term. P(X|Y) = probabilty of an event X (such as a die came up odd) given that some event Y occurred (such as die cam up prime). P(X|Y=y) = probabilty of an event X (such as a die came up odd) given that some specific outcome y occurred (such as die came up 3). … There is an implicit summation over all the outcomes y that comprise event Y.

CSE511a: Artificial Intelligence Spring 2013 Lecture 8: Maximum Expected Utility 02/11/2013 Robert Pless, course adopted from one given by Kilian Weinberger, many slides over the course adapted from either Dan Klein, Stuart Russell or Andrew Moore

Announcements Project 2: Multi-agent Search is out (due 02/28) Midterm, in class, March 6 (wed. before spring break) Homework (really, exam review) will be out soon. Due before exam.

Prediction of voting outcomes. solve for P(Vote | Polls, historical trends) Creates models for P( Polls | Vote ) Or P(Polls | voter preferences) + assumption preference  vote Keys: Understanding house effects, such as bias caused by only calling land-lines and other choices made in creating sample. Partly data driven --- house effects are inferred from errors in past polls

From PECOTA to politics, 2008 PECOTA originally stood for Pitcher Empirical Comparison [and] Optimization Test Algorithm Missed Indiana + 1 vote in nebraska. +35/35 on senate races.

2012

Utilities Utilities are functions from outcomes (states of the world) to real numbers that describe an agent’s preferences Where do utilities come from? In a game, may be simple (+1/-1) Utilities summarize the agent’s goals Theorem: any set of preferences between outcomes can be summarized as a utility function (provided the preferences meet certain conditions) In general, we hard-wire utilities and let actions emerge

Expectimax Search Chance nodes Chance nodes are like min nodes, except the outcome is uncertain Calculate expected utilities Chance nodes average successor values (weighted) Each chance node has a probability distribution over its outcomes (called a model) For now, assume we’re given the model Utilities for terminal states Static evaluation functions give us limited-depth search 1 search ply Estimate of true expectimax value (which would require a lot of work to compute) … 400 300 … 492 362 …

Quiz: Expectimax Quantities 8 8 3 7 Uniform Ghost 3 12 9 3 6 15 6

Expectimax Pruning? Uniform Ghost 3 12 9 3 6 15 6 Utilities need to be bounded because one outlier can wreck everything! e.g. The UNC major with the highest average salary is Geology. Why? Because Michael Jordan (the basketball player) Was a geology major, and “average” is not a robust statistic. 3 12 9 3 6 15 6 Only if utilities are strictly bounded.

Minimax Evaluation Evaluation functions quickly return an estimate for a node’s true value For minimax, evaluation function scale doesn’t matter We just want better states to have higher evaluations (get the ordering right) We call this insensitivity to monotonic transformations

Expectimax Evaluation For expectimax, we need relative magnitudes to be meaningful 40 20 30 1600 400 900 x2 Expectimax behavior is invariant under positive linear transformation

Expectimax for Pacman [demo: world assumptions] Results from playing 5 games Minimizing Ghost Random Ghost Minimax Pacman Expectimax Pacman python pacman.py -g SmartGhost -l trapped2 -p ExpectimaxAgent -a depth=4,evalFn=better -n 5 python pacman.py -g SmartGhost -l trapped2 -p AlphaBetaAgent -a depth=4,evalFn=better -n 5 python pacman.py -g RandomGhost -l trapped2 -p ExpectimaxAgent -a depth=4,evalFn=better -n 5 python pacman.py -g RandomGhost -l trapped2 -p AlphaBetaAgent -a depth=4,evalFn=better -n 5 Pacman uses depth 4 search with an eval function that avoids trouble Ghost uses depth 2 search with an eval function that seeks Pacman

Expectimax for Pacman [demo: world assumptions] Results from playing 5 games Minimizing Ghost Random Ghost Minimax Pacman Won 5/5 Avg. Score: 493 483 Expectimax Pacman Won 1/5 -303 503 python pacman.py -g SmartGhost -l trapped2 -p ExpectimaxAgent -a depth=4,evalFn=better Pacman used depth 4 search with an eval function that avoids trouble Ghost used depth 2 search with an eval function that seeks Pacman

Expectimax Search Arthur C. Clarke: Having a probabilistic belief about an agent’s action does not mean that agent is flipping any coins! What else could it mean? Magic? Or internal state that you cannot observe (the ghost attacks you, as hard and as fast as it can, but It has a gas tank good for 10 moves, then it has to recharge… so when you first see a ghost, you could model it as having a 1/10 chance Of stopping… Arthur C. Clarke: “Any sufficiently advanced technology is indistinguishable from magic.”

Expectimax Pseudocode def value(s) if s is a max node return maxValue(s) if s is an exp node return expValue(s) if s is a terminal node return evaluation(s) def maxValue(s) values = [value(s’) for s’ in successors(s)] return max(values) def expValue(s) weights = [probability(s, s’) for s’ in successors(s)] return expectation(values, weights) 8 4 5 6

Expectimax for Pacman Notice that we’ve gotten away from thinking that the ghosts are trying to minimize pacman’s score Instead, they are now a part of the environment Pacman has a belief (distribution) over how they will act Quiz: Can we see minimax as a special case of expectimax? Quiz: what would pacman’s computation look like if we assumed that the ghosts were doing 1-ply minimax and taking the result 80% of the time, otherwise moving randomly? If you take this further, you end up calculating belief distributions over your opponents’ belief distributions over your belief distributions, etc… Can get unmanageable very quickly!

Mixed Layer Types E.g. Backgammon Expectiminimax Environment is an extra player that moves after each agent Chance nodes take expectations, otherwise like minimax ExpectiMinimax-Value(state): If it’s terminal or we reached our depth limit, return utility

Maximum Expected Utility Principle of maximum expected utility: A rational agent should chose the action which maximizes its expected utility, given its knowledge Questions: Where do utilities come from? How do we know such utilities even exist? Why are we taking expectations of utilities (not, e.g. minimax)? What if our behavior can’t be described by utilities?

Utility and Decision Theory

Utilities: Unknown Outcomes Going to airport from home Take surface streets Take freeway Clear, 10 min Traffic, 50 min Clear, 20 min Arrive early Arrive very late Arrive a little late

Preferences An agent chooses among: Notation: Prizes: A, B, etc. Lotteries: situations with uncertain prizes Notation: Lottery: probability p, you get prize A probability 1-p, you get “prize B”, which might be negative, like the cost of the lottery ticket.

Rational Preferences Transitivity: We want some constraints on preferences before we call them rational For example: an agent with intransitive preferences can be induced to give away all of its money If B > C, then an agent with C would pay (say) 1 cent to get B If A > B, then an agent with B would pay (say) 1 cent to get A If C > A, then an agent with A would pay (say) 1 cent to get C Apple, banana, carrot

Rational Preferences Preferences of a rational agent must obey constraints. The axioms of rationality: Theorem: Rational preferences imply behavior describable as maximization of expected utility Do you believe these axioms? Orderability: Do you want a fur coat or a smoothie? Well, I’m not indifferent, but I don’t necessarily prefer one… Transitivity I like. Continuity: stupid answer. Sure, exists p. P = 1. then I get A which I like better than B. Other stupid answer: Maybe you dislike chance! Substitutability: ok Monotonicity: ok.

MEU Principle Theorem: Maximum expected utility (MEU) principle: [Ramsey, 1931; von Neumann & Morgenstern, 1944] Given any preferences satisfying these constraints, there exists a real-valued function U such that: Maximum expected utility (MEU) principle: Choose the action that maximizes expected utility Note: an agent can be entirely rational (consistent with MEU) without ever representing or manipulating utilities and probabilities E.g., a lookup table for perfect tictactoe, reflex vacuum cleaner Theorem says that you can convert pairwise preferences into a Utility function so that you can compute a numerical (real-valued) “utility” of each outcome, and comparing the number outputs of the utility function gives you the same pairwise answers, Alternatively… python does not have a “curly greater than”, so you better be able to make your utilities into numbers that act the same way with regular inequalities…

Utility Scales Normalized utilities: u+ = 1.0, u- = 0.0 Micromorts: one-millionth chance of death, useful for paying to reduce product risks, etc. QALYs: quality-adjusted life years, useful for medical decisions involving substantial risk Invariant under linear transformation ---- “relative magnitudes w.r.t arbitrary zero” QALYs account for non-fatal risks (hence the “quality” of the quality adjusted life years). BECAUSE the use of these utilities are invariant to scale, the scale can be chosen to have units that are visceral to a person.

Q: What do these actions have in common? smoking 1.4 cigarettes drinking 0.5 liter of wine spending 1 hour in a coal mine spending 3 hours in a coal mine living 2 days in New York or Boston living 2 months in Denver living 2 months with a smoker living 15 years within 20 miles (32 km) of a nuclear power plant drinking Miami water for 1 year eating 100 charcoal-broiled steaks eating 40 tablespoons of peanut butter travelling 6 minutes by canoe travelling 10 miles (16 km) by bicycle travelling 230 miles (370 km) by car travelling 6000 miles (9656 km) by train flying 1000 miles (1609 km) by jet flying 6000 miles (1609 km) by jet one chest X ray in a good hospital 1 ecstasy tablet Each of these is a “micromort” a 1 in a million chance of death. CONTEXT: 70x365 =~ 25,000 days that you live. So, if deaths are evenly averaged out, there is a 40 micromort chance of death each day. Each of the above adds 1. High risk events: Hang gliding, 8 micromorts per trip Scuba diving, 4.7 micromorts per dive. Skydiving, 7 micromorts per jump. Coal mine 1, black lung disease Coal mine 2, accident. Jet 1, accident. Jet 2, cancer from extra cosmic radiation Living in NYC or Boston (air pollution) Living in Denver (extra cosmic radiation). http://en.wikipedia.org/wiki/Micromort Source Wikipedia

Human Utilities Utilities map states to real numbers. Which numbers? Standard approach to assessment of human utilities: Compare a state A to a standard lottery Lp between “best possible prize” u+ with probability p “worst possible catastrophe” u- with probability 1-p Adjust lottery probability p until A ~ Lp Resulting p is a utility in [0,1] Estimate dollar value by assessing (e.g., how much they are willing to pay for safety features on cars) Pay $50

Money Money does not behave as a utility function, but we can talk about the utility of having money (or being in debt) Given a lottery L = [p, $X; (1-p), $Y] The expected monetary value EMV(L) is p*X + (1-p)*Y U(L) = p*U($X) + (1-p)*U($Y) Typically, U(L) < U( EMV(L) ): why? In this sense, people are risk-averse When deep in debt, we are risk-prone Utility curve: for what probability p am I indifferent between: Some sure outcome x A lottery [p,$M; (1-p),$0], M large Draw Jensen’s for utility curve

Money Money does not behave as a utility function Given a lottery L: Define expected monetary value EMV(L) Usually U(L) < U(EMV(L)) I.e., people are risk-averse Utility curve: for what probability p am I indifferent between: A prize x A lottery [p,$M; (1-p),$0] for large M? Typical empirical data, extrapolated with risk-prone behavior:

Example: Insurance Consider the lottery [0.5,$1000; 0.5,$0] What is its expected monetary value? ($500) What is its certainty equivalent? Monetary value acceptable in lieu of lottery $400 for most people Difference of $100 is the insurance premium There’s an insurance industry because people will pay to reduce their risk If everyone were risk-neutral, no insurance needed!

Example: Insurance Because people ascribe different utilities to different amounts of money, insurance agreements can increase both parties’ expected utility You own a car. Your lottery: LY = [0.8, $0 ; 0.2, -$200] i.e., 20% chance of crashing You do not want -$200! UY(LY) = 0.2*UY(-$200) = -200 UY(-$50) = -150 Amount Your Utility UY $0 -$50 -150 -$200 -1000 Utility of Lottery Y is 0.2 * Uy(- $200 ). ….. Because we choose the scale, let’s define this to be -200. For many

Example: Insurance Because people ascribe different utilities to different amounts of money, insurance agreements can increase both parties’ expected utility You own a car. Your lottery: LY = [0.8, $0 ; 0.2, -$200] i.e., 20% chance of crashing You do not want -$200! UY(LY) = 0.2*UY(-$200) = -200 UY(-$50) = -150 Insurance company buys risk: LI = [0.8, $50 ; 0.2, -$150] i.e., $50 revenue + your LY Insurer is risk-neutral: U(L)=U(EMV(L)) UI(LI) = U(0.8*50 + 0.2*(-150)) = U($10) > U($0) You are risk averse --- so any risk of -200 (even a small risk), you treat as -200. Draw insurance premium on Jensen curve

Example: Human Rationality? Famous example of Allais (1953) A: [0.8,$4k; 0.2,$0] B: [1.0,$3k; 0.0,$0] C: [0.2,$4k; 0.8,$0] D: [0.25,$3k; 0.75,$0] Most people prefer B > A, C > D But if U($0) = 0, then B > A  U($3k) > 0.8 U($4k) C > D  0.8 U($4k) > U($3k) B to A comparison is direct. C to D comparison is same ratio of risk to reward. However --- you have to multiply through, … and there is no guaruntee… Example of how stupid economists are. At the top, there is a premium for NO risk. At the bottom, both have “kinda equal” amounts of risk, so why wouldn’t you take the one that is better?

And now, for something completely different. … well actually rather related…

Digression Simulated Annealing

Example: Deciphering Intercepted message to prisoner in California state prison: [Persi Diaconis]

Search Problem Initial State: Assign letters arbitrarily to symbols

Modified Search Problem Initial State: Assign letters arbitrarily to symbols Action: Swap the assignment of two symbols Terminal State? Utility Function? =A =D =D =A

Utility Function Utility: Likelihood of randomly generating this exact text. Estimate probabilities of any character following any other. (Bigram model) P(‘a’|’d’) Learn model from arbitrary English document

Hill Climbing Diagram

Simulated Annealing Idea: Escape local maxima by allowing downhill moves But make them rarer as time goes on

From: http://math. uchicago