©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Uncertainty Chapter 13.1-13.3, + Introduction & Basics Probability Theory and Notation Unconditional, Joint, Conditional Probabilities Probability Distributions & Full Joint Prob. Dist. Table Action-State Utility Matrix Expected Utility & Optimal Decision Making 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Autonomous Agents Agent Environment Model of World (being updated) Prior Knowledge about the World Sensors Reasoning & Decisions Making List of Possible Actions Goals/Utility Effectors 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. How an Agent Operates Basic Cycle: use sensors to sense the environment update the world model reason about the world (infer new facts) update plan on how to reach goal make decision on next action use effectors to implement action Basic cycle is repeated until goal is reached 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Autonomous Agents Real World Driving Agent Model of vehicle location & status road status Prior Knowledge physics of movement rules of road Sensors camera tachometer engine status temperature Reasoning & Decisions Making Actions change speed change steering Goals drive home Effectors accelerator brakes steering 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

The Agent’s World Model World Model: internal representation of the external world that combines: current inputs prior states of world model background knowledge Necessarily, the world model is a simplification e.g. in driving we cannot represent every detail every pebble, leaf, snow flake on the road? every dent and scratch in every vehicle in sight? 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Uncertainty in the World Model The agent can never be completely certain about the state of the external world since there is ambiguity and uncertainty. Why? sensors have limited precision e.g. camera has only so many pixels to capture an image sensors have limited accuracy e.g. tachometer’s estimate of velocity is approximate there are hidden variables that sensors can’t “see” e.g. vehicle behind large truck or storm clouds approaching the future is unknown, uncertain, i.e. cannot foresee all possible future events which may happen 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Rules and Uncertainty Say we have a rule: if toothache then problem is cavity But not all patients have toothaches due to cavities so we could set up rules like: if toothache and not(gum disease) and not(filling) and ... then problem = cavity This gets complicated, a better method would be: if toothache then problem is cavity with 0.8 probability or P(cavity|toothache) = 0.8 the probability of cavity is 0.8 given toothache is all that is known 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Example of Uncertainty Assume a camera and vision system is used to estimate the curvature of the road ahead There's uncertainty about which way it curves limited pixel resolution, noise in image algorithm for “road detection” is not perfect This uncertainty can be represented with a simple probability model: P(road curves to left|E) = 0.6 P(road goes straight|E) = 0.3 P(road curves to right|E) = 0.1 where the probability of an event is a measure of agent’s belief in the event given the evidence E 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Uncertainty in the World Model True uncertainty: rules are probabilistic in nature quantum mechanics rolling dice, flipping a coin? Laziness: too hard to determine exceptionless rules takes too much work to determine all of the relevant factors too hard to use the enormous rules that result Theoretical ignorance: don't know all the rules problem domain has no complete theory (medical diagnosis) Practical ignorance: do know all the rules BUT haven't collected all relevant information for a particular case 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Logics Logics are characterized by what they commit to as "primitives". Logic What Exists in World Knowledge States Propositional facts true/false/unknown First-Order facts, objects, relations Temporal facts, objects, relations, times Probability Theory degree of belief 0..1 Fuzzy degree of truth 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Probability Theory Probability theory serves as a formal means for representing and reasoning with uncertain knowledge of manipulating degrees of belief in a proposition (event, conclusion, diagnosis, etc.) 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Probability Notation Uses a representation that is similar to propositional logic but more expressive being able to represent probabilities A proposition (event) is: A=a which means variable A takes value a e.g. RoadCurvature=left either true or false in the world agent has a degree of belief in the proposition 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Probability Notation Random Variables (RV): are capitalized (usually) e.g. Sky, RoadCurvature, Temperature refer to attributes of the world whose "status" is unknown have one and only one value at a time have a domain of values that are possible states of the world: boolean: domain = <true, false> Cavity=true abbreviated as cavity Cavity=false abbreviated as Øcavity discrete: domain is countable (includes boolean) values are exhaustive and mutually exclusive e.g. Sky domain = <clear, partly_cloudy, overcast> Sky=clear abbreviated as clear Sky¹clear also abbrv. as Øclear continuous: domain is real numbers (beyond scope of CS540) 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Probability Notation An agent’s uncertainty is represented by: P(A=a) or simply P(a), this is: the agent’s degree of belief that variable A takes on value a given no other information relating to A a single probability called an unconditional or prior probability Property:  P(ai) = P(a1) + P(a2) + ... + P(an) = 1 sum over all values in the domain of variable A is 1 because domain is exhaustive and mutually exclusive 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Example "Clue" Goal to determine who killed Mr. John Boddy, where and with what weapon: 6 characters (Mr. Green, Miss Scarlet, Prof. Plum, …) 6 weapons (rope, lead pipe, candlestick, knife, …) 9 locations (conservatory, kitchen, billiard room, …) Prior probabilities before any evidence is obtained each character has 1/6 probability of being Who murdered e.g. P(Who=green) = P(green) = 1/6 each weapon has 1/6 probability of being What caused murder each location has 1/9 probability of being Where murdered e.g. P(Where=kitchen) = P(kitchen) = 1/9 Total probability for each random variable is 1 Parker Bros. introduced in 1949 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Source of Probabilities Frequentists: probabilities come from experiments if 10 of 100 people tested have a cavity the P(cavity) = .1 probability means the fraction that would be observed in the limit of infinitely many samples Objectivists: probabilities are real aspects of the world objects have a propensity to behave in certain ways coin has a propensity to come up heads with a probability .5 Subjectivists: probabilities characterize an agent's belief have no external physical significance 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Atomic Events An atomic event is a complete specification of the state of the world about which the agent is uncertain. e.g. green Ù rope Ù kitchen or plum Ù knife Ù conservatory Properties of atomic events: they're mutually exclusive: at most one can be the case set of all possible atomic events is exhaustive: at least one must be the case any particular atomic event entails the truth/falsehood of every proposition any proposition is logically equivalent to the disjunction of all atomic events that entail the truth of that proposition 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Probability Distributions Given A is a RV taking values in <a1, a2, … , an> e.g. if A is Sky, then a is one of <clear, partly_cloudy, overcast> P(a) represents a single probability where A=a e.g. if A is Sky, then P(a) means any one of P(clear), P(partly_cloudy), P(overcast) e.g. P(conservatory) = 1/9 P(A) represents a probability distribution the set of values {P(a1), P(a2), …, P(an)} If A takes n values, then P(A) is a set of n probabilities e.g. if A is Sky, then P(Sky) is the set of probabilities: {P(clear), P(partly_cloudy), P(overcast)} e.g. P(Who) = {1/6, 1/6, 1/6, 1/6, 1/6, 1/6} 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Joint Probabilities Joint probabilities specify the probabilities for the conjunction of propositions. P(a,b,…): joint probability of A=a Ù B=b Ù … e.g. P(green, rope, library) = P(green Ù rope Ù library) = P(Who=green Ù What=rope Ù Where=library) P(A,B,…): joint probability distribution e.g. P(Who,Where) is a 6 * 9 table of probabilities P(green, library) P(green, conservatory) … 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Joint Probabilities Full joint probability distribution: e.g. P(Who,What,Where) is a 6 * 6 * 9 table of probabilities completely specifies all of the possible probabilistic information by assigning probabilities to all possible combinations of values of all the random variables, which encode all of the relevant information about a problem intractable representation since table grows exponentially in size, kn where n variables each have k possible values 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Conditional Probability Conditional (posterior) probabilities formalize the process of accumulating evidence and updating probabilities based on new evidence specify the belief in a proposition (event, conclusion, diagnosis, etc.) that is conditioned on a proposition (evidence, feature, symptom, etc.) being true P(a|e): conditional probability of A=a given E=e evidence is all that is known true P(a|e) = P(a Ù e) / P(e) = P(a, e) / P(e) the conditional probability can viewed as the joint probability P(a, e) normalized by the prior probability P(e) e.g. P(green|Øwhite) = P(green Ù Øwhite)/P(Øwhite) = 1/6/5/6 = 1/5 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Conditional Probability Conditional probabilities behave exactly like standard probabilities, for example: 0 <= P(a|e) <= 1 conditional probabilities are between 0 and 1 inclusive P(a1|e) + P(a2|e) + ... + P(an|e) = 1 conditional probabilities sum to 1 where a1, …, an are all values in the domain of RV A P(Øa|e) = 1 - P(a|e) negation for conditional probabilities 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Conditional Probability P(conjunction of events|e) P(a Ù b Ù c | e) or as P(a, b, c | e) is the agent’s belief in the sentence a Ù b Ù c conditioned on e being true P(a| conjunction of evidences) P(a | e Ù f Ù g) or as P(a | e, f, g) is the agent’s belief in the sentence a conditioned on e Ù f Ù g being true 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Kolmogorov's Axioms of Probability 0 <= P(a) <= 1 probabilities are between 0 and 1 inclusive P(true) = 1, P(false) = 0 probability of 1 for propositions believed to be absolutely true probability of 0 for propositions believed to be absolutely false P(a Ú b) = P(a) + P(b) - P(a Ù b) probability of two states is their sum minus their “intersection” Property: P(Øa) = 1 - P(a) belief in Øa is one minus agent's belief in a It has been shown that these are necessary if an agent is to behave rationally (see pp. 473-4). 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Actions and States What is the optimal action to take given an agent's uncertain model of the world? A rational agent will want to take the best action given information about the states. For example in what direction should a driving agent steer given: probabilities for state of road curvature: {P(left), P(straight), P(right)} and set of actions: {steer left, steer straight, steer right} 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Action-State Utility Matrix u(A, s) is the utility to an agent which would result from action A if the world were really in state s utility is usually measured in units of “negative cost” for the agent e.g. u(write $100 check, balance = $50) = -$10 fee For a set of actions A and a set of states S this gives a utility matrix 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Action-State Utility Matrix Example Random variable C is for curvature of the road domain is <l, s, r> l = left, s = straight, r = right Action A domain is <SL, SS, SR, Halt> SL = steer left, SS = steer straight, SR = steer right STATE of C ACTION A l s r SL -1000 SS -20 SR Halt -500 u(SL, l) = 0 u(SR, l) = -$20 go off the road u(SL, r) = -$1000 cross into oncoming traffic! 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Expected Utilities (EU) The agent reasons “hypothetically” since it never really exactly knows the state of the world How can the driving agent choose the best action given probabilities about the state of the world? e.g. say P(l) = 0.2, P(s) = 0.7, P(r) = 0.1 The expected utility of an action is the sum of the utility over all possible states of the world = sum over states of {utility(Action, state) * P(state)} e.g. EU(SL) = u(SL,l) * P(l) + u(SL,s) * P(s) + u(SL,r) * P(r) = -0 * 0.2 + -20 * 0.7 + -1000 * 0.1 = - 114 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Optimal Decision-Making Optimal decision-making: choosing the best action calculate the expected utility for each action choose action with optimal expected utility (EU) optimal EU is typically the minimum cost This strategy is the optimal strategy for an agent that must make decisions in an uncertain world assumes the probabilities are accurate assumes the utilities are accurate is a “greedy” strategy: only optimizes 1 step ahead 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

Optimal Decision-Making Example Use action-state utility matrix from prior slide State Probabilities are P(l) = 0.2, P(s) = 0.7, P(r) = 0.1 Calculate EU for all actions: EU(SL) = 0 * 0.2 + -1000 * 0.7 + -1000 * 0.1 = -800 EU(SS) = -20 * 0.2 + 0 * 0.7 + -1000 * 0.1 = -104 EU(SR) = -20 * 0.2 + -20 * 0.7 + 0 * 0.1 = -18 EU(Halt) = -500 * 0.2 + -500 * 0.7 + -500 * 0.1 = -500 Optimal utility action = steer right note in this case this is the least likely state of the world! but it is the one that has optimal expected utility i.e. it is the action that on average will minimize cost 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Summary Random variable (RV): a variable (uppercase) that takes on values (lowercase) from a domain of mutually exclusive and exhaustive values A=a: a proposition, world state, event, effect, etc. abbreviate: P(A=true) to P(a) abbreviate: P(A=false) to P(Øa) abbreviate: P(A=value) to P(value) abbreviate: P(A¹value) to P(Øvalue) Atomic event: a complete specification of the state of the world about which the agent is uncertain 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Summary P(a): a prior probability of RV A=a which is the degree of belief proposition a in absence of any other relevant information P(a|e): conditional probability of RV A=a given E=e which is the degree of belief in proposition a when all that is known is evidence e P(A): probability distribution, i.e. set of P(ai) for all i Joint probabilities are for conjunctions of propositions 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.

©2001-2004 James D. Skrentny from notes by C. Dyer, et. al. Summary Autonomous agents are involved in a cycle of: sensing estimating the state of the world reasoning, planning making decisions and taking actions Probability allows the agent to represent uncertainty about the world agent can assign probabilities to world states agent can assign utilities to action-state pairs Optimal decision-making is choosing the action with the optimal expected utility 2/24/2019 ©2001-2004 James D. Skrentny from notes by C. Dyer, et. al.