1. Intelligent Environments1 Computer Science and Engineering University of Texas at Arlington

2. Intelligent Environments2 Decision-Making for Intelligent Environments Motivation Techniques Issues

3. Intelligent Environments3 Motivation An intelligent environment acquires and applies knowledge about you and your surroundings in order to improve your experience. “acquires”  prediction “applies”  decision making

4. Intelligent Environments4 Motivation Why do we need decision-making? “Improve our experience” Usually alternative actions Which one to take? Example (Bob scenario: bedroom  ?) Turn on bathroom light? Turn on kitchen light? Turn off bedroom light?

5. Intelligent Environments5 Example Should I turn on the bathroom light? Issues Inhabitant’s location (current and future) Inhabitant’s task Inhabitant’s preferences Energy efficiency Security Other inhabitants

6. Intelligent Environments6 Qualities of a Decision Maker Ideal Complete: always makes a decision Correct: decision is always right Natural: knowledge easily expressed Efficient Rational Decisions made to maximize performance

7. Intelligent Environments7 Agent-based Decision Maker Russell & Norvig “AI: A Modern Approach” Rational agent Agent chooses an action to maximize its performance based on percept sequence

8. Intelligent Environments8 Agent Types Reflex agent Reflex agent with state Goal-based agent Utility-based agent

9. Intelligent Environments9 Reflex Agent

10. Intelligent Environments10 Reflex Agent with State

11. Intelligent Environments11 Goal-based Agent

12. Intelligent Environments12 Utility-based Agent

13. Intelligent Environments13 Intelligent Environments Decision-Making Techniques

14. Intelligent Environments14 Decision-Making Techniques Logic Planning Decision theory Markov decision process Reinforcement learning

15. Intelligent Environments15 Logical Decision Making If Equal(?Day,Monday) & GreaterThan(?CurrentTime,0600) & LessThan(?CurrentTime,0700) & Location(Bob,bedroom,?CurrentTime) & Increment(?CurrentTime,?NextTime) Then Location(Bob,bathroom,?NextTime) Query: Location(Bob,?Room,0800)

16. Intelligent Environments16 Logical Decision Making Rules and facts First-order predicate logic Inference mechanism Deduction: {A, A  B}  B Systems Prolog (PROgramming in LOGic) OTTER Theorem Prover

17. Intelligent Environments17 Prolog location(bob,bathroom,NextTime) :- dayofweek(Day), Day = monday, currenttime(CurrentTime), CurrentTime > 0600, CurrentTime < 0700, location(bob,bedroom,CurrentTime), increment(CurrentTime,NextTime). Facts: dayofweek(monday),... Query: location(bob,Room,0800).

18. Intelligent Environments18 OTTER (all d all t1 all t2 ((DayofWeek(d) & Equal(d,Monday) & CurrentTime(t1) & GreaterThan(t1,0600) & LessThan(t1,0700) & NextTime(t1,t2) & Location(Bob,Bedroom,t1)) -> Location(Bob,Bathroom,t2))). Facts: DayofWeek(Monday),... Query: ( exists r (Location(Bob,r,0800)))

19. Intelligent Environments19 Actions If Location(Bob,Bathroom,t1) Then Action(TurnOnBathRoomLight,t1) Preferences among actions If RecommendedAction(a1,t1) & RecommendedAction(a2,t1) & ActionPriority(a1) > ActionPriority(a2) Then Action(a1,t1)

20. Intelligent Environments20 Persistence Over Time If Location(Bob,room1,t1) & not Move(Bob,t1) & NextTime(t1,t2) Then Location(Bob,room1,t2) One for each attribute of Bob!

21. Intelligent Environments21 Logical Decision Making Assessment Complete? Yes Correct? Yes Efficient? No Natural? No Rational?

22. Intelligent Environments22 Decision Making as Planning Search for a sequence of actions to achieve some goal Requires Initial state of the environment Goal state Actions (operators) Conditions Effects (implied connection to effectors)

23. Intelligent Environments23 Example Initial: location(Bob,Bathroom) & light(Bathroom,off) Goal: happy(Bob) Action 1 Condition: location(Bob,?r) & light(?r,on) Effect: Add: happy(Bob) Action 2 Condition: light(?r,off) Effect: Delete: light(?r,off), Add: light(?r,on) Plan: Action 2, Action 1

24. Intelligent Environments24 Requirements Where do goals come from? System design Users Where do actions come from? Device “drivers” Learned macros E.g., SecureHome action

25. Intelligent Environments25 Planning Systems UCPOP (Univ. of Washington) Partial Order Planner with Universal quanitification and Conditional effects GraphPlan (CMU) Builds and prunes graph of possible plans

26. Intelligent Environments26 GraphPlan Example (:action lighton :parameters (?r) :precondition (light ?r off)) :effects (and (light ?r on) (not (light ?r off))))

27. Intelligent Environments27 Planning Assessment Complete? Yes Correct? Yes Efficient? No Natural? Better Rational?

28. Intelligent Environments28 Decision Theory Logical and planning approaches typically assume no uncertainty Decision theory = probability theory + utility theory Maximum Expected Utility principle Rational agent chooses actions yielding highest expected utility Averaged over all possible action outcomes Weight utility of an outcome by its probability of occurring

29. Intelligent Environments29 Probability Theory Random variables: X, Y, … Prior probability: P(X) Conditional probability: P(X|Y) Joint probability distribution P(X 1,…,X n ) is an n-dimensional table of probabilities Complete table allows computation of any probability Complete table typically infeasible

30. Intelligent Environments30 Probability Theory Bayes rule Example More likely to know P(wet|rain) In general, P(X|Y) =  * P(Y|X) * P(Y)  chosen so that  P(X|Y) = 1

31. Intelligent Environments31 Bayes Rule (cont.) How to compute P(rain|wet & thunder) P(r | w & t) = P(w & t | r) * P(r) / P(w & t) Know P(w & t | r) possibly, but tedious as evidence increases Conditional independence of evidence Thunder does not cause wet, and vice versa P(r | w & t) =  * P(w|r) * P(t|r) * P(r)

32. Intelligent Environments32 Where Do Probabilities Come From? Statistical sampling Universal principles Individual beliefs

33. Intelligent Environments33 Representation of Uncertain Knowledge Complete joint probability distribution Conditional probabilities and Bayes rule Assuming conditional independence Belief networks

34. Intelligent Environments34 Belief Networks Nodes represent random variables Directed link between X and Y implies that X “directly influences” Y Each node has a conditional probability table (CPT) quantifying the effects that the parents (incoming links) have on the node Network is a DAG (no directed cycles)

35. Intelligent Environments35 Belief Networks: Example

36. Intelligent Environments36 Belief Networks: Semantics Network represents the joint probability distribution Network encodes conditional independence knowledge Node conditionally independent of all other nodes except parents E.g., MaryCalls and Earthquake are conditionally independent

37. Intelligent Environments37 Belief Networks: Inference Given network, compute P(Query | Evidence) Evidence obtained from sensory percepts Possible inferences Diagnostic: P(Burglary | JohnCalls) = 0.016 Causal: P(JohnCalls | Burglary) P(Burglary | Alarm & Earthquake)

38. Intelligent Environments38 Belief Network Construction Choose variables Discretize continuous variables Order variables from causes to effects CPTs Specify each table entry Define as a function (e.g., sum, Gaussian) Learning Variables (evidential and hidden) Links (causation) CPTs

39. Intelligent Environments39 Combining Beliefs with Desires Maximum expected utility Rational agent chooses action maximizing expected utility Expected utility EU(A|E) of action A given evidence E EU(A|E) =  i P(Result i (A) | E, Do(A)) * U(Result i (A)) Result i (A) are possible outcome states after executing action A U(S) is the agent’s utility for state S Do(A) is the proposition that action A is executed in the current state

40. Intelligent Environments40 Maximum Expected Utility Assumptions Knowing evidence E completely requires significant sensory information P(Result | E, Do(A)) requires complete causal model of the environment U(Result) requires complete specification of state utilities One-shot vs. sequential decisions

41. Intelligent Environments41 Utility Theory Any set of preferences over possible outcomes can be expressed by a utility function Lottery L = [p 1,S 1 ; p 2,S 2 ;...; p n,S n ] p i is the probability of possible outcome S i S i can be another lottery Utility principle U(A) > U(B)  A preferred to B U(A) = U(B)  agent indifferent to A and B Maximum expected utility principle U([p 1,S 1 ; p 2,S 2 ;...; p n,S n ]) =  i p i * U(S i )

42. Intelligent Environments42 Utility Functions Possible outcomes [1.0, $1000; 0.0, $0] [0.5, $3000; 0.5, $0] Expected monetary value $1000 vs. $1500 But depends on value $k S k = state of possessing wealth $k EU(accept) = 0.5 * U(S k +3000) + 0.5 * U(S k ) EU(decline) = U(S k +1000) Will decline for some values of U, accept for others

43. Intelligent Environments43 Utility Functions (cont.) Studies show U(S k +n) = log 2 n Risk-adverse agents in positive part of curve Risk-seeking agents in negative part of curve

44. Intelligent Environments44 Decision Networks Also called influence diagrams Decision networks = belief networks + actions and utilities Describes agent’s Current state Possible actions State resulting from agent’s action Utility of resulting state

45. Intelligent Environments45 Example Decision Network

46. Intelligent Environments46 Decision Network Chance node (oval) Random variable and CPT Same as belief network node Decision node (rectangle) Can take on a value for each possible action Utility node (diamond) Parents are those chance nodes affecting utility Contains utility function mapping parents to utility value or lottery

47. Intelligent Environments47 Evaluating Decision Networks Set evidence variables according to current state For each action value of decision node Set value of decision node to action Use belief-net inference to calculate posteriors for parents of utility node Calculate utility for action Return action with highest utility

48. Intelligent Environments48 Sequential Decision Problems No intermediate utility on the way to the goal Transition model Probability of reaching state j after taking action a in state i Policy = complete mapping from states to actions Want policy maximizing expected utility Computed from transition model and state utilities

49. Intelligent Environments49 Example P(intended direction) = 0.8 P(right angle to intended) = 0.1 U(sequence) = terminal state’s value - (1/25)*length(sequence)

50. Intelligent Environments50 Example (cont.) Optimal PolicyUtilities

51. Intelligent Environments51 Markov Decision Process (MDP) Calculating optimal policy in fully- observable, stochastic environment with known transition model Markov property satisfied depends only on i and not previous states Partially-observable environments addressed by POMDPs

52. Intelligent Environments52 Value Iteration for MDPs Iterate the following for each state i until little change R(i) is the reward for entering state i -0.04 for all states except (4,3) and (4,2) +1 for (4,3) -1 for (4,2) Best policy policy*(i) is

53. Intelligent Environments53 Reinforcement Learning Basically MDP, but learns policy without the need for transition model Q-learning with temporal difference Assigns values Q(a,i) to action-state pairs Utility U(i) = max a Q(a,i) Update Q(a,i) after each observed transition from state i to state j Q(a,i) = Q(a,i) +  * (R(i) + max a’ Q(a’,j) - Q(a,i)) action in state i = argmax a Q(a,i)

54. Intelligent Environments54 Decision-Theoretic Agent Given Percept (sensor) information Maintain Decision network with beliefs, actions and utilities Do Update probabilities for current state Compute outcome probabilities for actions Select action with highest expected utility Return action

55. Intelligent Environments55 Decision-Theoretic Agent Modeling sensors

56. Intelligent Environments56 Sensor Modeling Combining evidence from multiple sensors

57. Intelligent Environments57 Sensor Modeling Detailed model of lane-position sensor

58. Intelligent Environments58 Dynamic Belief Network (DBN) Reasoning over time Big for lots of states But really only need two slices at a time

59. Intelligent Environments59 Dynamic Belief Network (DBN)

60. Intelligent Environments60 DBN for Lane Positioning

61. Intelligent Environments61 Dynamic Decision Network (DDN)

62. Intelligent Environments62 DDN-based Agent Capabilities Handles uncertainty Handles unexpected events (no fixed plan) Handles noisy and failed sensors Acts to obtain relevant information Needs Properties from first-order logic DDNs are propositional Goal directedness

63. Intelligent Environments63 Decision-Theoretic Agent Assessment Complete? No Correct? No Efficient? Better Natural? Yes Rational? Yes

64. Intelligent Environments64 Netica www.norsys.com Decision network simulator Chance nodes Decision nodes Utility nodes Learns probabilities from cases

65. Intelligent Environments65 Bob Scenario in Netica

66. Intelligent Environments66 Issues in Decision Making Rational agent design Dynamic decision-theoretic agent Knowledge engineering effort Efficiency vs. completeness Monolithic vs. distributed intelligence Degrees of autonomy