1. THE HONG KONG UNIVERSITY OF SCIENCE & TECHNOLOGY CSIT 5220: Reasoning and Decision under Uncertainty L09: Graphical Models for Decision Problems Nevin L. Zhang Room 3504, phone: 2358-7015, Email: lzhang@cs.ust.hk Home pagelzhang@cs.ust.hkHome page

2. CSIT 5220 L10: Graphical Models for Decision Problems l Introduction l Extending BN to Include a Single Decision l Fundamentals of Rational Decision Making l Decision Trees l Influence Diagrams l Solving influence Diagrams l Value of information Page 2

3. CSIT 5220 Probabilistic Reasoning and Decision l Method 1: Two-stage n In a BN, calculate posterior probabilities n Use the posteriors to make decisions l Method 2 n Combine the two stages n Extend BN to include decisions  Better reveal structure of decision problem  Compute optimal decisions directly from model l Reasoning: Jensen & Nielsen, Sections 9.1-9.4, 10.2, 11.1 Page 3

4. CSIT 5220 L10: Graphical Models for Decision Problems l Extending BN to Include a Single Decision l Fundamentals of Rational Decision Making l Decision Trees l Influence Diagrams l Solving influence Diagrams l Value of information Page 4

5. CSIT 5220 Poker l From Lecture 04 Page 5 l Extend the model so that I can calculate the probability that my hand is better than the opponent’s hand l MH: My Hand l BH: Best Hand

6. CSIT 5220 Fold or Call Page 6

7. CSIT 5220 Fold or Call l Information that I have: FC, SC, MH Page 7

8. CSIT 5220 Modeling One Action l Start with a BN l Add the decision node and utility nodes n What information we have when making the decision n What chance and utility variables will the decision influence Page 8

9. CSIT 5220 Including More Decisions l Things become a bit more complicated. l Will see later. Page 9

10. CSIT 5220 L10: Graphical Models for Decision Problems l Extending BN to Include Decisions l Fundamentals of Rational Decision Making l Decision Trees l Influence Diagrams l Solving influence Diagrams l Value of information Page 10

11. CSIT 5220 Decision Theory l Normative decision theory n How people should decide. (Rational agent) l Descriptive decision theory n How people actually decide. Page 11

12. CSIT 5220 Normative Decision Theory Page 12

13. CSIT 5220 Are you rational? l Lottery A: [$1mill] l Lottery B: 0.5[$2mill] + 0.5[$0mill] l Which one do you choose? l Most people would choose A U(1) > 0.5 U(2) + 0.5 U(0) l Most people are risk-averse, with concave utility function Page 13

14. CSIT 5220 Are your rational? l Suppose that you are $2mill in debt Page 14 l Lottery A: [$1mill] l Lottery B: 0.5[$2mill] + 0.5[$0mill] l Which one do you choose? l Probably B U(1) < 0.5 U(2) + 0.5 U(0) l You are being risk-seeking, with convex utility function

15. CSIT 5220 Utilities without Money Page 15

16. CSIT 5220 Utilities without Money Page 16

17. CSIT 5220 Marks as Utilities Page 17

18. CSIT 5220 Other Considerations l 2 is passing grade l If fail, can retake and hopefully get a better grade in transcript n In this case, 2 is the worst! Page 18

19. CSIT 5220 L10: Graphical Models for Decision Problems l Extending BN to Include Decisions l Fundamentals of Rational Decision Making l Decision Trees l Influence Diagrams l Solving influence Diagrams l Value of information Page 19

20. CSIT 5220 Decision Trees l Classical way to represent decision problems with multiple decisions l Explicitly show all possible sequences of decisions and observations. l Example: Oil Wildcatter Page 20 A wildcatter is a person who drills wildcat wells, which are oil wells drilled in areas not known to be oil fields.drills wildcat wellsoil wellsoil fields Test on Seismic structure

21. CSIT 5220 Decision Tree for Oil Wildcatter Page 21

22. CSIT 5220 Decision Trees l Decision nodes: Rectangles l Chance nodes: ellipses l Utility values: at leaves, some times inside diamonds l To be read from root to leaves n Branches from a decision node: possible actions n Branches from a chance node: possible outcomes and probs n A decision node follows a chance node:  The chance node is observed before the decision is made n No-forgetting  Decision-maker remembers all the labels from root to a decision node l Game between decision maker and nature Page 22

23. CSIT 5220 Solution to a Decision Tree l Strategy: Which decision node to pick at each decision node Page 23

24. CSIT 5220 Solution to a Decision Tree l Optimal Strategy: The strategy with the highest expected utility Page 24

25. CSIT 5220 Solving Decision Trees Page 25

26. CSIT 5220 Example Page 26 77.59

27. CSIT 5220 Page 27

28. CSIT 5220 L10: Graphical Models for Decision Problems l Extending BN to Include Decisions l Fundamentals of Rational Decision Making l Decision Trees l Influence Diagrams l Solving influence Diagrams l Value of information Page 28

29. CSIT 5220 Extending BN to Including one Decision Page 29 l Start with a BN l Add the decision node and utility nodes n What information we have when making the decision n What chance and utility variables will the decision influence l To include multiple decision nodes, n Need to consider the interactions among the decisions

30. CSIT 5220 Including Multiple Decisions l Two more decisions n MFC: my first change n MSC: my second change Page 30

31. CSIT 5220 Representing the Decision Sequence l First representation n All nodes observed before a decision are parents of that decision. n Information arcs. Page 31 l Assume that the decision maker doesn’t forget, then some links are redundant.

32. CSIT 5220 Representing the Decision Sequence l No-forgetting allows a more concise representation n Keep directed path going through all the decision node: Order of decision. n Arrows into a decision node only from those nodes observed immediately before that decision. n Implicit parents: parents of earlier decisions Page 32

33. CSIT 5220 Influence Diagram l A DAG with three types of nodes n Chance nodes, decision nodes, and utility nodes l There is a directed path containing all the decision nodes. l The utility nodes have no children. l Each chance node is associated with the conditional distribution given its parents. l Each utility node is associated with a utility function, a real-valued function of its parents. Page 33

34. CSIT 5220 Influence Diagram Page 34

35. CSIT 5220 l An influence diagram for the oil wildcatter problem n Decision: T: test = {y, n}; D: drill={y, n} n Utility: C: cost of test ; V: Benefit of drilling n Chance:  O: Oil ={dry, wet, soaking}  R: seismic structure {no-structure, open-structure, closed-structure, no-result} Influence Diagram Page 35

36. CSIT 5220 L10: Graphical Models for Decision Problems l Extending BN to Include Decisions l Fundamentals of Rational Decision Making l Decision Trees l Influence Diagrams l Solving influence Diagrams l Value of information Page 36

37. CSIT 5220 Strategy (Policy) l A policy specifies what to do for each decision l It is a function of observed variables Page 37 l Different policies lead to different expected utility l Optimal policy: the Policy that yields the maximum expected utility. l How to find the optimal policy?

38. CSIT 5220 Finding Optimal Policy l First idea: n Convert to decision tree and solve it l How to convert influence diagram into decision tree 1. Draw tree nRoot: the thing that happens first nChildren of root: the thing that happens next n…n… 2. Figure out numerical information Page 38

39. CSIT 5220 l Order of events l Tree structure l Numerical info n Prob for branches from chance node n Utility for leaves

40. CSIT 5220 A Side Note l Two decision trees for Oil Wildcatter n First  directly from problem specification.  Asymmetric n Second  from influence diagram  Symmetric l Pro of ID: compact l Con of ID: cannot represent assymetry n Need to introduce artificial state R = no-result

41. CSIT 5220 Finding Optimal Policy l First idea: n Convert to decision tree and solve it l Exponential still! l Next: n Variable Elimination Algorithm for solving influence diagrams n Note n BN inference: All orderings give correct result, but might have different complexity n ID: Must use “strong elimination orderings”. Page 41

42. CSIT 5220 Temporal Order and Decisions andd Observations l Notations n Decision nodes have a temporal order: D 1, D 2, …, D n n T 0 : Set of chance nodes observed prior to any decision n T i : Set of chance nodes observed after D i is taken and before D i+1 is taken l Oil Wildcatter n D 1 = T; D 2 = D n T 0 = {}; T 1 = {R}; T 2 ={O} l Partial temporal order n T 0, D 1, T 1, D 2, T 2, …., D n, T n n Oil Wildcatter: T, R, D, O Page 42

43. CSIT 5220 Temporal Order l T 0 ={}, T 1 ={T}, T 2 ={A, B, C} l Partial temporal ordering n D 1, T, D 2. {A, B, C} n No ordering among A, B, C Page 43

44. CSIT 5220 Strong Elimination Ordering l Partial temporal order n T 0, D 1, T 1, D 2, T 2, …., D n, T n l Strong elimination orders n First eliminate variables in T n n Then eliminate D n n Then eliminate variables in T n-1 n Then eliminate D n-1 n ….. l Oil Wildcatter n Temporal order:  T, R, D, O n Strong elimination ordering  O, D, R, T Page 44

45. CSIT 5220 Strong Elimination Ordering l T 0 ={}, T 1 ={T}, T 2 ={A, B, C} l Partial temporal ordering n D 1, T, D 2. {A, B, C} n No ordering among A, B, C l Strong elimination orderings n A, B, C, D 2, T, D1 n B, C, A, D 2, T, D 1 n C, A, B, D 2, T, D 1 n …. Page 45

46. CSIT 5220 Variable Elimination l Two set of potentials (factors): l Eliminate decision and chance nodes one by one according to a strong elimination ordering. l When eliminate variable X Page 46

47. CSIT 5220 Variable Elimination on Oil Wildcatter l Strong Elimination Ordering: O, D, R, T Page 47

48. CSIT 5220 l Eliminate: O Page 48 Variable Elimination on Oil Wildcatter

49. CSIT 5220 Page 49

50. CSIT 5220 Page 50

51. CSIT 5220 Potentials after Eliminating O Page 51

52. CSIT 5220 Potentials after Eliminating O Page 52

53. CSIT 5220 Eliminating D l No probability potential involves D Page 53 l Optimal decision for D

54. CSIT 5220 Potentials after Eliminating D Page 54

55. CSIT 5220 Eliminating R Page 55

56. CSIT 5220 Potentials after Eliminating R Page 56

57. CSIT 5220 Eliminating T Page 57 l Optimal decision for T l Results same as those by decision tree

58. CSIT 5220 Solving Influence Diagram Using Netica

59. CSIT 5220 Solving Influence Diagram Using Netica l Netica cannot handle multiple utility l So, combine U and V

60. CSIT 5220 Solving Influence Diagram Using Netica

61. CSIT 5220 l Or, Get optimal action by trying each actions

62. CSIT 5220 L10: Graphical Models for Decision Problems l Extending BN to Include Decisions l Fundamentals of Rational Decision Making l Decision Trees l Influence Diagrams l Solving influence Diagrams l Value of information Page 62

63. CSIT 5220 Two types of Decisions l Action decisions n Result in significant state change of variables of interest n Example:  D: Drill or not to drill l Test decisions n Look for more evidence n Example:  T: Test of Seismic structure Page 63

64. CSIT 5220 Two types of Decisions l Typical scenario n Need to make one decision n Want to get more information before making the decision n Question  Is it worthwhile to perform a particular test?  Which test to choose if multiple tests are available? Page 64

65. CSIT 5220 Value of Information l What is the value of a test? n Create two influence diagrams n Solve both n Compare their values l Example: Oil wildcatter n Is it worthwhile to perform the seismic test? n ID1: without the test n ID2: with the test Page 65

66. CSIT 5220 Value of Information l Expected utility of ID2 n U(ID2) = 22.55 l What is the expected utility of ID1? Page 66

67. CSIT 5220 Expected Utility of ID1 l Temporal ordering: D, O l Elimination ordering: O, D l Eliminate O:

68. CSIT 5220 Expected Utility of ID1 l Potentials after eliminating O l Eliminate D l Expected utility of ID1 n U(ID1) = 20 Page 68

69. CSIT 5220 Value of Information l Difference in expected utility n U(ID2) – U(ID1) = 22.55 – 20 = 2.55 n The expected value of the seismic test is 2.55 n The test is worthwhile Page 69

70. CSIT 5220 Value of Information l If there are multiple tests n T1, T2, T3, … n Compute the value of each test, pick the best one n If the value of the best is positive,  Pick the test among remain tests n Stop when value of the selected test is not positive Page 70