Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Introduction to Graphical Models Part 2 of 2 Friday, 04 November 2005 William H. Hsu Laboratory for Knowledge Discovery in Databases Department of Computing and Information Sciences Kansas State University This presentation is based upon: Lecture 30 of 42

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Graphical Models Overview [1]: Bayesian Networks P(20s, Female, Low, Non-Smoker, No-Cancer, Negative, Negative) = P(T) · P(F) · P(L | T) · P(N | T, F) · P(N | L, N) · P(N | N) · P(N | N) Conditional Independence –X is conditionally independent (CI) from Y given Z (sometimes written X  Y | Z) iff P(X | Y, Z) = P(X | Z) for all values of X, Y, and Z –Example: P(Thunder | Rain, Lightning) = P(Thunder | Lightning)  T  R | L Bayesian (Belief) Network –Acyclic directed graph model B = (V, E,  ) representing CI assertions over  –Vertices (nodes) V: denote events (each a random variable) –Edges (arcs, links) E: denote conditional dependencies Markov Condition for BBNs (Chain Rule): Example BBN X1X1 X3X3 X4X4 X5X5 Age Exposure-To-Toxins Smoking Cancer X6X6 Serum Calcium X2X2 Gender X7X7 Lung Tumor

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Adapted from slides by S. Russell, UC Berkeley Fusion –Methods for combining multiple beliefs –Theory more precise than for fuzzy, ANN inference –Data and sensor fusion –Resolving conflict (vote-taking, winner-take-all, mixture estimation) –Paraconsistent reasoning Propagation –Modeling process of evidential reasoning by updating beliefs –Source of parallelism –Natural object-oriented (message-passing) model –Communication: asynchronous –dynamic workpool management problem –Concurrency: known Petri net dualities Structuring –Learning graphical dependencies from scores, constraints –Two parameter estimation problems: structure learning, belief revision Fusion, Propagation, and Structuring

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Bayesian Learning Framework: Interpretations of Probability [Cheeseman, 1985] –Bayesian subjectivist view A measure of an agent’s belief in a proposition Proposition denoted by random variable (sample space: range) e.g., Pr(Outlook = Sunny) = 0.8 –Frequentist view: probability is the frequency of observations of an event –Logicist view: probability is inferential evidence in favor of a proposition Typical Applications –HCI: learning natural language; intelligent displays; decision support –Approaches: prediction; sensor and data fusion (e.g., bioinformatics) Prediction: Examples –Measure relevant parameters: temperature, barometric pressure, wind speed –Make statement of the form Pr(Tomorrow’s-Weather = Rain) = 0.5 –College admissions: Pr(Acceptance)  p Plain beliefs: unconditional acceptance (p = 1) or categorical rejection (p = 0) Conditional beliefs: depends on reviewer (use probabilistic model)

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Choosing Hypotheses Bayes’s Theorem MAP Hypothesis –Generally want most probable hypothesis given the training data –Define:  the value of x in the sample space  with the highest f(x) –Maximum a posteriori hypothesis, h MAP ML Hypothesis –Assume that p(h i ) = p(h j ) for all pairs i, j (uniform priors, i.e., P H ~ Uniform) –Can further simplify and choose the maximum likelihood hypothesis, h ML

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Propagation Algorithm in Singly-Connected Bayesian Networks – Pearl (1983) C1C1 C2C2 C3C3 C4C4 C5C5 C6C6 Upward (child-to- parent) messages  ’ (C i ’ ) modified during message-passing phase Downward  messages P ’ (C i ’ ) is computed during  message-passing phase Adapted from Neapolitan (1990), Guo (2000) Multiply-connected case: exact, approximate inference are #P-complete (counting problem is #P-complete iff decision problem is NP-complete)

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Inference by Clustering [1]: Graph Operations (Moralization, Triangulation, Maximal Cliques) Adapted from Neapolitan (1990), Guo (2000) A D BE G C H F Bayesian Network (Acyclic Digraph) A D BE G C H F Moralize A1A1 D8D8 B2B2 E3E3 G5G5 C4C4 H7H7 F6F6 Triangulate Clq6 D8D8 C4C4 G5G5 H7H7 C4C4 Clq5 G5G5 F6F6 E3E3 Clq4 G5G5 E3E3 C4C4 Clq3 A1A1 B2B2 Clq1 E3E3 C4C4 B2B2 Clq2 Find Maximal Cliques

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Inference by Clustering [2]: Junction Tree – Lauritzen & Spiegelhalter (1988) Input: list of cliques of triangulated, moralized graph G u Output: Tree of cliques Separators nodes S i, Residual nodes R i and potential probability  (Clq i ) for all cliques Algorithm: 1. S i = Clq i  (Clq 1  Clq 2  …  Clq i-1 ) 2. R i = Clq i - S i 3. If i >1 then identify a j < i such that Clq j is a parent of Clq i 4. Assign each node v to a unique clique Clq i that v  c(v)  Clq i 5. Compute  (Clq i ) =  f(v) Clqi = P(v | c(v)) {1 if no v is assigned to Clq i } 6. Store Clq i, R i, S i, and  (Clq i ) at each vertex in the tree of cliques Adapted from Neapolitan (1990), Guo (2000)

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Inference by Clustering [3]: Clique-Tree Operations Clq6 D8D8 C4C4G5G5 H7H7 C4C4 Clq5 G5G5 F6F6 E3E3 Clq4 G5G5 E3E3 C4C4 Clq3 A1A1 B2B2 Clq1 E3E3 C4C4 B2B2 Clq2  (Clq5) = P(H|C,G)  (Clq2) = P(D|C) Clq 1 Clq3 = {E,C,G} R3 = {G} S3 = { E,C } Clq1 = {A, B} R1 = {A, B} S1 = {} Clq2 = {B,E,C} R2 = {C,E} S2 = { B } Clq4 = {E, G, F} R4 = {F} S4 = { E,G } Clq5 = {C, G,H} R5 = {H} S5 = { C,G } Clq6 = {C, D} R5 = {D} S5 = { C}  (Clq 1 ) = P(B|A)P(A)  (Clq2) = P(C|B,E)  (Clq3) = 1  (Clq4) = P(E|F)P(G|F)P(F) AB BEC ECG EGF CGH CD B EC CGEG C R i : residual nodes S i : separator nodes  (Clq i ): potential probability of Clique i Clq 2 Clq 3 Clq 4 Clq 5 Clq 6 Adapted from Neapolitan (1990), Guo (2000)

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Inference by Loop Cutset Conditioning Split vertex in undirected cycle; condition upon each of its state values Number of network instantiations: Product of arity of nodes in minimal loop cutset Posterior: marginal conditioned upon cutset variable values X3X3 X4X4 X5X5 Exposure-To- Toxins Smoking Cancer X6X6 Serum Calcium X2X2 Gender X7X7 Lung Tumor X 1,1 Age = [0, 10) X 1,2 Age = [10, 20) X 1,10 Age = [100,  ) Deciding Optimal Cutset: NP-hard Current Open Problems –Bounded cutset conditioning: ordering heuristics –Finding randomized algorithms for loop cutset optimization

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Inference by Variable Elimination [1]: Intuition Adapted from slides by S. Russell, UC Berkeley

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Inference by Variable Elimination [2]: Factoring Operations Adapted from slides by S. Russell, UC Berkeley

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Inference by Variable Elimination [3]: Example A BC F G Season Sprinkler Rain Wet Slippery D Manual Watering P(A|G=1) = ? d = G D F B C A λ G (f) = Σ G=1 P(G|F) P(A), P(B|A), P(C|A), P(D|B,A), P(F|B,C), P(G|F) P(G|F) P(D|B,A) P(F|B,C) P(B|A) P(C|A) P(A) G=1 Adapted from Dechter (1996), Joehanes (2002)

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence Tools for Building Graphical Models Commercial Tools: Ergo, Netica, TETRAD, Hugin Bayes Net Toolbox (BNT) – Murphy (1997-present) –Distribution page –Development group Bayesian Network tools in Java (BNJ) – Hsu et al. (1999-present) –Distribution page –Development group –Current (re)implementation projects for KSU KDD Lab Continuous state: Minka (2002) – Hsu, Guo, Perry, Boddhireddy Formats: XML BNIF (MSBN), Netica – Guo, Hsu Space-efficient DBN inference – Joehanes Bounded cutset conditioning – Chandak

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence References [1]: Graphical Models and Inference Algorithms Graphical Models –Bayesian (Belief) Networks tutorial – Murphy (2001) –Learning Bayesian Networks – Heckerman (1996, 1999) Inference Algorithms –Junction Tree (Join Tree, L-S, Hugin): Lauritzen & Spiegelhalter (1988) –(Bounded) Loop Cutset Conditioning: Horvitz & Cooper (1989) –Variable Elimination (Bucket Elimination, ElimBel): Dechter (1986) –Recommended Books Neapolitan (1990, 2003); see Pearl (1988), Jensen (2001) Castillo, Gutierrez, Hadi (1997) Cowell, Dawid, Lauritzen, Spiegelhalter (1999) –Stochastic Approximation

Kansas State University Department of Computing and Information Sciences CIS 730: Introduction to Artificial Intelligence References [2]: Machine Learning, KDD, and Bioinformatics Machine Learning, Data Mining, and Knowledge Discovery –K-State KDD Lab: literature survey and resource catalog (1999-present) –Bayesian Network tools in Java (BNJ): Hsu, Barber, King, Meyer, Thornton (2002-present) –Machine Learning in Java (BNJ): Hsu, Louis, Plummer (2002) Bioinformatics –European Bioinformatics Institute Tutorial: Brazma et al. (2001) –Hebrew University: Friedman, Pe’er, et al. (1999, 2000, 2002) –K-State BMI Group: literature survey and resource catalog ( )