Bayesian Network : An Introduction May 2005 김 진형 KAIST

2 BN = graph theory + probability theory Qualitative part: graph theory Directed acyclic graph Nodes: variables Edges: dependency or influence of a variable on another. Quantitative part: probability theory Set of conditional probabilities for all variables Naturally handles the problem of complexity and uncertainty.

3 Bayesian Network is A framework for representing uncertainty in our knowledge A Graphical modeling framework of causality and influence A Representation of the dependencies among random variables A compact representation of a joint probability of variables on the basis of the concept of conditional independence. Earthquake RadioAlarm Burglar Alarm-Example

4 Bayesian Network Syntax A set of nodes, one per variable A diected, acyclic graph (link = “directly influences”) A conditional distribution for each node given its parents : P(Xi| Parents(Xi)) In the simplist case, conditional distribution represented as a conditional probability table (CPT) giving the distribution over Xi for each combination of parent values

5 Earthquake Example I’m at work, neighbor John calls to say my alarmis ringing, but neighbor Mary doesn’t call. Sometimes it’s set off by minor earthquakes. Is there a burglar ? Variable : Burglar, Earthquake, Alarm, JohnCalls, MaryCalls Network Topology A burglar can set the alarm off An earthquake can set the alarm off The alarm can cause Mary to call The alarm can cause John to call

6 Earthquake Example

7 Representation of Joint Probability Joint probability as a product of conditional probabilities Can dramatically reduce the parameters for data modeling in Bayesian networks. C E D B A

8 Causal Networks Node: event Arc: causal relationship between two nodes A  B: A causes B. Causal network for the car start problem [Jensen 01] Fuel Fuel Meter Standing Start Clean Spark Plugs

9 Reasoning with Causal Networks My car does not start.  increases the certainty of no fuel and dirty spark plugs.  increases the certainty of fuel meter’s standing for the empty. Fuel meter stands for the half.  decreases the certainty of no fuel  increases the certainty of dirty spark plugs. Fuel Fuel Meter Standing Start Clean Spark Plugs

10 Structuring Bayesian Network Initial configuration of Bayesian Network Root nodes Prior probabilities Non-root nodes Conditional probabilities given all possible combinations of direct predecessors AB D E C P(b) P(a) P(d|ab), P(d|a ㄱ b), P(d| ㄱ ab), P(d| ㄱ a ㄱ b) P(e|d) P(e| ㄱ d) P(c|a) P(c| ㄱ a)

11 Structuring Bayesian Network Fuel Fuel Meter Standing Start Clean Spark Plugs P(Fu = Yes) = 0.98P(CSP = Yes) = 0.96 P(St|Fu, CSP) P(FMS|Fu) FMS = Half Fu = No Fu = Yes FMS = Empty FMS = Full 10(No, Yes) (Yes, No) 10(No, No) (Yes, Yes) Start=NoStart=YES(Fu, CSP)

12 Independence assumptions & Complexity Problem of probability theory 2 n -1 joint distributions for n variables For 5 variables, 31 joint distributions Solution by BN For 5 variables, 10 joint distributions Bayesian Networks have built-in independence assumptions. AB D E C P(b) P(a) P(d|ab), P(d|a ㄱ b), P(d| ㄱ ab), P(d| ㄱ a ㄱ b) P(e|d) P(e| ㄱ d) P(c|a) P(c| ㄱ a)

13 Independent Assumptions CAB C AB A and B is marginally dependent CAB C AB A and B is conditionally independent C AB A and B is marginally independent C AB A and B is conditionally dependent

14 Independent Assumption : Car Start Problem 1.‘Start’ and ‘Fuel’ are dependent on each other. 2.‘Start’ and ‘Clean Spark Plugs’ are dependent on each other. 3.‘Fuel’ and ‘Fuel Meter Standing’ are dependent on each other. 4.‘Fuel’ and ‘Clean Spark Plugs’ are conditionally dependent on each other given the value of ‘Start’. 5.‘Fuel Meter Standing’ and ‘Start’ are conditionally independent given the value of ‘Fuel’. Fuel Fuel Meter Standing Start Clean Spark Plugs

15 Quantitative Specification by Probability Calculus Fundamentals Conditional Probability Product Rule Chain Rule: a successive application of the product rule.

16 Main Issues in BN Inference in Bayesian networks Given an assignment of a subset of variables (evidence) in a BN, estimate the posterior distribution over another subset of unobserved variables of interest. Learning Bayesian network from data Parameter Learning Given a data set, estimate local probability distributions P(X i |Pa(X i )). for all variables (nodes) comprising the BN. Structure learning For a data set, search a network structure G (dependency structure) which is best or at least plausible.

17 Evaluating networks Evaluation of network (inference) Computation of all node’s conditional probability given evidence Type of evaluation Exact inference NP-Hard Problem Approximate inference Not exact, but within small distance of the correct answer

18 Inference Task in Bayesian networks

19 Inference in Bayesian networks Joint distribution Definition of joint distribution Set of boolean variables (a,b) P(ab), P( ㄱ ab), P(a ㄱ b), P( ㄱ a ㄱ b) Role of joint distribution Joint distribution give all the information about probability distribution. Ex> P(a|b) = P(ab) / P(b) = P(ab) / ((P(ab)+P( ㄱ ab)) For n random variables, 2 n –1 joint distributions

20 Inference in Bayesian networks Joint distribution for BN is uniquely defined By the product individual distribution of R.V. Using chain-rule, topological sort and dependency ba cd e P(abcde) = P(a)P(b)P(c|a)P(d|ab)P(e|d) Ex)

21 Inference in Bayesian networks Example ba cd e P(a)P(b|a)P(c|ab)P(d|abc)P(e|abcd) Joint probability P(abcde) Chain-rule, Topological sort P(abcde) = P(a)P(b)P(c|a)P(d|ab)P(e|d) Independence assumption b is independent on a,c d is independent on c e is independent on a,b,c

22 Exact inference Two network types Singly connected network (polytree) Multiply connected network Complexity according to network type Singly connected network can be efficiently solved E CD AB Singly connected network A BC D Multiply connected network

23 Inference by enumeration

24 Evaluation Tree

25 Exact inference Multiply Connected Network Hard to evaluate multiply connection network A BC D D will affect C directly D will affect C indirectly p(C|D) ? evidence Probabilities can be affected by both neighbor nodes and other nodes

26 Exact inference Multiply Connected Network (cont.) Methodology to evaluate the network exactly Clustering To Combination of nodes until the resulting graph is singly connected Cloudy WetGrass Spr+Rain CP(S=F) P(S=T) FTFT CP(R=F) P(R=T) FTFT CP(S,R) FF FT TF TT FTFT C W R S

27 Inference by Stochastic Simulation

28 Sampling from an empty network

29 Real World Applications of BN Intelligent agents Microsoft Office assistant: Bayesian user modeling Medical diagnosis PATHFINDER (Heckerman, 1992): diagnosis of lymph node disease  commercialized as INTELLIPATH ( Control decision support system Speech recognition (HMMs) Genome data analysis gene expression, DNA sequence, a combined analysis of heterogeneous data. Turbocodes (channel coding)

30 MSBNx Tools for Baysian Network building tool programmed by Microsoft research Free downloadable Features Graphical Editing of Bayesian Networks Exact Probability Calculations XML Format MSBN3 ActiveX DLL provides an COM-based API for editing and evaluating Bayesian Networks.editingevaluating

31 Conclusion Bayesian Networks are solutions of problems in traditional probability theory Reason of using BN BN need not many numbers Efficient exact solution methods as well as a variety of approximation schemes