1. Semantically-Linked Bayesian Networks: A Framework for Probabilistic Inference Over Multiple Bayesian Networks PhD Dissertation Defense Advisor: Dr. Yun Peng Rong Pan Department of Computer Science and Electrical Engineering University of Maryland Baltimore County Aug 2, 2006

2. Outline ► Motivations ► Background ► Overview ► How Knowledge is Shared ► Inference on SLBN ► Concept Mapping using SLBN ► Future works

3. Motivations (1) ► Separately developed BNs about  related domains  different aspects of the same domain …

4. Motivations (2) ► Existing approach:  Multiply Sectioned Bayesian Networks (MSBN)  Every subnet is sectioned from a global BN  Strictly consistent subnets  Exactly identical shared variables with same distribution  All parents of the shared variables must appear in one subnet Sectioning

5. Motivations (3) ► Existing approach:  Agent Encapsulated Bayesian Networks (AEBN)  Distribution BN Model for a specific application  Hierarchical global structure  Very restricted expressiveness  Exactly identical shared variables with different prior distributions Agent Output Variable Input Variable Local Variable

6. Motivations (4) ► A distributed BN model was expected with features:  Uncertainty reasoning over separately developed BNs  Variables shared by different BNs can be similar but not identical  Principled, well justified  Support various applications

7. Background Bayesian Network ► DAG ► Variables  with Finite States ► Edges: causal influences ► Conditional Probability Table (CPT)

8. Background Evidences in BN Original BNHard Evidence: Male_Mammal = True Soft Evidence: Q(Male_Mammal) = (0.5 0.5) Virtual Evidence: L(Male_Mammal) = 0.8/0.2 Virtual Evidence = Soft Evidence: L(Male_Mammal) = 0.3/0.2

9. Background Jeffrey’s Rule (Soft Evidence) ► Given external observations Q(B i ), the rest of the BN is updated by Jeffrey’s Rule: where P(A| B i ) is the conditional probability before evidence, Q(B i ) is the soft evidence. where P(A| B i ) is the conditional probability before evidence, Q(B i ) is the soft evidence. ► Multiple Soft Evidences  Problem: update one variable’s distribution to its target value can make those of others’ off their targets  Solution: IPFP

10. Background Iterative Proportional Fitting Procedure (IPFP) ► Q 0 : initial distribution on the set of variables X, ► {P(S i )}: a consistent set of n marginal probability distributions, where X  Si  . ► The IPFP process where i is the iteration number, j = (i-1) mod n + 1 ► The distribution after IPFP satisfies the given constraints {P(S i )} and has minimum cross-entropy to the initial distribution Q 0

11. SLBN: Overview (1) ► Semantically-Linked Bayesian Networks (SLBN)  A theoretical framework that supports probabilistic inference over separately developed BNs Global Knowledge Similar variables

12. SLBN: Overview (2) ► Features  Inference over separate BNs that share semantically similar variables  Global knowledge: J-graph  Principled, well-justified ► In SLBN  BNs are linked at the similar variables  Probabilistic influences are propagated via the shared variables  Inference process utilizes Soft Evidence (Jeffrey’s Rule), Virtual Evidence, IPFP, and traditional BN inference

13. How knowledge is shared: Semantic Similarity (1) What is similarity? Similar: Pronunciation: 'si-m&-l&r, 'sim-l&r Function: adjective 1: having characteristics in common 2: alike in substance or essentials 3: not differing in shape but only in size or position –– www.merrian-webster.com www.merrian-webster.com High-tech Company Employee V.S. High-income People Computer Keyboard V.S. Typewriter

14. How knowledge is shared: Semantic Similarity (2) ► Semantic Similarity of concepts  Share of common instances  Quantified and utilized with direction  Quantified by the ratio of the shared instances to all the instances  Natural language’s definition for “similar” is vague  Hard to formalize  Hard to quantify  Hard to utilize in intelligence Conditional Probability P(High-tech Company Employee | High-income People)

15. Man V.S. Woman How knowledge is shared: Variable Linkage (1) ► In Bayesian Network (BN) / SLBN  Concepts are represented by variables  Semantic similarities are between propositions We say “High-tech Company Employee” is similar to “High-income People” We mean “High-tech Company Employee = True” is similar to “High-income People = True”

16. How knowledge is shared: Variable Linkage (2) ► Variable linkages  Represent semantic similarities in SLBN  Are between variables in different BNs A : Source Variable B : Destination Variable N A : Source BN N B : Destination BN : Quantification of the similarity is a m × n matrix:

17. How knowledge is shared: Variable Linkage (3) ► Variable Linkage V.S. BN Edge Variable Linkage BN Edge Representation Of Semantic Similarity Causal Influences Conditional Probability Quantification of Similarity; Invariant w.r.t. any event Conditional dependency; May be changed by events Prob. Influence Propagation Along the direction Both directions ( -msg. and  -msg.) ( -msg. and  -msg.)

18. How knowledge is shared: Variable Linkage (4) ► Expressiveness of Variable Linkage  Logical relationships defined in OWL syntax: Equivalent, Union, Intersection, and Subclass complement.  Relaxation of logical relationships by replacing set inclusion by overlapping: Overlap, Superclass, Subclass  Equivalence relations but same concepts are modeled as different variables

19. How knowledge is shared: Examples (1) … … Identical Union

20. How knowledge is shared: Examples (2) Overlap Superclass …

21. How knowledge is shared: Consistent Linked Variables ► The priori beliefs on the linked variables on both sides must be consistent with the variable linkage:  P 2 (B) = ∑ i P S (B|A=a i )P 1 (A=a i )  There exists a single distribution consistent with the prior belief on A, B,  A,  B, and the linkage’s similarity. ► examined by IPFP AB AA BB P 1 (  A ) P 1 (A|  A ) P 1 (A) P 2 (  B ) P 2 (B|  A ) P 2 (B) P S (B| A)

22. Inference on SLBN The Process 1. Enter Evidence 2. Propagate 4. Updated Result 3. Enter Soft/Virtual Evidences; BN Belief Update With traditional Inference BN Belief Update With traditional Inference SLBN Rules for Probabilistic Influence Propagation SLBN Rules for Probabilistic Influence Propagation BN Belief Update with Soft Evidence BN Belief Update with Soft Evidence

23. Inference on SLBN The Theory Bayes’ RuleJeffrey’s RuleIPFP Soft Evidence BN Inference Virtual Evidence SLBN Theoretical Basis Implementation (Existing) Implementation (SLBN)

24. Inference on SLBN Assumptions/Restrictions ► All linked BNs are consistent with the linkages ► One variable can only be involved in one linkage ► Causal precedence in all linked BNs are consistent Linked BNs with inconsistent causal sequences Linked BNs with consistent causal sequences

25. Inference on SLBN Assumptions/Restrictions (Cont.) ► For a variable linkage, the causes/effects of source is also the causes/effects of the destination  Linkages cannot cross each other … … …... Crossed linkages

26. Inference on SLBN Inference on SLBN SLBN Rules for Probabilistic Influence Propagation (1) ► Some hard evidence influence the source from bottom Y1 Y1 Y2Y2 Y3 Y3 … … X1 X1 ► Propagated influences are represented by soft evidences ► Beliefs of destination BN are update with SE

27. Inference on SLBN Inference on SLBN SLBN Rules for Probabilistic Influence Propagation (2) ► Some hard evidence influence the source from top Y1 Y1 Y2Y2 Y3 Y3 … … X1 X1 ► Additional soft evidences are created to cancel the influences from the linkage to parent(dest(L))

28. Inference on SLBN Inference on SLBN SLBN Rules for Probabilistic Influence Propagation (3) ► Some hard evidence influence the source from both top and bottom Y1 Y1 Y2Y2 Y3 Y3 … … X1 X1 ► Additional soft evidences are created to propagate the combined influences from the linkage to parent(dest(L))

29. Inference on SLBN Belief Update with Soft Evidence (1) ► Represent soft evidences by virtual evidences  Belief update with soft evidence is IPFP  Belief update with one virtual evidence is one step of IPFP ► Therefore, we can  Use virtual evidence to implement IPFP on BN  Use virtual evidence to implement soft evidence ► SE VE  Iterate on the whole BN  Iterate on soft evidence variables

30. Inference on SLBN Belief Update with Soft Evidence (2) ► Iterate on whole BN AB (0.3, 0.7) (0.8, 0.2) (0.6, 0.4) (0.66, 0.34) (0.47, 0.53) (0.5, 0.5) (0.6, 0.4) (0.60, 0.40) (0.55, 0.45) (0.5, 0.5) …………. (0.6, 0.4) (0.5, 0.5) Q(B) = (0.5, 0.5) Q(A) = (0.6, 0.4) ve AB

31. Inference on SLBN Belief Update with Soft Evidence (1) ► Iterate on SE variables Q(B) = (0.5, 0.5)Q(A) = (0.6, 0.4) ve AB A = t A = f B = t 0.20.6 B = f 0.10.1 P(A, B) = IPFP with Q(A), Q(B) A = t A = f B = t 0.2360.264 B = f 0.3640.136 Q(A, B) =

32. Inference on SLBN Belief Update with Soft Evidence (3) ► Existing approaches : Big-Clique Big-Clique Iteration on whole BN Iteration on se variables BN Inference Basis RewriteJunction-Tree Wrapper of any method Time for each Iteration O(e |C| ) O(BN Inf.) O(e |V| ) Space O(e |C| ) O(|V|) O(e |V| ) C: the big clique V: se variables |C|≥|V| Iteration on whole BN: Small BNs, many soft evidences Iteration on se variables: Large BNs, a few soft evidences

33. J-Graph (1) Overview ► Joint-graph (J-graph) is a graphical probability model that represents  The joint distribution of SLBN  The interdependencies between variables across variable linkages ► Usage  Check if all assumptions are satisfied  Justify Inference Process

34. J-Graph (2) Definition ► J-Graph is constructed by merging all linked BNs and linkages into one graph  DAG  Variable nodes, Linkage Nodes  Edges: all edges in the linked BNs have a representation in J-graph  CPT: Q(A|  A ) = P(A|  A ), Q(A|B) = P S (A|B) for ► Q: distribution in J-graph, P: original distribution

35. J-Graph (3) Example A BC D A’ B’ C’ D’ A1A1 B→B’; 1→2 C→C’; 1→2 D2D2 A’ 2 D’ 2  Linkage nodes  represent all linked variables and the linkage  encode the similarity of the linkage in CPT  merge the CPTs by IPFP Linkage Node

36. Concept Mapping using SLBN (1) Motivations ► Ontology mappings are seldom certain  Existing approaches ► use hard threshold to filter mappings ► throw similarities away after mappings are created ► mappings are identical and 1-1  But ► often one concept is similar to more than one concept ► Semantically similar concepts are hard to be represented logically

37. Probabilistic Information Learner BayesOWL Concept Mapping using SLBN (2) The Framework Onto 2 Onto 1 WWW BN 2 BN 1 Variable Linkages BayesOWL SLBN

38. Concept Mapping using SLBN (3) Objection ► Discover new and complex concept mappings  Make full use of the learned similarity in SLBN’s inference  Create an expression for a concept in another ontology ► Find how similar “Onto1:B  Onto1:C” is to “Onto2:A” ► Experiments have shown encouraging results

39. Concept Mapping using SLBN (3) Experiment ► Artificial Intelligence sub-domain from ACM Topic Taxonomy DMOZ (Open Directory) hierarchies Learned Similarities: J(dmoz.sw, acm.rs) = 0.64 J(dmoz.sw, acm.sn) = 0.61 J(dmoz.sw, acm.krfm) = 0.49 After SLBN Inference: J(dmoz.sw, acm.rs  acm.sn) = 0.7250 Q (acm.rs = True  acm.sn = True | dmoz.sw = True) = 0.9646

40. Future Works ► Modeling with SLBN  Discover semantic similar concepts by machine learning algorithms  Create effective and correct linkages from learned algorithms ► Distributed Inference methods ► Loosing the restrictions  Inference with linkages of both directions  Use functions to represent similarities

41. Thank You! ► Questions?

44. Background Semantics of BN ► Chain rule where  (a i ) is the parent set of a i. ► d-separation: B CA CA B A B C serials diverging converging Instantiated Not instantiated d-separated variables do not influence each other.