Learning Equivalence Classes of Bayesian-Network Structures David M. Chickering Presented by Dmitry Zinenko

Heuristic Search  We are looking for the best state in the search space. Na ï vely: state = a particular DAG search space = all possible DAGs over our variables  Move between related states using search operators. Naively: Egde addition/removal/inversion

Heuristic Search Challenges  Search space graph should be well- connected To reach good states quickly To avoid local maxima  Search space graph should not be too dense Computationally efficient scoring and transformations

Equivalence  G 1 and G 2 are equivalent if the set of distributions that can be represented by them is identical  Equivalence is an equivalence relationship! XY XY XY P

Score Equivalence  If all we care about is the probability distribution, all we need is the equivalence class  The scoring function should give equal scores to structures from the same class Called score equivalent  Why prefer one representation of the class to another?

Equivalence Classes Are Good For You  We are ultimately looking for a probability representation, not a particular DAG  Searching individual DAGs is bad: Some operators lead to the same class  Efficiency  Bad state connectivity for greedy

Theorem 1 (Verma & Pearl 1990)  Two DAGs are equivalent if and only if they have the same skeletons and the same v-structures X Y X Y Z X Y Z Z X Y Z

Partially Directed Acyclic Graph  A directed edge is called compelled in G, if for every G ’ equivalent to G, that edge has the same direction  Otherwise we call it reversible  Partially Directed Acyclic Graph (PDAG) Contains both directed and undirected edges Does not contain any directed circles  Theorem 1 extends naturally to PDAGs A DAG is also a PDAG

CPDAG and Consistent Extension  Completed PDAG for Class(G) contains directed edges for the compelled edges of G undirected edges for the reversible edges of G  G is consistent extension of P if G has the same skeleton and v-structures Every directed edge in P has the same orientation in G XYZXYZXYWZ

CPDAGs And Equivalence  Every consistent extension of P is in Class(P)  If P c is a completed PDAG, then every PDAG G in Class(P c ) is a consistent extension of P c  If P 1 and P 2 are completed PDAGs that admit consistent extension, then P 1 =P 2 if and only if Class(P 1 )=Class(P 2 ) A completed PDAG uniquely represents its equivalence class

DAG to CPDAG (Meek 1995)  Undirect all edges except those that are in the v-structures  Direct (mark as compelled) undirected edges that match particular patterns X Y ZX Y Z X Y Z W

Constructing Consistent Extension (I)  “ Theorem 26 ” : The undirected components of a CPDAG are chordal In any cycle of length >3 in a DAG, there must be a v-structure! Let {K i } be the set of undirected components of a completed PDAG P c. Let {G i } be consistent extensions of {K i } A graph G that results from replacing each reversible edge in K i with the directed edge from corresponding G i is a consistent extension of P c

Constructing Consistent Extension (II)  Use decreasing maximum cardinality search to direct edges in each one of the chordal components Property of dMCS: Every path between any pair of non-adjacent x, y contains a node numbered higher than x or y  Resulting graph is a consistent extension of P c  Works only on completed PDAGs

PDAG-to-DAG (Dor & Tarsi 1992)  Select a node x in P s.t. x has no outgoing edges Vertices adjacent to x form a clique  Direct all edges (x―y) toward x x becomes a sink  Remove x from P  Works only on any PDAG

Applying the Operators

Operators  The set of operators should: Ensure global connectivity (completeness) and good connectivity in general Be easy to check for applicability (validity) Avoid redundancy Allow for efficient scoring  Local scoring – local changes in G cause “ local ” changes in score(G)

Score Decomposability  A scoring function S is decomposable if it is a product (or sum) of factors s, each depending only on one node and its parents  For example: XY XYZ Z

Used Operators

Operator Scoring  Chickering 1996a Apply the operator and score the consistent extension (DAG)  Drawbacks: Need to apply PDAG-to-DAG for every operator Local operators may cause non-local changes when applied to CPDAG  Cannot benefit from local scoring

Local Operator Scoring

InsertU Operator – “Theorem 34”  Let P c be any completed PDAG for which nodes x and y are not adjacent.  If after adding an edge between x and y P c admits a consistent extension, then  The edge x―y is reversible if and only if x and y have exactly the same parents in the original PDAG

InsertU Operator – “Theorem 6”  The insertion of the undirected edge x―y in a CPDAG P c is valid if and only if: x and y have the same parents in P c every undirected path between x and y contains at least one of their common neighbors  Only if (+Theorem 34): Take the shortest undirected path from x to y in P c that does not include any common neighbor of x and y  Length at least 3 and has no chord  After adding x―y becomes a cycle of length 4

InsertU Operator – “Lemma 32”  Let P c be any completed PDAG, and let x and y be any pair of nodes that are not adjacent.  There exists a consistent extension of P c in which all the reversible edges adjacent to x are directed away from x all the reversible edges between y and the common neighbors of x and y are directed toward y all the other reversible edges adjacent to y are directed away from y  If and only if every undirected path between x and y passes through a common neighbor of x and y

InsertU Operator – Theorem 6 “If” proof outline  Use consistent extension from Lemma 32 as G  Add a directed edge x → y to G to get G ’ (the other direction is symmetric)  Show that G ’ is a consistent extension of P ’ (P with the addition of the undirected edge x―y) G ’ is acyclic Same skeleton Same v-structures

InsertU Operator – Theorem 6 G’ is a DAG  Assume by contradiction that there is a directed path from y to x in G  All the reversible edges are directed away from x, so the last edge in that path w → x is compelled  Then w is a parent of x in P, and it must also be a parent of y  In G there is a cycle y → w → y XY W

InsertU Operator – “Lemma 24”  Let P c be a completed PDAG, and let P ’ denote a PDAG that results from adding a single edge between x and y to P c  Consider any consistent extension G of P c, and G ’ that results by inserting a directed edge between x and y in G  Then any v-structure in G ’ but not in P ’, or any v-structure in P ’ but not in G ’ must include the edge between x and y

InsertU Operator – Theorem 6 G’ is a consistent extension of P’  By Lemma 24, any v-structure different between G ’ and P ’ must include the edge x―y  The v-structure must be in G ’, because in P ’ this edge is undirected  The other edge in the v-structure cannot be reversible in G ’ x does not have reversible parents y ’ s reversible parents are adjacent to x  But any compelled parent of x or y is a parent of both Q.E.D

Local Operator Evaluation  Since the only difference between G and G ’ is the edge x → y, we can use score decomposability to compute the score of P ’ in O(1) time s(P ’ ) = s(P c )+s(y,N x,y {x} y )-s(y,N x,y  y )  In general we do not need to transform the CPDAG to compute neighbor scores: Calculate scores for all the neighbor states (locally!) Check operator validity (efficiently!) starting from the highest score