Recent Development on Elimination Ordering Group 1

Recent Development on Elimination Ordering By Florence Lo Andrew Yan

Table of Contents Problem Statement Background Motivations Approaches to Elimination Orderings An approximation algorithm for triangulation Q & A

Problem Statement Minimizing Total Cost in Vertices Elimination

Background BN specifies a complete joint probability distribution (JPD) over all variables. Given the JPD, one can answer all possible inference queries by summing out over irrelevant variables. Assumed all variables have 2 states and there are n variables => O(2 n )

Background (cont) To make inference more efficient, 2 exact inference algorithms are used 1) Variable Elimination Algorithm (VE) 2) Clique Tree Propagation (CTP)

Variable Elimination (VE) Use the factored representation of the JPD to do marginalization efficiently. The key idea is to “push sum in” as far as possible when summing out irrelevant terms Complexity of VE is based on the cost of variables elimination (i.e. elimination orderings)

Clique Tree Propagation (CTP) Inference in BN is formulated as message passing in a junction tree First step in CTP is triangulated graph, one way is to eliminate vertices one by one to add the extra edges (i.e. elimination orderings)

Discussion Different elimination ordering leads to different cost NP-hard

3 Approaches to Establish Elimination Orderings 1. Elimination Ordering Heuristics 2. Triangulation 3. Simulated Annealing

Elimination Ordering Heuristics Maximum Cardinality Search (Tarjan et. al. 1985) Minimum Deficiency Heuristics (Bertele et. al. 1972) Minimum Degree Heuristics (Rose 1972)

Discussions Easy to implement Concepts are easy Linear time complexity Good approximation to the optimal solution

Triangulation Objectives 1. Size of the maximal clique is minimum 2. Minimal triangulation

Minimal Triangulation Computing a minimal triangulation consists in embedding a given graph with a triangulated graph by adding a set of edges (called a fill). If the set of edges added is inclusion-minimal, the fill is said to be minimal, and the corresponding triangulated graph is called a minimal triangulation

Discussion Minimal triangulation generally improves the total cost May stuck in the local minima of the cost function NP-hard

Simulated Annealing Stochastic optimization algorithm to find global minimum cost configuration of NP- complete combinatorial problems with cost functions having many local minima A combination of deterministic descent search and a Monte Carlo method Accept cost function increase with positive probability dependent on the state of the search process

Pseudo -code Select an initial solution s 0 Select an initial temperature t 0 > 0 Select a temperature reduction function α Repeat repeat randomly select s from N(s 0 ) δ = f(s) – f(s 0 ) If δ < 0 then s 0 = s else generate random x uniformly in the range (0, 1) if x < exp(- δ/t) then s 0 = s until iteration_count = nrep Set t = α(t) Until stopping condition = true S 0 is the approximation to the optimal solution

2 Categories of Decisions to make 1. Parameters of the annealing algorithm - t0, cooling schedule governed by nrep and α, stopping conditions 2. Problem specific and involve the choice of the space of feasible solutions, the form of the cost function, neighborhood structure employed

Discussion Time consuming Extensive experiments on the control parameters Enhancement and modifications to speed up computation time

An approximation algorithm for triangulation By Ann Becker and Dan Geiger in 1996 Same state space size Optimality criterion – Cliquewidth, k Cliquewidth 2  +1 – approximation –  = approximation ratio for 3-way vertex cut3-way vertex cut O(2 (2  +1)k npoly(n)) – poly(n) = linear programming Divide and conquer

The algorithm Triangulate(G,W,k) If |V| < (2  +1)K then make a clique out of G Else Find a W-decomposition (X,A,B,C) of G wrt (k,  )W-decomposition If not found Return “cliquewidth > k” W A  W  A, W B  W  B, W C  W  C Call Triangulate( G[A  X], W A,k) Call Triangulate( G[B  X], W B,k) Call Triangulate( G[C  X], W C,k) Make a clique of G[W  X]

Trial and error Try for k = 1, 2, 3… until success!!!

Example (k=3,  =1)

Improvement Processing the input – Simplicial vertex – Removed repeatedly – Improves the running time

Improvement Termination condition – |V| < (2  +1)k  from a clique – Junction tree instead – W as a clique – Approximation may improved

Improvement Post-processing the output – May not minimal triangulated – Remove some added edges, still triangulated – Kjaerulff’s algorithm

In practice Time complexity O(2 (2  +1)k npoly(n)) |W| < k in most case – k  2 2k W consists of two subsets – No 3-way vertex cut Step further for k Find a W-decomposition (X,A,B,C) of G wrt (k,  ) If not found Return “cliquewidth > k”

A tighter bound l be the size of the largest clique in the output Test against (2  +1)k – The ratio can be smaller than 2  +1 – Optimal cliquewidth can be larger than k l/k instead of 2  +1 Instance-specific posteriori bound

The weighted problem Different state spaces of each vertex – w(v) = log 2 (state space size) – w(clique) = sum of w(v) in the clique Weighted W-decomposition When terminate in recursion, run greedy algorithm (minimum weight heuristics) (2  +1)m

Results Medianus I (43 vertices, 110 edges) Compare with the enhanced minimum weight heuristics – Better when state space increase l/k = 10/6 (not 3.66) Run time: one or two minutes

Discussion O(2 4.66k npoly(n)) – Polynomial for k = O(log n) Exponential time for an arbitrary inference

What is cliquewidth? Undirected graph The size of the largest clique in the junction free of the graph in which the size of the largest clique is minimized back

What is 3-way vertex cut? A weighted undirected graph 3 vertices A set of vertices of minimum weight whose removal  three vertices disconnected 4/3 – approximation 2 – approximation back

What is W-decomposition? An integer k  1, a real number   1 A graph G = ( V, E ), |V|  ( 2  + 1 ) k W  V A decomposition ( X, A, B, C ) wrt ( k,  )decomposition – |W| < (  + 1 )k – |X| <  k – |(W  A)  X| < (  + 1 )k – |(W  B)  X| < (  + 1 )k – |(W  C)  X| < (  + 1 )k back

What is decomposition? A graph G = ( V, E ) A partition ( X, A, B, C ) – A, B   – No edges between A, B, C back