Multiplicative Bounds for Metric Labeling M. Pawan Kumar École Centrale Paris École des Ponts ParisTech INRIA Saclay, Île-de-France Joint work with Phil Torr, Daphne Koller

Energy Minimization Variables V = { V 1, V 2, …, V n }

Energy Minimization Variables V = { V 1, V 2, …, V n }

Energy Minimization VaVa VbVb Labels L = { l 1, l 2, …, l h } Variables V = { V 1, V 2, …, V n } Labeling f: { 1, 2, …, n}  {1, 2, …, h} E(f) = Σ a θ a (f(a)) + Σ (a,b) θ ab (f(a),f(b)) min f θ a (f(a)) θ b (f(b)) θ ab (f(a),f(b))

Energy Minimization VaVa VbVb E(f) min f NP hard = Σ a θ a (f(a)) + Σ (a,b) θ ab (f(a),f(b))

Metric Labeling VaVa VbVb E(f) min f = Σ a θ a (f(a)) + Σ (a,b) θ ab (f(a),f(b))

Metric Labeling VaVa VbVb s ab is non-negative d(.) is a metric distance function Low-level vision applications NP hard E(f) min f = Σ a θ a (f(a)) + Σ (a,b) s ab d(f(a),f(b))

 Minka. Expectation Propagation for Approximate Bayesian Inference, UAI, 2001  Murphy et al. Loopy Belief Propagation: An Empirical Study, UAI, 1999  Winn et al. Variational Message Passing, JMLR, 2005  Yedidia et al. Generalized Belief Propagation, NIPS, 2001  Besag. On the Statistical Analysis of Dirty Pictures, JRSS, 1986  Boykov et al. Fast Approximate Energy Minimization via Graph Cuts, PAMI, 2001  Komodakis et al. Fast, Approximately Optimal Solutions for Single and Dynamic MRFs, CVPR, 2007  Lempitsky et al. Fusion Moves for Markov Random Field Optimization, PAMI, 2010  Chekuri et al. Approximation Algorithms for Metric Labeling, SODA, 2001  Goemans et al. Improved Approximate Algorithms for Maximum-Cut, JACM, 1995  Muramatsu et al. A New SOCP Relaxation for Max-Cut, JORJ, 2003  Ravikumar et al. QP Relaxations for Metric Labeling, ICML, 2006  Alahari et al. Dynamic Hybrid Algorithms for MAP Inference, PAMI 2010  Kohli et al. On Partial Optimality in Multilabel MRFs, ICML, 2008  Rother et al. Optimizing Binary MRFs via Extended Roof Duality, CVPR, Approximate Algorithms

Multiplicative Bounds f*: Optimal Labelingf: Estimated Labeling Σ a θ a (f(a)) + Σ (a,b) s ab d(f(a),f(b)) Σ a θ a (f*(a)) + Σ (a,b) s ab d(f*(a),f*(b)) ≥

Multiplicative Bounds f*: Optimal Labelingf: Estimated Labeling ≤ M Σ a θ a (f(a)) + Σ (a,b) s ab d(f(a),f(b)) Σ a θ a (f*(a)) + Σ (a,b) s ab d(f*(a),f*(b))

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models

Integer Linear Program Number of facets grows exponentially in problem size Minimize a linear function over a set of feasible solutions

Linear Programming Relaxation Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003

Conic Programming Relaxation Muramatsu and Suzuki, 2003; Ravikumar and Lafferty, 2006

Convex Relaxations Time LP 1976 SOCP 2003 QP 2006 Tightness Expected Analyzed Kumar, Kolmogorov and Torr, NIPS 2007

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models

Move-Making Algorithms Space of All Labelings f

Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {l α } Update f = f α Boykov, Veksler and Zabih, 2001 Repeat until convergence

Expansion Algorithm Variables take label l α or retain current label Slide courtesy Pushmeet Kohli

Expansion Algorithm Sky House Tree Ground Initialize with TreeStatus:Expand GroundExpand HouseExpand Sky Slide courtesy Pushmeet Kohli Variables take label l α or retain current label

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models

Multiplicative Bounds ExpansionLP Potts2M2 d max = max i≠ k d(i,k)d min = min i≠ k d(i,k) M = d max d min

Multiplicative Bounds ExpansionLP Potts22 Metric2MO(log h) h = number of putative labels

Multiplicative Bounds ExpansionLP Potts22 Metric2MO(log h) Truncated Linear 2M2 + √2 Truncated Quadratic 2MO(√M)

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models Kumar and Koller, UAI 2009

Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {α} Update f = f α Repeat until convergence Boykov, Veksler and Zabih, 2001

Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {α} Update f = f α Repeat until convergence

Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {M α (a)} Update f = f α Repeat until convergence Any label M α (a) instead of the same label α

Modified Expansion Algorithm Multiplicative Bound = 2 d max = max i,k d(i,k) d max d min i = M α (a), k = M β (b), α ≠ β d min = min i,k d(i,k) i = M α (a), k = M β (b), α ≠ β

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Visualizing Metrics Uniform Metric Hierarchically Separated Tree (HST) Metrics General Metrics Move-Making for Truncated Convex Models

Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l3 w1w1 w2w2 w3w3 w4w4 w5w5 w6w6 w7w7 w9w9 w8w8 d( i, j ) : shortest path defined by the graph

Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l d( i, j ) : shortest path defined by the graph

Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l d( i, j ) : shortest path defined by the graph d(1,4) = 3

Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l d( i, j ) : shortest path defined by the graph d(1,2) = 5

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Visualizing Metrics Uniform Metric Hierarchically Separated Tree (HST) Metrics General Metrics Move-Making for Truncated Convex Models

Uniform Metric l1l1 l2l2 l3l3 w w w

Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {M α (a)} Update f = f α Repeat until convergence Any label M α (a) instead of the same label α

Uniform Metric l1l1 l2l2 l3l3 w w w M α (a) = l α for all random variables V a Multiplicative Bound = 2 d max = max i,k d(i,k) d max d min i = M α (a), k = M β (b), α ≠ β d min = min i,k d(i,k) i = M α (a), k = M β (b), α ≠ β

Uniform Metric l1l1 l2l2 l3l3 w w w M α (a) = l α for all random variables V a Multiplicative Bound = 2 d max = 2w d max d min d min = min i,k d(i,k) i = M α (a), k = M β (b), α ≠ β

Uniform Metric l1l1 l2l2 l3l3 w w w M α (a) = l α for all random variables V a Multiplicative Bound = 2 d max = 2w d max d min d min = 2w

Uniform Metric l1l1 l2l2 l3l3 w w w M α (a) = l α for all random variables V a Multiplicative Bound = 2 d max = 2w d min = 2w Same bound as LP

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Visualizing Metrics Uniform Metric Hierarchically Separated Tree (HST) Metrics General Metrics Move-Making for Truncated Convex Models

HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 Graph is a Tree. Labels are leaves

HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 Edge lengths for all children are the same

HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 Edge lengths decrease from root to leaf by factor r ≥ 2

HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 w 2 ≤ w 1 /r w 3 ≤ w 1 /r w 4 ≤ w 1 /r

HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 w 5 ≤ w 2 /r w 6 ≤ w 2 /r w 7 ≤ w 3 /rw 8 ≤ w 3 /r

Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4

l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 M α (a) = l α

Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 Multiplicative bound of 2 for (a,b) where f*(a), f*(b)  {1,2,3}

Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 Multiplicative bound of 2 for (a,b) where f*(a), f*(b)  {4,5,6} f2f2

Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 Multiplicative bound of 2 for (a,b) where f*(a), f*(b)  {7,8,9} f2f2 f3f3

Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 f2f2 f3f3 M 1 (a) = f 1 (a)M 2 (a) = f 2 (a) M 3 (a) = f 3 (a)

Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 f2f2 f3f3 Multiplicative bound of at most 2(1+1/r)

Metric Labeling for 4-level HST Multiplicative bound of at most 2(1+1/r + 1/r 2 ) w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8

Metric Labeling for L-level HST Multiplicative bound = 2(1+1/r + … + 1/r L-2 ) < 2r/(r-1) Constant bound for all HST metrics

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Visualizing Metrics Uniform Metric Hierarchically Separated Tree (HST) Metrics General Metrics Move-Making for Truncated Convex Models

General Metrics Any metric can be approximated using a mixture of O(h log h) HST metrics with distortion O(log h). Fakcharoenphol, Rao and Talwar, 2003 Σ T ρ(T)d T (i,j) Σ T ρ(T) = 1 d(i,j) ≤ ≤ O(log h) d(i,j)

Metric Labeling for General Metrics Approximate d using Σ T ρ(T)d T Solve metric labeling for each HST d T Pick the labeling with the lowest energy Multiplicative bound = 2r/(r-1)*O(log h) = O(log h) Same bound as LP

Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models Kumar and Torr, NIPS 2008

Truncated Convex Models VaVa VbVb s ab is non-negative d(.) is a convex function E(f) min f = Σ a θ a (f(a)) + Σ (a,b) s ab min{d(f(a),f(b)), M}

Truncated Convex Models s ab is non-negative d(.) is a convex function Low-level vision applications NP hard E(f) min f = Σ a θ a (f(a)) + Σ (a,b) s ab min{d(f(a),f(b)), M} Truncated Linear Truncated Quadratic

Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {M α (a)} Update f = f α Repeat until convergence Any label M α (a) instead of the same label l α

Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {M α (a)} Update f = f α Repeat until convergence Any interval of labels M α (a) instead of a single label

Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {α, α+1, …, α+L} Update f = f α Repeat until convergence Any interval of labels M α (a) instead of a single label

Length of the Interval Submodular problem Exact optimization in polynomial time Small interval implies bigger ratio of d max /d min

Length of the Interval Non-submodular problem

Length of the Interval Submodular problem Optimization of upper bound in polynomial time Big interval implies smaller ratio of d max /d min

Length of the Interval Truncated Linear Bound = 2 + max 2M, L LM L = √2M Bound = 2 + √2 Truncated Quadratic Bound = O(√M)L = √M

Conclusion Move Making LP Potts22 MetricO(log h) Truncated Linear 2 + √2 Truncated Quadratic O(√M)

Future Work Move-making and convex relaxations connection Better measures than multiplicative bounds Designing hybrid algorithms

Questions?