Multiplicative Bounds for Metric Labeling M. Pawan Kumar École Centrale Paris Joint work with Phil Torr, Daphne Koller

Metric Labeling Variables V = { V 1, V 2, …, V n }

Metric Labeling Variables V = { V 1, V 2, …, V n }

Metric Labeling VaVa VbVb Labels L = { l 1, l 2, …, l h } Variables V = { V 1, V 2, …, V n } Labeling f: { 1, 2, …, n}  {1, 2, …, h} E(f) = Σ a θ a (f(a)) + Σ (a,b) w ab d(f(a),f(b)) min f θ a (f(a)) θ b (f(b)) w ab d(f(a),f(b)) w ab ≥ 0 d is metric

Metric Labeling VaVa VbVb E(f) min f NP hard = Σ a θ a (f(a)) + Σ (a,b) w ab d(f(a),f(b)) Low-level vision applications

 Minka. Expectation Propagation for Approximate Bayesian Inference, UAI, 2001  Murphy et al. Loopy Belief Propagation: An Empirical Study, UAI, 1999  Winn et al. Variational Message Passing, JMLR, 2005  Yedidia et al. Generalized Belief Propagation, NIPS, 2001  Besag. On the Statistical Analysis of Dirty Pictures, JRSS, 1986  Boykov et al. Fast Approximate Energy Minimization via Graph Cuts, PAMI, 2001  Komodakis et al. Fast, Approximately Optimal Solutions for Single and Dynamic MRFs, CVPR, 2007  Lempitsky et al. Fusion Moves for Markov Random Field Optimization, PAMI, 2010  Chekuri et al. Approximation Algorithms for Metric Labeling, SODA, 2001  Goemans et al. Improved Approximate Algorithms for Maximum-Cut, JACM, 1995  Muramatsu et al. A New SOCP Relaxation for Max-Cut, JORJ, 2003  Ravikumar et al. QP Relaxations for Metric Labeling, ICML, 2006  Alahari et al. Dynamic Hybrid Algorithms for MAP Inference, PAMI 2010  Kohli et al. On Partial Optimality in Multilabel MRFs, ICML, 2008  Rother et al. Optimizing Binary MRFs via Extended Roof Duality, CVPR, Approximate Algorithms

Outline Linear Programming Relaxation Move-Making Algorithms Comparison Rounding-based Moves

Integer Linear Program Number of facets grows exponentially in problem size Minimize a linear function over a set of feasible solutions Indicator x a (i)  {0,1} for each variable V a and label l i

Linear Programming Relaxation Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003 Indicator x a (i)  {0,1} for each variable V a and label l i

Linear Programming Relaxation Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003 Indicator x a (i)  [0,1] for each variable V a and label l i

Outline Linear Programming Relaxation Move-Making Algorithms Comparison Rounding-based Moves

Move-Making Algorithms Space of All Labelings f

Expansion Algorithm Variables take label l α or retain current label Slide courtesy Pushmeet Kohli Boykov, Veksler and Zabih, 2001

Expansion Algorithm Sky House Tree Ground Initialize with TreeStatus:Expand GroundExpand HouseExpand Sky Slide courtesy Pushmeet Kohli Variables take label l α or retain current label Boykov, Veksler and Zabih, 2001

Outline Linear Programming Relaxation Move-Making Algorithms Comparison Rounding-based Moves

Multiplicative Bounds f*: Optimal Labelingf: Estimated Labeling Σ a θ a (f(a)) + Σ (a,b) s ab d(f(a),f(b)) Σ a θ a (f*(a)) + Σ (a,b) s ab d(f*(a),f*(b)) ≥

Multiplicative Bounds f*: Optimal Labelingf: Estimated Labeling ≤ B Σ a θ a (f(a)) + Σ (a,b) s ab d(f(a),f(b)) Σ a θ a (f*(a)) + Σ (a,b) s ab d(f*(a),f*(b))

Multiplicative Bounds ExpansionLP Potts22 Metric2MO(log h) Truncated Linear 2M2 + √2 Truncated Quadratic 2MO(√M) M = ratio of maximum and minimum distance

Outline Linear Programming Relaxation Move-Making Algorithms Comparison Rounding-based Moves –Complete Rounding –Interval Rounding –Hierarchical Rounding

Complete Rounding Treat x a (i)  [0,1] as probability that f(a) = i Cumulative probability y a (i) = Σ j≤i x a (j) 0y a (1) y a (2) y a (h) = 1 r Generate a random number r  (0,1] Assign the label next to r y a (k) y a (i)

Complete Move VaVa VbVb θ ab (i,k) = s ab d(i,k)NP-hard

Complete Move VaVa VbVb θ ab (i,k) = s ab d’(i,k) d’(i,k) ≥ d(i,k) d’ is submodular

Complete Move VaVa VbVb θ ab (i,k) = s ab d’(i,k) d’(i,k) ≥ d(i,k) d’ is submodular

Complete Move New problem can be solved using minimum cut Same multiplicative bound as complete rounding Multiplicative bound is tight

Outline Linear Programming Relaxation Move-Making Algorithms Comparison Rounding-based Moves –Complete Rounding –Interval Rounding –Hierarchical Rounding

Interval Rounding Treat x a (i)  [0,1] as probability that f(a) = i Cumulative probability y a (i) = Σ j≤i x a (j) 0y a (1) y a (2) y a (h) = 1 y a (k) y a (i) Choose an interval of length h’

Interval Rounding Treat x a (i)  [0,1] as probability that f(a) = i Cumulative probability y a (i) = Σ j≤i x a (j) r Generate a random number r  (0,1] Assign the label next to r if it is within the interval y a (k) y a (i) Choose an interval of length h’ REPEAT

Interval Move VaVa VbVb θ ab (i,k) = s ab d(i,k) Choose an interval of length h’

Interval Move VaVa VbVb θ ab (i,k) = s ab d(i,k) Choose an interval of length h’ Add the current labels

Interval Move VaVa VbVb θ ab (i,k) = s ab d’(i,k) Choose an interval of length h’ Add the current labels d’(i,k) ≥ d(i,k) d’ is submodular Solve to update labels Repeat until convergence

Interval Move Each problem can be solved using minimum cut Same multiplicative bound as interval rounding Multiplicative bound is tight Kumar and Torr, NIPS 2008

Outline Linear Programming Relaxation Move-Making Algorithms Comparison Rounding-based Moves –Complete Rounding –Interval Rounding –Hierarchical Rounding

Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Hierarchical clustering of labels (e.g. r-HST metrics)

Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Assign variables to labels L 1, L 2 or L 3 Move down the hierarchy until the leaf level

Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Assign variables to labels l 1, l 2 or l 3

Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Assign variables to labels l 4, l 5 or l 6

Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Assign variables to labels l 7, l 8 or l 9

Hierarchical Move L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Hierarchical clustering of labels (e.g. r-HST metrics)

Hierarchical Move L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Obtain labeling f 1 restricted to labels {l 1,l 2,l 3 }

Hierarchical Move L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Obtain labeling f 2 restricted to labels {l 4,l 5,l 6 }

Hierarchical Move L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Obtain labeling f 3 restricted to labels {l 7,l 8,l 9 }

Hierarchical Move L1L1 L2L2 L3L3 VaVa VbVb f 1 (a) f 2 (a) f 3 (a) Move up the hierarchy until we reach the root f 1 (b) f 2 (b) f 3 (b)

Hierarchical Move Each problem can be solved using minimum cut Same multiplicative bound as hierarchical rounding Multiplicative bound is tight Kumar and Koller, UAI 2009

Conclusion Move- Making LP Potts22 MetricO(log h) Truncated Linear 2 + √2 Truncated Quadratic O(√M) M = ratio of maximum and minimum distance

Open Problems Moves for general rounding schemes Higher-order energy functions Better comparison criterion

Questions?