1. Multiplicative Bounds for Metric Labeling M. Pawan Kumar École Centrale Paris École des Ponts ParisTech INRIA Saclay, Île-de-France Joint work with Phil Torr, Daphne Koller

2. Energy Minimization Variables V = { V 1, V 2, …, V n }

3. Energy Minimization Variables V = { V 1, V 2, …, V n }

4. Energy Minimization VaVa VbVb Labels L = { l 1, l 2, …, l h } Variables V = { V 1, V 2, …, V n } Labeling f: { 1, 2, …, n}  {1, 2, …, h} E(f) = Σ a θ a (f(a)) + Σ (a,b) θ ab (f(a),f(b)) min f θ a (f(a)) θ b (f(b)) θ ab (f(a),f(b))

5. Energy Minimization VaVa VbVb E(f) min f NP hard = Σ a θ a (f(a)) + Σ (a,b) θ ab (f(a),f(b))

6. Metric Labeling VaVa VbVb E(f) min f = Σ a θ a (f(a)) + Σ (a,b) θ ab (f(a),f(b))

7. Metric Labeling VaVa VbVb s ab is non-negative d(.) is a metric distance function Low-level vision applications NP hard E(f) min f = Σ a θ a (f(a)) + Σ (a,b) s ab d(f(a),f(b))

8.  Minka. Expectation Propagation for Approximate Bayesian Inference, UAI, 2001  Murphy et al. Loopy Belief Propagation: An Empirical Study, UAI, 1999  Winn et al. Variational Message Passing, JMLR, 2005  Yedidia et al. Generalized Belief Propagation, NIPS, 2001  Besag. On the Statistical Analysis of Dirty Pictures, JRSS, 1986  Boykov et al. Fast Approximate Energy Minimization via Graph Cuts, PAMI, 2001  Komodakis et al. Fast, Approximately Optimal Solutions for Single and Dynamic MRFs, CVPR, 2007  Lempitsky et al. Fusion Moves for Markov Random Field Optimization, PAMI, 2010  Chekuri et al. Approximation Algorithms for Metric Labeling, SODA, 2001  Goemans et al. Improved Approximate Algorithms for Maximum-Cut, JACM, 1995  Muramatsu et al. A New SOCP Relaxation for Max-Cut, JORJ, 2003  Ravikumar et al. QP Relaxations for Metric Labeling, ICML, 2006  Alahari et al. Dynamic Hybrid Algorithms for MAP Inference, PAMI 2010  Kohli et al. On Partial Optimality in Multilabel MRFs, ICML, 2008  Rother et al. Optimizing Binary MRFs via Extended Roof Duality, CVPR, 2007...... Approximate Algorithms

9. Multiplicative Bounds f*: Optimal Labelingf: Estimated Labeling Σ a θ a (f(a)) + Σ (a,b) s ab d(f(a),f(b)) Σ a θ a (f*(a)) + Σ (a,b) s ab d(f*(a),f*(b)) ≥

10. Multiplicative Bounds f*: Optimal Labelingf: Estimated Labeling ≤ M Σ a θ a (f(a)) + Σ (a,b) s ab d(f(a),f(b)) Σ a θ a (f*(a)) + Σ (a,b) s ab d(f*(a),f*(b))

11. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models

12. Integer Linear Program Number of facets grows exponentially in problem size Minimize a linear function over a set of feasible solutions

13. Linear Programming Relaxation Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003

14. Conic Programming Relaxation Muramatsu and Suzuki, 2003; Ravikumar and Lafferty, 2006

15. Convex Relaxations Time LP 1976 SOCP 2003 QP 2006 Tightness Expected Analyzed Kumar, Kolmogorov and Torr, NIPS 2007

16. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models

17. Move-Making Algorithms Space of All Labelings f

18. Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {l α } Update f = f α Boykov, Veksler and Zabih, 2001 Repeat until convergence

19. Expansion Algorithm Variables take label l α or retain current label Slide courtesy Pushmeet Kohli

20. Expansion Algorithm Sky House Tree Ground Initialize with TreeStatus:Expand GroundExpand HouseExpand Sky Slide courtesy Pushmeet Kohli Variables take label l α or retain current label

21. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models

22. Multiplicative Bounds ExpansionLP Potts2M2 d max = max i≠ k d(i,k)d min = min i≠ k d(i,k) M = d max d min

23. Multiplicative Bounds ExpansionLP Potts22 Metric2MO(log h) h = number of putative labels

24. Multiplicative Bounds ExpansionLP Potts22 Metric2MO(log h) Truncated Linear 2M2 + √2 Truncated Quadratic 2MO(√M)

25. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models Kumar and Koller, UAI 2009

26. Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {α} Update f = f α Repeat until convergence Boykov, Veksler and Zabih, 2001

27. Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {α} Update f = f α Repeat until convergence

28. Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {M α (a)} Update f = f α Repeat until convergence Any label M α (a) instead of the same label α

29. Modified Expansion Algorithm Multiplicative Bound = 2 d max = max i,k d(i,k) d max d min i = M α (a), k = M β (b), α ≠ β d min = min i,k d(i,k) i = M α (a), k = M β (b), α ≠ β

30. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Visualizing Metrics Uniform Metric Hierarchically Separated Tree (HST) Metrics General Metrics Move-Making for Truncated Convex Models

31. Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l3 w1w1 w2w2 w3w3 w4w4 w5w5 w6w6 w7w7 w9w9 w8w8 d( i, j ) : shortest path defined by the graph

32. Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l3 1 15 1 1 1 1 13 2 d( i, j ) : shortest path defined by the graph

33. Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l3 1 15 1 1 1 1 13 2 d( i, j ) : shortest path defined by the graph d(1,4) = 3

34. Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l3 1 15 1 1 1 1 13 2 d( i, j ) : shortest path defined by the graph d(1,2) = 5

35. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Visualizing Metrics Uniform Metric Hierarchically Separated Tree (HST) Metrics General Metrics Move-Making for Truncated Convex Models

36. Uniform Metric l1l1 l2l2 l3l3 w w w

37. Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {M α (a)} Update f = f α Repeat until convergence Any label M α (a) instead of the same label α

38. Uniform Metric l1l1 l2l2 l3l3 w w w M α (a) = l α for all random variables V a Multiplicative Bound = 2 d max = max i,k d(i,k) d max d min i = M α (a), k = M β (b), α ≠ β d min = min i,k d(i,k) i = M α (a), k = M β (b), α ≠ β

39. Uniform Metric l1l1 l2l2 l3l3 w w w M α (a) = l α for all random variables V a Multiplicative Bound = 2 d max = 2w d max d min d min = min i,k d(i,k) i = M α (a), k = M β (b), α ≠ β

40. Uniform Metric l1l1 l2l2 l3l3 w w w M α (a) = l α for all random variables V a Multiplicative Bound = 2 d max = 2w d max d min d min = 2w

41. Uniform Metric l1l1 l2l2 l3l3 w w w M α (a) = l α for all random variables V a Multiplicative Bound = 2 d max = 2w d min = 2w Same bound as LP

42. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Visualizing Metrics Uniform Metric Hierarchically Separated Tree (HST) Metrics General Metrics Move-Making for Truncated Convex Models

43. HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 Graph is a Tree. Labels are leaves

44. HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 Edge lengths for all children are the same

45. HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 Edge lengths decrease from root to leaf by factor r ≥ 2

46. HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 w 2 ≤ w 1 /r w 3 ≤ w 1 /r w 4 ≤ w 1 /r

47. HST Metric w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8 w 5 ≤ w 2 /r w 6 ≤ w 2 /r w 7 ≤ w 3 /rw 8 ≤ w 3 /r

48. Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4

49. l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 M α (a) = l α

50. Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 Multiplicative bound of 2 for (a,b) where f*(a), f*(b)  {1,2,3}

51. Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 Multiplicative bound of 2 for (a,b) where f*(a), f*(b)  {4,5,6} f2f2

52. Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 Multiplicative bound of 2 for (a,b) where f*(a), f*(b)  {7,8,9} f2f2 f3f3

53. Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 f2f2 f3f3 M 1 (a) = f 1 (a)M 2 (a) = f 2 (a) M 3 (a) = f 3 (a)

54. Metric Labeling for 3-level HST l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 w4w4 w4w4 w4w4 f1f1 f2f2 f3f3 Multiplicative bound of at most 2(1+1/r)

55. Metric Labeling for 4-level HST Multiplicative bound of at most 2(1+1/r + 1/r 2 ) w1w1 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 w1w1 w1w1 w2w2 w2w2 w3w3 w3w3 w4w4 w5w5 w5w5 w6w6 w6w6 w7w7 w7w7 w8w8 w8w8

56. Metric Labeling for L-level HST Multiplicative bound = 2(1+1/r + … + 1/r L-2 ) < 2r/(r-1) Constant bound for all HST metrics

57. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Visualizing Metrics Uniform Metric Hierarchically Separated Tree (HST) Metrics General Metrics Move-Making for Truncated Convex Models

58. General Metrics Any metric can be approximated using a mixture of O(h log h) HST metrics with distortion O(log h). Fakcharoenphol, Rao and Talwar, 2003 Σ T ρ(T)d T (i,j) Σ T ρ(T) = 1 d(i,j) ≤ ≤ O(log h) d(i,j)

59. Metric Labeling for General Metrics Approximate d using Σ T ρ(T)d T Solve metric labeling for each HST d T Pick the labeling with the lowest energy Multiplicative bound = 2r/(r-1)*O(log h) = O(log h) Same bound as LP

60. Outline Convex Relaxations Move-Making Algorithms Comparison Move-Making for Metric Labeling Move-Making for Truncated Convex Models Kumar and Torr, NIPS 2008

61. Truncated Convex Models VaVa VbVb s ab is non-negative d(.) is a convex function E(f) min f = Σ a θ a (f(a)) + Σ (a,b) s ab min{d(f(a),f(b)), M}

62. Truncated Convex Models s ab is non-negative d(.) is a convex function Low-level vision applications NP hard E(f) min f = Σ a θ a (f(a)) + Σ (a,b) s ab min{d(f(a),f(b)), M} Truncated Linear Truncated Quadratic

63. Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {M α (a)} Update f = f α Repeat until convergence Any label M α (a) instead of the same label l α

64. Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {M α (a)} Update f = f α Repeat until convergence Any interval of labels M α (a) instead of a single label

65. Modified Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 1, for all V a ) For α = 1, 2, …, h End f α = argmin f’ E(f’) s.t. f’(a)  {f(a)} U {α, α+1, …, α+L} Update f = f α Repeat until convergence Any interval of labels M α (a) instead of a single label

66. Length of the Interval Submodular problem Exact optimization in polynomial time Small interval implies bigger ratio of d max /d min

67. Length of the Interval Non-submodular problem

68. Length of the Interval Submodular problem Optimization of upper bound in polynomial time Big interval implies smaller ratio of d max /d min

69. Length of the Interval Truncated Linear Bound = 2 + max 2M, L LM L = √2M Bound = 2 + √2 Truncated Quadratic Bound = O(√M)L = √M

70. Conclusion Move Making LP Potts22 MetricO(log h) Truncated Linear 2 + √2 Truncated Quadratic O(√M)

71. Future Work Move-making and convex relaxations connection Better measures than multiplicative bounds Designing hybrid algorithms

72. Questions? http://cvc.centrale-ponts.fr/personnel/pawan