1. Discrete Optimization in Computer Vision M. Pawan Kumar pawan.kumar@ecp.fr Slides will be available online http://www.centrale-ponts.fr/personnel/pawan/

2. About the Tutorial Emphasis on ‘optimization’ A bit math-y M. Ade-Up, N. Ames. A large-scale study of receptibility to math. The Fake Journal of Convenient Results, September, 2013. Best time for math is now (9am – 11am) Ask questions anytime

3. Outline Problem Formulation –Energy Function –Energy Minimization Graph Cuts Algorithms – Part I Message Passing Algorithms – Part II

4. Markov Random Field VaVa VbVb VcVc VdVd Label l 0 Label l 1 Random Variables V = {V a, V b, ….} Labels L = {l 0, l 1, ….} Labelling f: {a, b, …. }  {0,1, …}

5. Energy Function VaVa VbVb VcVc VdVd Q(f) = ∑ a  a;f(a) Unary Potential 2 5 4 2 6 3 3 7 Label l 0 Label l 1 Easy to minimize Neighbourhood

6. Energy Function VaVa VbVb VcVc VdVd E : (a,b)  E iff V a and V b are neighbours E = { (a,b), (b,c), (c,d) } 2 5 4 2 6 3 3 7 Label l 0 Label l 1

7. Energy Function VaVa VbVb VcVc VdVd +∑ (a,b)  ab;f(a)f(b) Pairwise Potential 0 1 1 0 0 2 1 1 41 0 3 2 5 4 2 6 3 3 7 Label l 0 Label l 1 Q(f) = ∑ a  a;f(a)

8. Energy Function VaVa VbVb VcVc VdVd 0 1 1 0 0 2 1 1 41 0 3 Parameter 2 5 4 2 6 3 3 7 Label l 0 Label l 1 +∑ (a,b)  ab;f(a)f(b) Q(f;  )= ∑ a  a;f(a)

9. Outline Problem Formulation –Energy Function –Energy Minimization Graph Cuts Algorithms – Part I Message Passing Algorithms – Part II

10. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) Label l 0 Label l 1

11. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13 Label l 0 Label l 1

12. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) Label l 0 Label l 1

13. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) 5 + 1 + 4 + 0 + 6 + 4 + 7 = 27 Label l 0 Label l 1

14. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) f* = arg min Q(f;  ) In general, NP-hard Label l 0 Label l 1

15. Energy Minimization f(a)f(b)f(c)f(d) Q(f;  ) 000018 000115 001027 001120 010022 010119 011027 011120 16 possible labellings f(a)f(b)f(c)f(d) Q(f;  ) 100016 100113 101025 101118 1100 110115 111023 111116 f* = {1, 0, 0, 1}

16. Outline Problem Formulation Graph Cuts Algorithms – Part I Message Passing Algorithms – Part II

17. Interactive Binary Segmentation

18. Foreground histogram of RGB values FG Background histogram of RGB values BG ‘1’ indicates foreground and ‘0’ indicates background

19. Interactive Binary Segmentation More likely to be foreground than background

20. Interactive Binary Segmentation More likely to be background than foreground θ a;0 proportional to -log(BG(d a )) θ a;1 proportional to -log(FG(d a ))

21. Interactive Binary Segmentation More likely to belong to same label

22. Interactive Binary Segmentation Less likely to belong to same label θ ab;ik proportional to exp(-(d a -d b ) 2 ) if i ≠ k θ ab;ik = 0 if i = k

23. Outline – Graph Cuts Algorithms Minimum Cut Problem Two-Label Submodular Energy Functions Move-Making Algorithms

24. Directed Graph n1n1 n2n2 n3n3 n4n4 10 5 3 2 Positive arc lengths D = (N, A)

25. Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 Let N 1 and N 2 such that N 1 “union” N 2 = N N 1 “intersection” N 2 = Φ C is a set of arcs such that (n 1,n 2 )  A n 1  N 1 n 2  N 2 D = (N, A) C is a cut in the digraph D

26. Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 What is C? D = (N, A) N1N1 N2N2 {(n 1,n 2 ),(n 1,n 4 )} ? {(n 1,n 4 ),(n 3,n 2 )} ? {(n 1,n 4 )} ? ✓

27. Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 What is C? D = (N, A) N1N1 N2N2 {(n 1,n 2 ),(n 1,n 4 ),(n 3,n 2 )} ? {(n 1,n 4 ),(n 3,n 2 )} ? {(n 4,n 3 )} ? ✓

28. Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 What is C? D = (N, A) N2N2 N1N1 {(n 1,n 2 ),(n 1,n 4 ),(n 3,n 2 )} ? {(n 1,n 4 ),(n 3,n 2 )} ? {(n 3,n 2 )} ? ✓

29. Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 Let N 1 and N 2 such that N 1 “union” N 2 = N N 1 “intersection” N 2 = Φ C is a set of arcs such that (n 1,n 2 )  A n 1  N 1 n 2  N 2 D = (N, A) C is a cut in the digraph D

30. Weight of a Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 Sum of length of all arcs in C D = (N, A)

31. Weight of a Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 w(C) = Σ (n 1,n 2 )  C l(n 1,n 2 ) D = (N, A)

32. Weight of a Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 What is w(C)? D = (N, A) N1N1 N2N2 3

33. Weight of a Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 What is w(C)? D = (N, A) N1N1 N2N2 5

34. Weight of a Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 What is w(C)? D = (N, A) N2N2 N1N1 15

35. st-Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 A source “s” C is a cut such that s  N 1 t  N 2 D = (N, A) C is an st-cut s t A sink “t” 12 73

36. Weight of an st-Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 D = (N, A) s t 12 73 w(C) = Σ (n 1,n 2 )  C l(n 1,n 2 )

37. Weight of an st-Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 D = (N, A) s t 12 73 What is w(C)? 3

38. Weight of an st-Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 D = (N, A) s t 12 73 What is w(C)? 15

39. Minimum Cut Problem n1n1 n2n2 n3n3 n4n4 10 5 3 2 D = (N, A) s t 12 73 Find a cut with the minimum weight !! C* = argmin C w(C)

40. [Slide credit: Andrew Goldberg] Augmenting Path and Push-Relabel n: #nodes m: #arcs U: maximum arc length Solvers for the Minimum-Cut Problem

41. Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 Let N 1 and N 2 such that N 1 “union” N 2 = N N 1 “intersection” N 2 = Φ C is a set of arcs such that (n 1,n 2 )  A n 1  N 1 n 2  N 2 D = (N, A) C is a cut in the digraph D

42. st-Cut n1n1 n2n2 n3n3 n4n4 10 5 3 2 A source “s” C is a cut such that s  N 1 t  N 2 D = (N, A) C is an st-cut s t A sink “t” 12 73

43. Minimum Cut Problem n1n1 n2n2 n3n3 n4n4 10 5 3 2 D = (N, A) s t 12 73 Find a cut with the minimum weight !! C* = argmin C w(C) w(C) = Σ (n 1,n 2 )  C l(n 1,n 2 )

44. Remember … Positive arc lengths

45. Outline – Graph Cuts Algorithms Minimum Cut Problem Two-Label Submodular Energy Functions Move-Making Algorithms Hammer, 1965; Kolmogorov and Zabih, 2004

46. Overview Energy Q Digraph D Digraph D One node per random variable N = N 1 U N 2 N = N 1 U N 2 Compute Minimum Cut + Additional nodes “s” and “t” Labeling f* Labeling f* n a  N 1 implies f(a) = 0 n a  N 2 implies f(a) = 1

47. Outline Minimum Cut Problem Two-Label Submodular Energy Functions Unary Potentials Pairwise Potentials Move-Making Algorithms

48. Digraph for Unary Potentials VaVa θ a;0 θ a;1 P Q f(a) = 0 f(a) = 1

49. Digraph for Unary Potentials nana P Q s t f(a) = 0 f(a) = 1

50. Digraph for Unary Potentials nana P Q s t Let P ≥ Q P-Q 0 Q Q + Constant P-Q f(a) = 0 f(a) = 1

51. Digraph for Unary Potentials nana P Q s t Let P ≥ Q P-Q 0 Q Q + Constant P-Q f(a) = 1 w(C) = 0 f(a) = 0 f(a) = 1

52. Digraph for Unary Potentials nana P Q s t Let P ≥ Q P-Q 0 Q Q + Constant P-Q f(a) = 0 w(C) = P-Q f(a) = 0 f(a) = 1

53. Digraph for Unary Potentials nana P Q s t Let P < Q 0 Q-P P P + Constant Q-P f(a) = 0 f(a) = 1

54. Digraph for Unary Potentials nana P Q s t Let P < Q 0 Q-P P P + Constant f(a) = 1 w(C) = Q-P Q-P f(a) = 0 f(a) = 1

55. Digraph for Unary Potentials nana P Q s t Let P < Q 0 Q-P P P + Constant f(a) = 0 w(C) = 0 Q-P f(a) = 0 f(a) = 1

56. Outline – Graph Cuts Algorithms Minimum Cut Problem Two-Label Submodular Energy Functions Unary Potentials Pairwise Potentials Move-Making Algorithms

57. Digraph for Pairwise Potentials VaVa θ ab;11 VbVb θ ab;00 θ ab;01 θ ab;10 PR QS f(a) = 0f(a) = 1 f(b) = 0 f(b) = 1 00 Q-P 0S-Q 0 0R+Q-S-P 00 + + + PP PP

58. Digraph for Pairwise Potentials nana nbnb PR QS f(a) = 0f(a) = 1 f(b) = 0 f(b) = 1 00 Q-P 0S-Q 0 0R+Q-S-P 00 + + + PP PP s t Constant

59. Digraph for Pairwise Potentials nana nbnb PR QS 00 Q-P 0S-Q 0 0R+Q-S-P 00 + + s t Unary Potential f(b) = 1 Q-P f(a) = 0f(a) = 1 f(b) = 0 f(b) = 1

60. Digraph for Pairwise Potentials nana nbnb PR QS 0S-Q 0 0R+Q-S-P 00 + s t Unary Potential f(a) = 1 Q-PS-Q f(a) = 0f(a) = 1 f(b) = 0 f(b) = 1

61. Digraph for Pairwise Potentials nana nbnb PR QS 0R+Q-S-P 00 s t Pairwise Potential f(a) = 1, f(b) = 0 Q-PS-Q f(a) = 0f(a) = 1 f(b) = 0 f(b) = 1 R+Q-S-P

62. Digraph for Pairwise Potentials nana nbnb PR QS s t Q-PS-Q f(a) = 0f(a) = 1 f(b) = 0 f(b) = 1 R+Q-S-P R+Q-S-P ≥ 0 General 2-label MAP estimation is NP-hard

63. Overview Energy Q Digraph D Digraph D One node per random variable N = N 1 U N 2 N = N 1 U N 2 Compute Minimum Cut + Additional nodes “s” and “t” Labeling f* Labeling f* n a  N 1 implies f(a) = 0 n a  N 2 implies f(a) = 1

64. Outline – Graph Cuts Algorithms Minimum Cut Problem Two-Label Submodular Energy Functions Move-Making Algorithms

65. Stereo Correspondence Disparity Map

66. Stereo Correspondence L = {disparities} Pixel (x a,y a ) in left corresponds to pixel (x a +v a,y a ) in right

67. Stereo Correspondence L = {disparities} θ a;i is proportional to the difference in RGB values

68. Stereo Correspondence L = {disparities} θ ab;ik = w ab d(i,k) w ab proportional to exp(-(d a -d b ) 2 )

69. Move-Making Algorithms Space of All Labelings f

70. Expansion Algorithm Variables take label l α or retain current label Slide courtesy Pushmeet Kohli

71. Expansion Algorithm Sky House Tree Ground Initialize with TreeStatus:Expand GroundExpand HouseExpand Sky Slide courtesy Pushmeet Kohli Variables take label l α or retain current label

72. Expansion Algorithm Initialize labeling f = f 0 (say f 0 (a) = 0, for all V a ) For α = 0, 2, …, h-1 End f α = argmin f’ Q(f’) s.t. f’(a)  {f(a)} U {l α } Update f = f α Boykov, Veksler and Zabih, 2001 Repeat until convergence

73. Expansion Algorithm Restriction on pairwise potentials? θ ab;ik + θ ab;αα ≤ θ ab;iα + θ ab;αk Metric Labeling

74. Outline Problem Formulation –Energy Function –Energy Minimization Graph Cuts Algorithms – Part I Message Passing Algorithms – Part II

75. Pose Estimation Courtesy Pedro Felzenszwalb

76. Pose Estimation Courtesy Pedro Felzenszwalb

77. Pose Estimation Variables are body partsLabels are positions

78. Pose Estimation Unary potentials θ a;i proportional to fraction of foreground pixels Variables are body partsLabels are positions

79. Pose Estimation Pairwise potentials θ ab;ik proportional to d 2 Head Torso Joint location according to ‘head’ part Joint location according to ‘torso’ part d

80. Pose Estimation Pairwise potentials θ ab;ik proportional to d 2 Head Torso d Head Torso d >

81. Outline – Message Passing Preliminaries Energy Minimization for Trees Dual Decomposition

82. Energy Function VaVa VbVb VcVc VdVd 0 1 1 0 0 2 1 1 41 0 3 2 5 4 2 6 3 3 7 Label l 0 Label l 1 +∑ (a,b)  ab;f(a)f(b) Q(f;  )= ∑ a  a;f(a)

83. Outline – Message Passing Preliminaries –Min-Marginals –Reparameterization Energy Minimization for Trees Dual Decomposition

84. Min-Marginals VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 f* = arg min Q(f;  ) such that f(a) = i Min-marginal q a;i Label l 0 Label l 1

85. Min-Marginals 16 possible labellings q a;0 = 15 f(a)f(b)f(c)f(d) Q(f;  ) 000018 000115 001027 001120 010022 010119 011027 011120 f(a)f(b)f(c)f(d) Q(f;  ) 100016 100113 101025 101118 1100 110115 111023 111116

86. Min-Marginals 16 possible labellings q a;1 = 13 f(a)f(b)f(c)f(d) Q(f;  ) 100016 100113 101025 101118 1100 110115 111023 111116 f(a)f(b)f(c)f(d) Q(f;  ) 000018 000115 001027 001120 010022 010119 011027 011120

87. Min-Marginals and MAP Minimum min-marginal of any variable = energy of MAP labelling min f Q(f;  ) such that f(a) = i q a;i min i min i ( ) V a has to take one label min f Q(f;  )

88. Outline – Message Passing Preliminaries –Computing min-marginals –Reparameterization Energy Minimization for Trees Dual Decomposition

89. Reparameterization VaVa VbVb 2 5 4 2 0 1 1 0 f(a)f(b) Q(f;  ) 007 0110 105 116 2 + - 2 Add a constant to all  a;i Subtract that constant from all  b;k

90. Reparameterization f(a)f(b) Q(f;  ) 007 + 2 - 2 0110 + 2 - 2 105 + 2 - 2 116 + 2 - 2 Add a constant to all  a;i Subtract that constant from all  b;k Q(f;  ’) = Q(f;  ) VaVa VbVb 2 5 4 2 0 0 2 + - 2 1 1

91. Reparameterization VaVa VbVb 2 5 4 2 0 1 1 0 f(a)f(b) Q(f;  ) 007 0110 105 116 - 3 + 3 Add a constant to one  b;k Subtract that constant from  ab;ik for all ‘i’ - 3

92. Reparameterization VaVa VbVb 2 5 4 2 0 1 1 0 f(a)f(b) Q(f;  ) 007 0110 - 3 + 3 105 116 - 3 + 3 - 3 + 3 - 3 Q(f;  ’) = Q(f;  ) Add a constant to one  b;k Subtract that constant from  ab;ik for all ‘i’

93. Reparameterization VaVa VbVb 2 5 4 2 31 0 1 2 VaVa VbVb 2 5 4 2 31 1 0 1 - 2 + 2 + 1 - 1 VaVa VbVb 2 5 4 2 31 2 1 0 - 4+ 4 - 4  ’ a;i =  a;i  ’ b;k =  b;k  ’ ab;ik =  ab;ik + M ab;k - M ab;k + M ba;i - M ba;i Q(f;  ’) = Q(f;  )

94. Reparameterization Q(f;  ’) = Q(f;  ), for all f  ’ is a reparameterization of , iff  ’    ’ b;k =  b;k  ’ a;i =  a;i  ’ ab;ik =  ab;ik + M ab;k - M ab;k + M ba;i - M ba;i Equivalently Kolmogorov, PAMI, 2006 VaVa VbVb 2 5 4 2 0 0 2 + - 2 11

95. Recap Energy Minimization f* = arg min Q(f;  ) Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) Min-marginals q a;i = min Q(f;  ) s.t. f(a) = i Q(f;  ’) = Q(f;  ), for all f  ’   Reparameterization

96. Outline – Message Passing Preliminaries Energy Minimization for Trees Dual Decomposition Pearl, 1988

97. Belief Propagation Belief Propagation is exact for chains Some problems are easy Exact MAP for trees Clever Reparameterization

98. Outline – Message Passing Preliminaries Energy Minimization for Trees –Two Variables –Three Variables –Chains –Trees Dual Decomposition

99. Two Variables VaVa VbVb 2 5 2 1 0 VaVa VbVb 2 5 40 1 Choose the right constant  ’ b;k = q b;k Add a constant to one  b;k Subtract that constant from  ab;ik for all ‘i’

100. VaVa VbVb 2 5 2 1 0 VaVa VbVb 2 5 40 1 Choose the right constant  ’ b;k = q b;k  a;0 +  ab;00 = 5 + 0  a;1 +  ab;10 = 2 + 1 min M ab;0 = Two Variables

101. VaVa VbVb 2 5 5 -2 -3 VaVa VbVb 2 5 40 1 Choose the right constant  ’ b;k = q b;k Two Variables

102. VaVa VbVb 2 5 5 -2 -3 VaVa VbVb 2 5 40 1 Choose the right constant  ’ b;k = q b;k f(a) = 1  ’ b;0 = q b;0 Two Variables Potentials along the red path add up to 0

103. VaVa VbVb 2 5 5 -2 -3 VaVa VbVb 2 5 40 1 Choose the right constant  ’ b;k = q b;k  a;0 +  ab;01 = 5 + 1  a;1 +  ab;11 = 2 + 0 min M ab;1 = Two Variables

104. VaVa VbVb 2 5 5 -2 -3 VaVa VbVb 2 5 6-2 Choose the right constant  ’ b;k = q b;k f(a) = 1  ’ b;0 = q b;0 f(a) = 1  ’ b;1 = q b;1 Minimum of min-marginals = MAP estimate Two Variables

105. VaVa VbVb 2 5 5 -2 -3 VaVa VbVb 2 5 6-2 Choose the right constant  ’ b;k = q b;k f(a) = 1  ’ b;0 = q b;0 f(a) = 1  ’ b;1 = q b;1 f*(b) = 0f*(a) = 1 Two Variables

106. VaVa VbVb 2 5 5 -2 -3 VaVa VbVb 2 5 6-2 Choose the right constant  ’ b;k = q b;k f(a) = 1  ’ b;0 = q b;0 f(a) = 1  ’ b;1 = q b;1 We get all the min-marginals of V b Two Variables

107. Recap We only need to know two sets of equations General form of Reparameterization  ’ a;i =  a;i  ’ ab;ik =  ab;ik + M ab;k - M ab;k + M ba;i - M ba;i  ’ b;k =  b;k Reparameterization of (a,b) in Belief Propagation M ab;k = min i {  a;i +  ab;ik } M ba;i = 0

108. Outline – Message Passing Preliminaries Energy Minimization for Trees –Two Variables –Three Variables –Chains –Trees Dual Decomposition

109. Three Variables VaVa VbVb 2 5 2 1 0 VcVc 460 1 0 1 3 2 3 Reparameterize the edge (a,b) as before l0l0 l1l1

110. VaVa VbVb 2 5 5 -3 VcVc 660 1 -2 3 Reparameterize the edge (a,b) as before f(a) = 1 -22 3 Three Variables l0l0 l1l1

111. VaVa VbVb 2 5 5 -3 VcVc 660 1 -2 3 Reparameterize the edge (a,b) as before f(a) = 1 Potentials along the red path add up to 0 -22 3 Three Variables l0l0 l1l1

112. VaVa VbVb 2 5 5 -3 VcVc 660 1 -2 3 Reparameterize the edge (b,c) as before f(a) = 1 Potentials along the red path add up to 0 -22 3 Three Variables l0l0 l1l1

113. VaVa VbVb 2 5 5 -3 VcVc 612-6 -5 -2 9 Reparameterize the edge (b,c) as before f(a) = 1 Potentials along the red path add up to 0 f(b) = 1 f(b) = 0 -2-4 -3 Three Variables l0l0 l1l1

114. VaVa VbVb 2 5 5 -3 VcVc 612-6 -5 -2 9 Reparameterize the edge (b,c) as before f(a) = 1 Potentials along the red path add up to 0 f(b) = 1 f(b) = 0 q c;0 q c;1 -2-4 -3 Three Variables l0l0 l1l1

115. VaVa VbVb 2 5 5 -3 VcVc 612-6 -5 -2 9 f(a) = 1 f(b) = 1 f(b) = 0 q c;0 q c;1 f*(c) = 0 f*(b) = 0f*(a) = 1 Generalizes to any length chain -2-4 -3 Three Variables l0l0 l1l1

116. Outline – Message Passing Preliminaries Energy Minimization for Trees –Two Variables –Three Variables –Chains –Trees Dual Decomposition

117. Belief Propagation on Chains Start from left, go to right Reparameterize current edge (a,b) M ab;k = min i {  a;i +  ab;ik }  ’ ab;ik =  ab;ik + M ab;k - M ab;k  ’ b;k =  b;k Repeat till the end of the chain

118. Belief Propagation on Chains A way of computing reparam constants Generalizes to chains of any length Forward Pass - Start to End MAP estimate Min-marginals of final variable Backward Pass - End to start All other min-marginals

119. Computational Complexity Each constant takes O(|L|) Number of constants - O(|E||L|) O(|E||L| 2 ) Memory required ? O(|E||L|)

120. Outline – Message Passing Preliminaries Energy Minimization for Trees –Two Variables –Three Variables –Chains –Trees Dual Decomposition

121. Belief Propagation on Trees VbVb VaVa Forward Pass: Leaf  Root All min-marginals are computed Backward Pass: Root  Leaf VcVc VdVd VeVe VgVg VhVh

122. Outline – Message Passing Preliminaries Energy Minimization for Trees Dual Decomposition

123. min x ∑ i g i (x) s.t. x  C Minimize the energy of an MRF Each variable is assigned exactly one label

124. Dual Decomposition min x,x i ∑ i g i (x i ) s.t. x i  C x i = x

125. Dual Decomposition min x,x i ∑ i g i (x i ) s.t. x i  C

126. Dual Decomposition min x,x i ∑ i g i (x i ) + ∑ i λ i T (x i -x) s.t. x i  C max λ i KKT Condition: ∑ i λ i = 0

127. Dual Decomposition min x,x i ∑ i g i (x i ) + ∑ i λ i T x i s.t. x i  C max λ i

128. Dual Decomposition min x i ∑ i (g i (x i ) + λ i T x i ) s.t. x i  C Projected Supergradient Ascent max λ i Supergradient s of h(z) at z 0 h(z) - h(z 0 ) ≤ s T (z-z 0 ), for all z in the feasible region

129. Dual Decomposition min x i ∑ i (g i (x i ) + λ i T x i ) s.t. x i  C Initialize λ i 0 = 0 max λ i

130. Dual Decomposition min x i ∑ i (g i (x i ) + λ i T x i ) s.t. x i  C Compute supergradients max λ i s i = argmin x i ∑ i (g i (x i ) + (λ i t ) T x i )

131. Dual Decomposition min x i ∑ i (g i (x i ) + λ i T x i ) s.t. x i  C Project supergradients max λ i p i = s i - ∑ j s j /m where ‘m’ = number of subproblems (slaves)

132. Dual Decomposition min x i ∑ i (g i (x i ) + λ i T x i ) s.t. x i  C Update dual variables max λ i λ i t+1 = λ i t + η t p i where η t = learning rate = 1/(t+1) for example

133. Dual Decomposition Initialize λ i 0 = 0 Compute projected supergradients s i = argmin x i ∑ i (g i (x i ) + (λ i t ) T x i ) p i = s i - ∑ j s j /m Update dual variables λ i t+1 = λ i t + η t p i REPEAT

134. Outline – Message Passing Preliminaries Energy Minimization for Trees Dual Decomposition –Example 1 –Example 2 –Energy Minimization –Choice of Subproblems

135. DD – Example 1 VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 Strong Tree Agreement f 1 (a) = 0f 1 (b) = 0f 2 (b) = 0f 2 (c) = 0f 3 (c) = 0f 3 (a) = 0 l0l0 l1l1

136. DD – Example 1 VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 l0l0 l1l1 6.5 7

137. DD – Example 1 VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 l0l0 l1l1 1010--1010-- x a;0 x a;1 x b;0 x b;1 x c;0 x c;1 Optimal solution

138. Supergradients VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 l0l0 l1l1 1010--1010-- s a;0 s a;1 s b;0 s b;1 s c;0 s c;1 --1010--1010 10--1010--10

139. Projected Supergradients VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 l0l0 l1l1 0000--0000-- p a;0 p a;1 p b;0 p b;1 p c;0 p c;1 --0000--0000 00--0000--00

140. Objective VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 l0l0 l1l1 No further increase in dual objective 6.5 7

141. DD VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 l0l0 l1l1 Strong Tree Agreement implies DD stops No further increase in dual objective 6.5 7

142. Outline – Message Passing Preliminaries Energy Minimization for Trees Dual Decomposition –Example 1 –Example 2 –Energy Minimization –Choice of Subproblems

143. DD – Example 2 VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 0 0 0 0 1 04 8 Weak Tree Agreement f 1 (a) = 1f 1 (b) = 1f 2 (b) = 1f 2 (c) = 0f 3 (c) = 1f 3 (a) = 1 f 2 (b) = 0f 2 (c) = 1 -0.2

144. DD – Example 2 VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 0 0 0 0 1 0 -0.2 4 8 40 4

145. DD – Example 2 VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 0 0 0 0 1 04 8 0101--0101-- x a;0 x a;1 x b;0 x b;1 x c;0 x c;1 Optimal solution -0.2

146. Supergradients VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 0 0 0 0 1 04 8 0101--0101-- s a;0 s a;1 s b;0 s b;1 s c;0 s c;1 --0110--0110 01--0101--01 -0.2

147. Projected Supergradients VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 0 0 0 0 1 04 8 0000--0000-- p a;0 p a;1 p b;0 p b;1 p c;0 p c;1 - 0 0.5 -0.5 0 - -0.5 0.5 -0.2

148. Update with Learning Rate η t = 1 VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 0 0 0 0 1 04 8 0000--0000-- p a;0 p a;1 p b;0 p b;1 p c;0 p c;1 - 0 0.5 -0.5 0 - -0.5 0.5 -0.2

149. Objective VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 -0.5 0.5 0 0 1 0.54 8 -0.7 4 -0.5 4.3 Decrease in dual objective

150. Supergradients VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 -0.5 0.5 0 0 1 0.54 8 -0.7 0101--0101-- s a;0 s a;1 s b;0 s b;1 s c;0 s c;1 --1001--1001 01--1001--10

151. Projected Supergradients VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 -0.5 0.5 0 0 1 0.54 8 -0.7 0 -0.5 0.5 - p a;0 p a;1 p b;0 p b;1 p c;0 p c;1 - 0.5 -0.5 0.5 0 - 0.5 -0.5

152. VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2 2 1 0 0 1 0 0 -0.5 0.5 0 0 1 0.54 8 -0.7 0 -0.5 0.5 - p a;0 p a;1 p b;0 p b;1 p c;0 p c;1 - 0.5 -0.5 0.5 0 - 0.5 -0.5 Update with Learning Rate η t = 1/2

153. VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2.25 1.75 1 0 0 1 -0.25 0.25 -0.25 0.25 0 0 1 0.254 8 -0.45 Updated Subproblems

154. VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2.25 1.75 1 0 0 1 -0.25 0.25 -0.25 0.25 0 0 1 0.254 8 -0.45 Objective 4.25 0 Increase in dual objective

155. VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2.25 1.75 1 0 0 1 -0.25 0.25 -0.25 0.25 0 0 1 0.254 8 -0.45 DD 4.25 0 Increase in dual objective

156. VaVa VbVb VbVb VcVc VcVc VaVa l0l0 l1l1 -2 -2 4 8 2.25 1.75 1 0 0 1 -0.25 0.25 -0.25 0.25 0 0 1 0.254 8 -0.45 DD 4.25 0 DD eventually converges Satisfies weak tree agreement

157. Outline – Message Passing Preliminaries Energy Minimization for Trees Dual Decomposition –Example 1 –Example 2 –Energy Minimization –Choice of Subproblems

158. Dual Decomposition Komodakis et al., 2007 VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi 11 22 33 44 55 66 1010 s 1 a = 1010 s 4 a = Slaves agree on label for V a

159. Dual Decomposition Komodakis et al., 2007 VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi 11 22 33 44 55 66 1010 s 1 a = 1010 s 4 a = 0000 p 1 a = 0000 p 4 a =

160. Dual Decomposition Komodakis et al., 2007 VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi 11 22 33 44 55 66 1010 s 1 a = 0101 s 4 a = Slaves disagree on label for V a

161. Dual Decomposition Komodakis et al., 2007 VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi 11 22 33 44 55 66 1010 s 1 a = 0101 s 4 a = 0.5 -0.5 p 1 a = -0.5 0.5 p 4 a = Unary cost increases

162. Dual Decomposition Komodakis et al., 2007 VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi 11 22 33 44 55 66 1010 s 1 a = 0101 s 4 a = 0.5 -0.5 p 1 a = -0.5 0.5 p 4 a = Unary cost decreases

163. Dual Decomposition Komodakis et al., 2007 VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi 11 22 33 44 55 66 1010 s 1 a = 0101 s 4 a = 0.5 -0.5 p 1 a = -0.5 0.5 p 4 a = Push the slaves towards agreement

164. Outline – Message Passing Preliminaries Energy Minimization for Trees Dual Decomposition –Example 1 –Example 2 –Energy Minimization –Choice of Subproblems

165. Subproblems VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi Black edges submodular Red edges supermodular Binary labeling problem VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi

166. Subproblems VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi Black edges submodular Red edges supermodular Binary labeling problem VaVa VbVb VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi Remains submodular over iterations

167. Tighter Relaxations VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VdVd VeVe VbVb VcVc VeVe VfVf VdVd VeVe VgVg VhVh VeVe VfVf VhVh ViVi Relaxation that is tight for the above 4-cycles

168. High-Order Potentials VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VdVd VeVe VgVg VhVh ViVi VaVa VdVd VeVe VfVf VgVg VhVh ViVi VbVb VcVc VeVe VfVf

169. VbVb VcVc VeVe VfVf Labeling y for Clique Value of Potential θ c;y Subproblem: min y θ c;y + λ T y O(h |C| )!!

170. Sparse High-Order Potentials VbVb VcVc VeVe VfVf Labeling y for Clique Value of Potential θ c;y Subproblem: min y θ c;y + λ T y O(h|C|)!! Σ a y a;0 = 0 Σ a y a;0 > 0

171. Sparse High-Order Potentials Many useful potentials are sparse P n Potts Model Pattern-based Potentials Uniqueness constraints Covering constraints And now you can solve them efficiently !!