1. Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online http://cvn.ecp.fr/personnel/pawan/

2. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) Label l 0 Label l 1

3. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13 Label l 0 Label l 1

4. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) Label l 0 Label l 1

5. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) 5 + 1 + 4 + 0 + 6 + 4 + 7 = 27 Label l 0 Label l 1

6. Energy Minimization VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) f* = arg min Q(f;  ) q* = min Q(f;  ) = Q(f*;  ) Label l 0 Label l 1

7. Min-Marginals VaVa VbVb VcVc VdVd 2 5 4 2 6 3 3 7 0 1 1 0 0 2 1 1 41 0 3 f* = arg min Q(f;  ) such that f(a) = i Min-marginal q a;i Label l 0 Label l 1

8. Min-Marginals and MAP Minimum min-marginal of any variable = energy of MAP labelling min f Q(f;  ) such that f(a) = i q a;i min i min i ( ) V a has to take one label min f Q(f;  )

9. Recap We only need to know two sets of equations General form of Reparameterization  ’ a;i =  a;i  ’ ab;ik =  ab;ik + M ab;k - M ab;k + M ba;i - M ba;i  ’ b;k =  b;k Reparameterization of (a,b) in Belief Propagation M ab;k = min i {  a;i +  ab;ik } M ba;i = 0

10. Dynamic Programming 3 variables  2 variables + book-keeping n variables  (n-1) variables + book-keeping Start from left, go to right Reparameterize current edge (a,b) M ab;k = min i {  a;i +  ab;ik }  ’ ab;ik =  ab;ik + M ab;k - M ab;k  ’ b;k =  b;k Repeat

11. LP Relaxation and its Dual TRW Message Passing Outline

12. Integer Programming Formulation min ∑ a ∑ i  a;i y a;i + ∑ (a,b) ∑ ik  ab;ik y ab;ik y a;i  {0,1} ∑ i y a;i = 1 y ab;ik = y a;i y b;k

13. Integer Programming Formulation min  T y y a;i  {0,1} ∑ i y a;i = 1 y ab;ik = y a;i y b;k  = [ …  a;i …. ; …  ab;ik ….] y = [ … y a;i …. ; … y ab;ik ….]

14. Linear Programming Relaxation min  T y y a;i  {0,1} ∑ i y a;i = 1 y ab;ik = y a;i y b;k Two reasons why we can’t solve this

15. Linear Programming Relaxation min  T y y a;i  [0,1] ∑ i y a;i = 1 y ab;ik = y a;i y b;k One reason why we can’t solve this

16. Linear Programming Relaxation min  T y y a;i  [0,1] ∑ i y a;i = 1 ∑ k y ab;ik = ∑ k y a;i y b;k One reason why we can’t solve this

17. Linear Programming Relaxation min  T y y a;i  [0,1] ∑ i y a;i = 1 One reason why we can’t solve this = 1 ∑ k y ab;ik = y a;i ∑ k y b;k

18. Linear Programming Relaxation min  T y y a;i  [0,1] ∑ i y a;i = 1 ∑ k y ab;ik = y a;i One reason why we can’t solve this

19. Linear Programming Relaxation min  T y y a;i  [0,1] ∑ i y a;i = 1 ∑ k y ab;ik = y a;i No reason why we can’t solve this * * memory requirements, time complexity

20. Dual Let’s try to understand it intuitively

21. Dual of the LP Relaxation Wainwright et al., 2001 VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi  VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi 11 22 33 44 55 66  i   i  

22. Dual of the LP Relaxation Wainwright et al., 2001 q*(  1 )  i   i   q*(  2 ) q*(  3 ) q*(  4 )q*(  5 )q*(  6 )  q*(  i ) Dual of LP  VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi max

23. Dual of the LP Relaxation Wainwright et al., 2001  i   i   max  q*(  i ) I can easily compute q*(  i ) I can easily maintain reparam constraint So can I easily solve the dual?

24. LP Relaxation and its Dual TRW Message Passing Outline

25. Things to Remember Forward-pass computes min-marginals of root BP is exact for trees Every iteration provides a reparameterization

26. TRW Message Passing Kolmogorov, 2006 VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi 11 22 33 44 55 66  i   i    q*(  i ) Pick a variable VaVa

27. TRW Message Passing Kolmogorov, 2006  i   i    q*(  i ) VcVc VbVb VaVa  1 c;0  1 c;1  1 b;0  1 b;1  1 a;0  1 a;1 VaVa VdVd VgVg  4 a;0  4 a;1  4 d;0  4 d;1  4 g;0  4 g;1

28. TRW Message Passing Kolmogorov, 2006  1 +  4 +  rest   q*(  1 ) + q*(  4 ) + K VcVc VbVb VaVa VaVa VdVd VgVg Reparameterize to obtain min-marginals of V a  1 c;0  1 c;1  1 b;0  1 b;1  1 a;0  1 a;1  4 a;0  4 a;1  4 d;0  4 d;1  4 g;0  4 g;1

29. TRW Message Passing Kolmogorov, 2006  ’ 1 +  ’ 4 +  rest VcVc VbVb VaVa  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’ 1 a;0  ’ 1 a;1 VaVa VdVd VgVg  ’ 4 a;0  ’ 4 a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1 One pass of Belief Propagation q*(  ’ 1 ) + q*(  ’ 4 ) + K

30. TRW Message Passing Kolmogorov, 2006  ’ 1 +  ’ 4 +  rest   VcVc VbVb VaVa VaVa VdVd VgVg Remain the same q*(  ’ 1 ) + q*(  ’ 4 ) + K  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’ 1 a;0  ’ 1 a;1  ’ 4 a;0  ’ 4 a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1

31. TRW Message Passing Kolmogorov, 2006  ’ 1 +  ’ 4 +  rest   min{  ’ 1 a;0,  ’ 1 a;1 } + min{  ’ 4 a;0,  ’ 4 a;1 } + K VcVc VbVb VaVa VaVa VdVd VgVg  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’ 1 a;0  ’ 1 a;1  ’ 4 a;0  ’ 4 a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1

32. TRW Message Passing Kolmogorov, 2006  ’ 1 +  ’ 4 +  rest   VcVc VbVb VaVa VaVa VdVd VgVg Compute average of min-marginals of V a  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’ 1 a;0  ’ 1 a;1  ’ 4 a;0  ’ 4 a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1 min{  ’ 1 a;0,  ’ 1 a;1 } + min{  ’ 4 a;0,  ’ 4 a;1 } + K

33. TRW Message Passing Kolmogorov, 2006  ’ 1 +  ’ 4 +  rest   VcVc VbVb VaVa VaVa VdVd VgVg  ’’ a;0 =  ’ 1 a;0 +  ’ 4 a;0 2  ’’ a;1 =  ’ 1 a;1 +  ’ 4 a;1 2  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’ 1 a;0  ’ 1 a;1  ’ 4 a;0  ’ 4 a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1 min{  ’ 1 a;0,  ’ 1 a;1 } + min{  ’ 4 a;0,  ’ 4 a;1 } + K

34. TRW Message Passing Kolmogorov, 2006  ’’ 1 +  ’’ 4 +  rest VcVc VbVb VaVa VaVa VdVd VgVg  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’’ a;0  ’’ a;1  ’’ a;0  ’’ a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1  ’’ a;0 =  ’ 1 a;0 +  ’ 4 a;0 2  ’’ a;1 =  ’ 1 a;1 +  ’ 4 a;1 2 min{  ’ 1 a;0,  ’ 1 a;1 } + min{  ’ 4 a;0,  ’ 4 a;1 } + K

35. TRW Message Passing Kolmogorov, 2006  ’’ 1 +  ’’ 4 +  rest   VcVc VbVb VaVa VaVa VdVd VgVg  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’’ a;0  ’’ a;1  ’’ a;0  ’’ a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1  ’’ a;0 =  ’ 1 a;0 +  ’ 4 a;0 2  ’’ a;1 =  ’ 1 a;1 +  ’ 4 a;1 2 min{  ’ 1 a;0,  ’ 1 a;1 } + min{  ’ 4 a;0,  ’ 4 a;1 } + K

36. TRW Message Passing Kolmogorov, 2006 VcVc VbVb VaVa VaVa VdVd VgVg 2 min{  ’’ a;0,  ’’ a;1 } + K  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’’ a;0  ’’ a;1  ’’ a;0  ’’ a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1  ’’ 1 +  ’’ 4 +  rest    ’’ a;0 =  ’ 1 a;0 +  ’ 4 a;0 2  ’’ a;1 =  ’ 1 a;1 +  ’ 4 a;1 2

37. TRW Message Passing Kolmogorov, 2006 VcVc VbVb VaVa VaVa VdVd VgVg  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’’ a;0  ’’ a;1  ’’ a;0  ’’ a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1 min {p 1 +p 2, q 1 +q 2 }min {p 1, q 1 } + min {p 2, q 2 } ≥ 2 min{  ’’ a;0,  ’’ a;1 } + K  ’’ 1 +  ’’ 4 +  rest  

38. TRW Message Passing Kolmogorov, 2006 VcVc VbVb VaVa VaVa VdVd VgVg Objective function increases or remains constant  ’ 1 c;0  ’ 1 c;1  ’ 1 b;0  ’ 1 b;1  ’’ a;0  ’’ a;1  ’’ a;0  ’’ a;1  ’ 4 d;0  ’ 4 d;1  ’ 4 g;0  ’ 4 g;1 2 min{  ’’ a;0,  ’’ a;1 } + K  ’’ 1 +  ’’ 4 +  rest  

39. TRW Message Passing Initialize  i. Take care of reparam constraint Choose random variable V a Compute min-marginals of V a for all trees Node-average the min-marginals REPEAT Kolmogorov, 2006 Can also do edge-averaging

40. Preliminaries LP Relaxation and its Dual TRW Message Passing –Examples –Primal Solution –Results Outline

41. Example 1 VaVa VbVb 0 1 1 0 2 5 4 2 l0l0 l1l1 VbVb VcVc 0 2 3 1 4 2 6 3 VcVc VaVa 1 4 1 0 6 3 6 4 56 7 Pick variable V a. Reparameterize.

42. Example 1 VaVa VbVb -3 -2 -2 5 7 4 2 VbVb VcVc 0 2 3 1 4 2 6 3 VcVc VaVa -3 1 6 3 10 7 56 7 Average the min-marginals of V a l0l0 l1l1

43. Example 1 VaVa VbVb -3 -2 -2 7.5 7 4 2 VbVb VcVc 0 2 3 1 4 2 6 3 VcVc VaVa -3 1 6 3 7.5 7 76 7 Pick variable V b. Reparameterize. l0l0 l1l1

44. Example 1 VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.5 7 VbVb VcVc -5 -3 -3 9 6 6 3 VcVc VaVa 1 6 3 7.5 7 76 7 Average the min-marginals of V b l0l0 l1l1

45. Example 1 VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 6.5 7 Value of dual does not increase l0l0 l1l1

46. Example 1 VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 6.5 7 Maybe it will increase for V c NO l0l0 l1l1

47. Example 1 VaVa VbVb -7.5 -7 -5.5 -7 7.5 7 8.75 6.5 VbVb VcVc -5 -3 -3 8.75 6.5 6 3 VcVc VaVa -3 1 6 3 7.5 7 Strong Tree Agreement Exact MAP Estimate f 1 (a) = 0f 1 (b) = 0f 2 (b) = 0f 2 (c) = 0f 3 (c) = 0f 3 (a) = 0 l0l0 l1l1

48. Example 2 VaVa VbVb 0 1 1 0 2 5 2 2 VbVb VcVc 1 0 0 1 0 0 0 0 VcVc VaVa 0 1 1 0 0 3 4 8 40 4 Pick variable V a. Reparameterize. l0l0 l1l1

49. Example 2 VaVa VbVb -2 -2 4 7 2 2 VbVb VcVc 1 0 0 1 0 0 0 0 VcVc VaVa 0 0 1 0 3 4 9 40 4 Average the min-marginals of V a l0l0 l1l1

50. Example 2 VaVa VbVb -2 -2 4 8 2 2 VbVb VcVc 1 0 0 1 0 0 0 0 VcVc VaVa 0 0 1 0 3 4 8 40 4 Value of dual does not increase l0l0 l1l1

51. Example 2 VaVa VbVb -2 -2 4 8 2 2 VbVb VcVc 1 0 0 1 0 0 0 0 VcVc VaVa 0 0 1 0 3 4 8 40 4 Maybe it will decrease for V b or V c NO l0l0 l1l1

52. Example 2 VaVa VbVb -2 -2 4 8 2 2 VbVb VcVc 1 0 0 1 0 0 0 0 VcVc VaVa 0 0 1 0 3 4 8 f 1 (a) = 1f 1 (b) = 1f 2 (b) = 1f 2 (c) = 0f 3 (c) = 1f 3 (a) = 1 f 2 (b) = 0f 2 (c) = 1 Weak Tree Agreement Not Exact MAP Estimate l0l0 l1l1

53. Example 2 VaVa VbVb -2 -2 4 8 2 2 VbVb VcVc 1 0 0 1 0 0 0 0 VcVc VaVa 0 0 1 0 3 4 8 Weak Tree Agreement Convergence point of TRW l0l0 l1l1 f 1 (a) = 1f 1 (b) = 1f 2 (b) = 1f 2 (c) = 0f 3 (c) = 1f 3 (a) = 1 f 2 (b) = 0f 2 (c) = 1

54. Preliminaries LP Relaxation and its Dual TRW Message Passing –Examples –Primal Solution –Results Outline

55. Obtaining the Labelling Only solves the dual. Primal solutions? VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi  ’ =   i   Fix the label Of V a

56. Obtaining the Labelling Only solves the dual. Primal solutions? VaVa VbVb VcVc VdVd VeVe VfVf VgVg VhVh ViVi  ’ =   i   Fix the label Of V b Continue in some fixed order Meltzer et al., 2006

57. Computational Issues of TRW Speed-ups for some pairwise potentials Basic Component is Belief Propagation Felzenszwalb & Huttenlocher, 2004 Memory requirements cut down by half Kolmogorov, 2006 Further speed-ups using monotonic chains Kolmogorov, 2006

58. Theoretical Properties of TRW Always converges, unlike BP Kolmogorov, 2006 Strong tree agreement implies exact MAP Wainwright et al., 2001 Optimal MAP for two-label submodular problems Kolmogorov and Wainwright, 2005  ab;00 +  ab;11 ≤  ab;01 +  ab;10

59. Preliminaries LP Relaxation and its Dual TRW Message Passing –Examples –Primal Solution –Results Outline

60. Results Binary Segmentation Szeliski et al., 2008 Labels - {foreground, background} Unary Potentials: -log(likelihood) using learnt fg/bg models Pairwise Potentials: 0, if same labels 1 - exp(|d a - d b |), if different labels

61. Results Binary Segmentation Labels - {foreground, background} Unary Potentials: -log(likelihood) using learnt fg/bg models Szeliski et al., 2008 Pairwise Potentials: 0, if same labels 1 - exp(|d a - d b |), if different labels TRW

62. Results Binary Segmentation Labels - {foreground, background} Unary Potentials: -log(likelihood) using learnt fg/bg models Szeliski et al., 2008 Belief Propagation Pairwise Potentials: 0, if same labels 1 - exp(|d a - d b |), if different labels

63. Results Stereo Correspondence Szeliski et al., 2008 Labels - {disparities} Unary Potentials: Similarity of pixel colours Pairwise Potentials: 0, if same labels 1 - exp(|d a - d b |), if different labels

64. Results Szeliski et al., 2008 Labels - {disparities} Unary Potentials: Similarity of pixel colours Pairwise Potentials: 0, if same labels 1 - exp(|d a - d b |), if different labels TRW Stereo Correspondence

65. Results Szeliski et al., 2008 Labels - {disparities} Unary Potentials: Similarity of pixel colours Belief Propagation Pairwise Potentials: 0, if same labels 1 - exp(|d a - d b |), if different labels Stereo Correspondence

66. Results Non-submodular problems Kolmogorov, 2006 BP TRW-S 30x30 grid K 50 BP TRW-S BP outperforms TRW-S

67. Code + Standard Data http://vision.middlebury.edu/MRF