1. Relaxations and Moves for MAP Estimation in MRFs M. Pawan Kumar STANFORDSTANFORD Vladimir KolmogorovPhilip TorrDaphne Koller

2. Our Problem v1v1 v2v2 v3v3 v4v4 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 Label l 1 Label l 2 Random Variables V = {v 1,...,v 4 } Label Set L = {l 1, l 2 } Labeling f: V  L (shown in red)

3. Our Problem v1v1 v2v2 v3v3 v4v4 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 Label l 1 Label l 2 Random Variables V = {v 1,...,v 4 } Label Set L = {l 1, l 2 } Labeling f: V  L (shown in red) Energy of Labeling E(f) = 13 (shown in green)

4. Our Problem v1v1 v2v2 v3v3 v4v4 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 Label l 1 Label l 2 Find f* = argmin f E(f) Arbitrary topology, discrete label set, potentials (NP-hard) Pairwise energy function: unary and pairwise potentials (still NP-hard)

5. Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

6. Integer Programming Formulation v1v1 v2v2 2 5 4 2 0 1 3 0 Unary Potentials Unary Potential u = [ 5 Cost of v 1 = 1 2 Cost of v 1 = 2 ; 2 4 ] Labeling f shown in red Label l 1 Label l 2

7. Label vector x = [ -1 v 1  1 1 v 1 = 2 ; 1 -1 ] T Recall that the aim is to find the optimal x Integer Programming Formulation v1v1 v2v2 2 5 4 2 0 1 3 0 Unary Potentials Labeling f shown in red Label l 1 Label l 2 Unary Potential u = [ 5 2 ; 2 4 ]

8. Label vector x = [ -11; 1 -1 ] T Sum of Unary Potentials = 1 2 ∑ i u i (1 + x i ) Integer Programming Formulation v1v1 v2v2 2 5 4 2 0 1 3 0 Unary Potentials Labeling f shown in red Label l 1 Label l 2 Unary Potential u = [ 5 2 ; 2 4 ]

9. 0 Cost of v 1 = 1 and v 1 = 1 0 00 0 Cost of v 1 = 1 and v 2 = 1 3 Cost of v 1 = 1 and v 2 = 2 10 00 00 10 30 Pairwise Potential P Integer Programming Formulation v1v1 v2v2 2 5 4 2 0 1 3 0 Pairwise Potentials Labeling f shown in red Label l 1 Label l 2

10. Pairwise Potential P 00 00 0 3 10 00 00 10 30 Sum of Pairwise Potentials 1 4 ∑ ij P ij (1 + x i )(1+x j ) Integer Programming Formulation v1v1 v2v2 2 5 4 2 0 1 3 0 Pairwise Potentials Labeling f shown in red Label l 1 Label l 2

11. Sum of Pairwise Potentials 1 4 ∑ ij P ij (1 + x i +x j + x i x j ) 1 4 ∑ ij P ij (1 + x i + x j + X ij )= X = x x T X ij = x i x j Integer Programming Formulation v1v1 v2v2 2 5 4 2 0 1 3 0 Pairwise Potentials Labeling f shown in red Label l 1 Label l 2 Pairwise Potential P 00 00 0 3 10 00 00 10 30

12. Constraints Uniqueness Constraint ∑ x i = 2 - |L| i  v a Integer Constraints x i  {-1,1} X = x x T Integer Programming Formulation

13. x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  {-1,1} X = x x T Convex Non-Convex Integer Programming Formulation

14. Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

15. LP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  {-1,1} X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

16. LP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

17. LP Relaxation X = x x T Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  v b

18. LP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

19. LP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1], Retain Convex Part Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  v b

20. Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

21. SDP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  {-1,1} X = x x T Retain Convex Part Lasserre, 2000 Relax Non-Convex Constraint

22. SDP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X = x x T Relax Non-Convex Constraint Lasserre, 2000 Retain Convex Part

23. x1x1 x2x2 xnxn 1... 1x1x1 x2x2... xnxn 1xTxT x X = Rank = 1 X ii = 1 Positive Semidefinite Convex Non-Convex SDP Relaxation

24. x1x1 x2x2 xnxn 1... 1x1x1 x2x2... xnxn 1xTxT x X = X ii = 1 Positive Semidefinite Convex SDP Relaxation

25. x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X = x x T Relax Non-Convex Constraint Lasserre, 2000 Retain Convex Part

26. SDP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X ii = 1 X - xx T 0 Accurate Inefficient Lasserre, 2000 Retain Convex Part Positive Semidefinite

27. Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

28. SOCP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  v a x i  [-1,1] X ii = 1 X - xx T 0 Derive SOCP relaxation from the SDP relaxation Further Relaxation

29. SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 Choose a matrix C 2 = UU T 0 REPEAT

30. Outline Convex Relaxations – Integer Programming Formulation – LP Relaxation – SDP Relaxation – SOCP Relaxation – Comparing Relaxations Move Making Algorithms Some Interesting Open Problems

31. Dominating Relaxation For all MAP Estimation problem (u, P) A dominates B A B ≥ Dominating relaxations are better

32. SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 If G is a tree, LP dominates SOCP

33. Examples Muramatsu and Suzuki, 2003 (MAXCUT) Ravikumar and Lafferty, 2006 (QP Relaxation) Kumar, Torr and Zisserman, 2006 (Equivalent SOCP Relaxation)

34. SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 If G is a cycle with non-negative P

35. SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 If G is an even cycle with non-positive P

36. SOCP Relaxation Choose a matrix C 1 = UU T 0 Kim and Kojima, 2000 Choose a sub-graph G Variables x G and X G (X G - x G x G T ) C1C1 ≥ 0 If G is an odd cycle with 1 non-positive P

37. SOCP Relaxation What about other cycles? Dominated by linear cycle inequalities Cliques? Dominated by clique inequalities Kumar, Kolmogorov and Torr, 2007

38. Outline Convex Relaxations Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves Some Interesting Open Problems

39. MRFs in Vision vava vbvb lili lklk P ab (i,k) P ab (i,k) = w ab min{ d(i-k), M } w ab is non-negative Truncated Linear Truncated Quadratic d(.) is a semi-metric distance u a (i) u b (k)

40. Move Making Search Neighbourhood Current Solution Optimal Move Solution Space Energy Slide courtesy of Pushmeet Kohli

41. Outline Convex Relaxations Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves Some Interesting Open Problems

42. Expansion Move Variables take label  or retain current label Boykov, Veksler, Zabih 2001 Slide courtesy of Pushmeet Kohli

43. Sky House Tree Ground Initialize with TreeStatus:Expand GroundExpand HouseExpand Sky [Boykov, Veksler, Zabih] Expansion Move Variables take label  or retain current label Boykov, Veksler, Zabih 2001 Slide courtesy of Pushmeet Kohli

44. Outline Convex Relaxations Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves Some Interesting Open Problems

45. Multiplicative Bounds LP Move- Making Potts Truncated Linear Truncated Quadratic Metric Labeling 22 2 + √22M O(√M)2M O(log h) 2M Expansion Bounds as bad as ICM Bounds

46. Outline Convex Relaxations Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves Some Interesting Open Problems

47. Randomized Rounding 0y’ 0 y’ i y’ k y’ h = 1 y’ i = y 0 + y 1 + … + y i Choose an interval of length L’ y i = (1 + x i )/2

48. Randomized Rounding 0y’ 0 y’ i y’ k y’ h = 1 Generate a random number r  (0,1] r y’ i = y 0 + y 1 + … + y i y i = (1 + x i )/2

49. Randomized Rounding 0y’ 0 y’ i y’ k y’ h = 1 Assign label next to r (if within the interval) r y’ i = y 0 + y 1 + … + y i y i = (1 + x i )/2

50. Move Making vava vbvb Initialize the labeling Choose interval I of L’ labels Each variable can Retain old label Choose a label from I Choose best labeling Iterate over intervals Non-submodular move? Submodular overestimation

51. Truncated Convex Models P ab (i,k) = w ab min{ d(i-k), M } Truncated Linear Truncated Quadratic d(.) is convexd(x+1) - 2d(x) + d(x-1) ≥ 0

52. Move Making vava vbvb Choose interval I of L’ labels Each variable can Retain old label Choose a label from I Choose best labeling Large L’ => Non-submodular

53. Move Making vava vbvb Submodular problem

54. Move Making vava vbvb Non-submodular Problem

55. Move Making vava vbvb Submodular problem Ishikawa, 2003; Veksler, 2007

56. Move Making vava vbvb a m+1 a m+2 anan t b m+1 b m+2 bnbn s

57. LP Bounds Kumar and Torr, NIPS 08 Type of ProblemBound Potts2 Truncated Linear 2 + √2 Truncated Quadratic O(√M) Metric Labeling O(log h) Kumar and Koller, UAI 09 Move Making

58. Outline Convex Relaxations Move Making Algorithms Some Interesting Open Problems

59. Problem 1 Relationship between rounding and move-making? What happens when n < h ?? (Should we even use move-making here??) What about semi-metric MRFs??

60. Problem 2 Graph-cuts based image segmentation Vicente, Kolmogorov, Rother, 2008

61. Problem 2 Image segmentation with connectivity prior Vicente, Kolmogorov, Rother, 2008

62. Problem 2++ Kumar and Koller, 20??

63. Questions?? http://ai.stanford.edu/~pawan