1. Discrete Optimization Lecture 2 – Part I M. Pawan Kumar pawan.kumar@ecp.fr Slides available online http://cvn.ecp.fr/personnel/pawan

2. Recap VaVa VbVb VcVc dada dbdb dcdc Label l 0 Label l 1 D : Observed data (image) V : Unobserved variables L : Discrete, finite label set Labeling f : V  L

3. Recap VaVa VbVb VcVc dada dbdb dcdc Label l 0 Label l 1 D : Observed data (image) V : Unobserved variables L : Discrete, finite label set Labeling f : V  L V a is assigned l f(a)

4. Recap VaVa VbVb VcVc 2 5 4 2 6 3 0 1 1 0 0 2 1 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) Label l 0 Label l 1  bc;f(b)f(c)  a;f(a) dada dbdb dcdc

5. Recap VaVa VbVb VcVc 2 5 4 2 6 3 0 1 1 0 0 2 1 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) Label l 0 Label l 1 f* = argmin f Q(f;  ) dada dbdb dcdc

6. Recap VaVa VbVb VcVc 2 5 4 2 6 3 0 1 1 0 0 2 1 3 Q(f;  ) = ∑ a  a;f(a) + ∑ (a,b)  ab;f(a)f(b) Label l 0 Label l 1 f* = argmin f Q(f;  )

7. Outline Convex Optimization Integer Programming Formulation Convex Relaxations Comparison Generalization of Results

8. Mathematical Optimization min g 0 (x) s.t. g i (x) ≤ 0 h i (x) = 0 Objective function Inequality constraints Equality constraints x is a feasible point  g i (x) ≤ 0, h i (x) = 0 x is a strictly feasible point  g i (x) < 0, h i (x) = 0 Feasible region - set of all feasible points

9. Convex Optimization min g 0 (x) s.t. g i (x) ≤ 0 h i (x) = 0 Objective function Inequality constraints Equality constraints Objective function is convex Feasible region is convex Convex set? Convex function?

10. Convex Set x1x1 x2x2 c x 1 + (1 - c) x 2 c  [0,1] Line Segment Endpoints

11. Convex Set x1x1 x2x2 All points on the line segment lie within the set For all line segments with endpoints in the set

12. Non-Convex Set x1x1 x2x2

13. Examples of Convex Sets x1x1 x2x2 Line Segment

14. Examples of Convex Sets x1x1 x2x2 Line

15. Examples of Convex Sets Hyperplane a T x - b = 0

16. Examples of Convex Sets Halfspace a T x - b ≤ 0

17. Examples of Convex Sets Second-order Cone ||x|| ≤ t t x2x2 x1x1

18. Examples of Convex Sets Semidefinite Cone {X | X 0} a T Xa ≥ 0, for all a  R n All eigenvalues of X are non-negative a T X 1 a ≥ 0 a T X 2 a ≥ 0 a T (cX 1 + (1-c)X 2 )a ≥ 0

19. Operations that Preserve Convexity Intersection Polyhedron / Polytope

20. Operations that Preserve Convexity Intersection

21. Operations that Preserve Convexity Affine Transformation x  Ax + b

22. Convex Function x g(x) Blue point always lies above red point x1x1 x2x2

23. Convex Function x g(x) g( c x 1 + (1 - c) x 2 ) ≤ c g(x 1 ) + (1 - c) g(x 2 ) x1x1 x2x2 Domain of g(.) has to be convex

24. Convex Function x g(x) x1x1 x2x2 -g(.) is concave g( c x 1 + (1 - c) x 2 ) ≤ c g(x 1 ) + (1 - c) g(x 2 )

25. Convex Function Once-differentiable functions g(y) +  g(y) T (x - y) ≤ g(x) x g(x) (y,g(y)) g(y) +  g(y) T (x - y) Twice-differentiable functions  2 g(x) 0

26. Convex Function and Convex Sets x g(x) Epigraph of a convex function is a convex set

27. Examples of Convex Functions Linear function a T x p-Norm functions (x 1 p + x 2 p + x n p ) 1/p, p ≥ 1 Quadratic functions x T Q x Q 0

28. Operations that Preserve Convexity Non-negative weighted sum x g 1 (x) w1w1 x g 2 (x) + w 2 + …. x T Q x + a T x + b Q 0

29. Operations that Preserve Convexity Pointwise maximum x g 1 (x) max x g 2 (x), Pointwise minimum of concave functions is concave

30. Convex Optimization min g 0 (x) s.t. g i (x) ≤ 0 h i (x) = 0 Objective function Inequality constraints Equality constraints Objective function is convex  Feasible region is convex 

31. Linear Programming min g 0 (x) s.t. g i (x) ≤ 0 h i (x) = 0 Objective function Inequality constraints Equality constraints min g 0 T x s.t. g i T x ≤ 0 h i T x = 0 Linear function Linear constraints

32. Quadratic Programming min g 0 (x) s.t. g i (x) ≤ 0 h i (x) = 0 Objective function Inequality constraints Equality constraints min x T Qx + a T x + b s.t. g i T x ≤ 0 h i T x = 0 Quadratic function Linear constraints

33. Second-Order Cone Programming min g 0 (x) s.t. g i (x) ≤ 0 h i (x) = 0 Objective function Inequality constraints Equality constraints min g 0 T x s.t. x T Q i x + a i T x + b i ≤ 0 h i T x = 0 Linear function Quadratic constraints Linear constraints

34. Semidefinite Programming min g 0 (x) s.t. g i (x) ≤ 0 h i (x) = 0 Objective function Inequality constraints Equality constraints min Q  X s.t. X 0 A i  X = 0 Linear function Semidefinite constraints Linear constraints

35. Outline Convex Optimization Integer Programming Formulation Convex Relaxations Comparison Generalization of Results

36. Integer Programming Formulation 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘ 0 ’ Label ‘ 1 ’ Unary Cost Unary Cost Vector u = [ 5 Cost of V 1 = 0 2 Cost of V 1 = 1 ; 2 4 ] Labeling = {1, 0}

37. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘ 0 ’ Label ‘ 1 ’ Unary Cost Unary Cost Vector u = [ 5 2 ; 2 4 ] T Label vector x = [ -1 V 1  0 1 V 1 = 1 ; 1 -1 ] T Recall that the aim is to find the optimal x Integer Programming Formulation Labeling = {1, 0}

38. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘ 0 ’ Label ‘ 1 ’ Unary Cost Unary Cost Vector u = [ 5 2 ; 2 4 ] T Label vector x = [ -11; 1 -1 ] T Sum of Unary Costs = 1 2 ∑ i u i (1 + x i ) Integer Programming Formulation Labeling = {1, 0}

39. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘ 0 ’ Label ‘ 1 ’ Pairwise Cost 0 Cost of V 1 = 0 and V 1 = 0 0 00 0 Cost of V 1 = 0 and V 2 = 0 3 Cost of V 1 = 0 and V 2 = 1 10 00 00 10 30 Pairwise Cost Matrix P Integer Programming Formulation Labeling = {1, 0}

40. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘ 0 ’ Label ‘ 1 ’ Pairwise Cost Pairwise Cost Matrix P 00 00 0 3 10 00 00 10 30 Sum of Pairwise Costs 1 4 ∑ ij P ij (1 + x i )(1+x j ) Integer Programming Formulation Labeling = {1, 0}

41. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘ 0 ’ Label ‘ 1 ’ Pairwise Cost Pairwise Cost Matrix P 00 00 0 3 10 00 00 10 30 Sum of Pairwise Costs 1 4 ∑ ij P ij (1 + x i +x j + x i x j ) 1 4 ∑ ij P ij (1 + x i + x j + X ij )= X = x x T X ij = x i x j Integer Programming Formulation Labeling = {1, 0}

42. Constraints Uniqueness Constraint ∑ x i = 2 - |L| i  V a Integer Constraints x i  {-1,1} X = x x T Integer Programming Formulation

43. x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  {-1,1} X = x x T Convex Non-Convex Integer Programming Formulation

44. Outline Convex Optimization Integer Programming Formulation Convex Relaxations –Linear Programming (LP-S) –Semidefinite Programming (SDP-L) –Second Order Cone Programming (SOCP-MS) Comparison Generalization of Results

45. LP-S x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  {-1,1} X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

46. LP-S x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

47. LP-S X = x x T Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  V b

48. LP-S x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

49. LP-S x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1], Retain Convex Part Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  V b LP-S

50. Outline Convex Optimization Integer Programming Formulation Convex Relaxations –Linear Programming (LP-S) –Semidefinite Programming (SDP-L) –Second Order Cone Programming (SOCP-MS) Comparison Generalization of Results

51. SDP-L x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  {-1,1} X = x x T Retain Convex Part Lasserre, 2000 Relax Non-Convex Constraint

52. SDP-L x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X = x x T Retain Convex Part Relax Non-Convex Constraint Lasserre, 2000

53. x1x1 x2x2 xnxn 1... 1x1x1 x2x2... xnxn 1xTxT x X = Rank = 1 X ii = 1 Positive Semidefinite Convex Non-Convex SDP-L

54. x1x1 x2x2 xnxn 1... 1x1x1 x2x2... xnxn X ii = 1 Positive Semidefinite Convex SDP-L 1xTxT x X =

55. Schur’s Complement AB BTBT C = I0 B T A -1 I A0 0 C - B T A -1 B IA -1 B 0 I 0 A 0 C -B T A -1 B 0

56. X - xx T 0 1xTxT x X = 10 x I 10 0 X - xx T IxTxT 0 1 Schur ’ s Complement SDP-L

57. x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X = x x T Retain Convex Part Relax Non-Convex Constraint Lasserre, 2000

58. SDP-L x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] Retain Convex Part X ii = 1 X - xx T 0 Accurate SDP-L Inefficient Lasserre, 2000

59. Outline Convex Optimization Integer Programming Formulation Convex Relaxations –Linear Programming (LP-S) –Semidefinite Programming (SDP-L) –Second Order Cone Programming (SOCP-MS) Comparison Generalization of Results

60. SOCP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X ii = 1 X - xx T 0 Derive SOCP relaxation from the SDP relaxation Further Relaxation

61. 1-D Example X - xx T 0 X - x 2 ≥ 0 For two semidefinite matrices, Frobenius inner product is non-negative A A  0 x 2  X SOC of the form || v || 2  st = 1

62. 2-D Example X 11 X 12 X 21 X 22 1X 12 1 = X = x1x1x1x1 x1x2x1x2 x2x1x2x1 x2x2x2x2 xx T = x12x12 x1x2x1x2 x1x2x1x2 = x22x22

63. 2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2  0 10 00 X 12 -x 1 x 2 1 - x 2 2 x 1 2  1 -1  x 1  1 C 1   0 C 1 0 

64. 2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2  0 00 01 X 12 -x 1 x 2 1 - x 2 2 C 2   0 C 2 0  x 2 2  1 -1  x 2  1

65. 2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2  0 11 11 X 12 -x 1 x 2 1 - x 2 2 C 3   0 C 3 0  (x 1 + x 2 ) 2  2 + 2X 12 SOC of the form || v || 2  st

66. 2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2  0 1 1 X 12 -x 1 x 2 1 - x 2 2 C 4   0 C 4 0  (x 1 - x 2 ) 2  2 - 2X 12 SOC of the form || v || 2  st

67. SOCP Relaxation Consider a matrix C 1 = UU T 0 (X - xx T ) ||U T x || 2  X  C 1 C 1   0 Continue for C 2, C 3, …, C n SOC of the form || v || 2  st Kim and Kojima, 2000

68. SOCP Relaxation How many constraints for SOCP = SDP ? Infinite. For all C 0 Specify constraints similar to the 2-D example xixi xjxj X ij (x i + x j ) 2  2 + 2X ij (x i + x j ) 2  2 - 2X ij

69. SOCP-MS x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X ii = 1 X - xx T 0 Muramatsu and Suzuki, 2003

70. SOCP-MS x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] (x i + x j ) 2  2 + 2X ij (x i - x j ) 2  2 - 2X ij Specified only when P ij  0 Muramatsu and Suzuki, 2003

71. Outline Convex Optimization Integer Programming Formulation Convex Relaxations Comparison Generalization of Results Kumar, Kolmogorov and Torr, JMLR 2010

72. Dominating Relaxation For all MAP Estimation problem (u, P) A dominates B A B ≥ Dominating relaxations are better

73. Equivalent Relaxations A dominates B A B = B dominates A For all MAP Estimation problem (u, P)

74. Strictly Dominating Relaxation A dominates B A B > B does not dominate A For at least one MAP Estimation problem (u, P)

75. SOCP-MS (x i + x j ) 2  2 + 2X ij (x i - x j ) 2  2 - 2X ij Muramatsu and Suzuki, 2003 P ij ≥ 0 (x i + x j ) 2 2 - 1 X ij = P ij < 0 (x i - x j ) 2 2 1 - X ij = SOCP-MS is a QP Same as QP by Ravikumar and Lafferty, 2005 SOCP-MS ≡ QP-RL

76. LP-S vs. SOCP-MS Differ in the way they relax X = xx T X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  V b LP-S (x i + x j ) 2  2 + 2X ij (x i - x j ) 2  2 - 2X ij SOCP-MS F(LP-S) F(SOCP-MS)

77. LP-S vs. SOCP-MS LP-S strictly dominates SOCP-MS LP-S strictly dominates QP-RL Where have we gone wrong? A Quick Recap !

78. Recap of SOCP-MS xixi xjxj X ij 11 11 C = (x i + x j ) 2  2 + 2X ij

79. Recap of SOCP-MS xixi xjxj X ij 1 1 C = (x i - x j ) 2  2 - 2X ij Can we use different C matrices ?? Can we use a different subgraph ??

80. Outline Convex Optimization Integer Programming Formulation Convex Relaxations Comparison Generalization of Results –SOCP Relaxations on Trees –SOCP Relaxations on Cycles Kumar, Kolmogorov and Torr, JMLR 2010

81. SOCP Relaxations on Trees Choose any arbitrary tree

82. SOCP Relaxations on Trees Choose any arbitrary C 0 Repeat over trees to get relaxation SOCP-T LP-S strictly dominates SOCP-T LP-S strictly dominates QP-T

83. Outline Convex Optimization Integer Programming Formulation Convex Relaxations Comparison Generalization of Results –SOCP Relaxations on Trees –SOCP Relaxations on Cycles Kumar, Kolmogorov and Torr, JMLR 2010

84. SOCP Relaxations on Cycles Choose an arbitrary even cycle P ij ≥ 0 P ij ≤ 0OR

85. SOCP Relaxations on Cycles Choose any arbitrary C 0 Repeat over even cycles to get relaxation SOCP-E LP-S strictly dominates SOCP-E LP-S strictly dominates QP-E

86. SOCP Relaxations on Cycles True for odd cycles with P ij ≤ 0 True for odd cycles with P ij ≤ 0 for only one edge True for odd cycles with P ij ≥ 0 for only one edge True for all combinations of above cases

87. The SOCP-C Relaxation Include all LP-S constraintsTrue SOCP ab 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 Submodular Non-submodularSubmodular

88. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 Frustrated Cycle

89. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 LP-S Solution ab 1 0 0 0 0 bc 0 11 0 0 0 ca 0 0 0 0 1 1 1 Objective Function = 0

90. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 LP-S Solution ab 1 0 0 0 0 bc 0 11 0 0 0 ca 0 0 0 0 1 1 1 Define an SOC Constraint using C = 1

91. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 LP-S Solution ab 1 0 0 0 0 bc 0 11 0 0 0 ca 0 0 0 0 1 1 1 (x i + x j + x k ) 2  3 + 2 (X ij + X jk + X ki )

92. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 SOCP-C Solution Objective Function = 0.75 SOCP-C strictly dominates LP-S ab 0.8 0.6 -0.8 -0.6 0.6 -0.6 bc 0.6 -0.6 0 0 ca 0 0 0.8 0.6 -0.6 0.4 -0.4 0.4 -0.3 0.3 -0.3

93. The SOCP-Q Relaxation Include all cycle inequalitiesTrue SOCP ab cd Clique of size n Define an SOCP Constraint using C = 1 (Σ x i ) 2 ≤ n + (Σ X ij ) SOCP-Q strictly dominates LP-S SOCP-Q strictly dominates SOCP-C

94. 4-Neighbourhood MRF Test SOCP-C 50 binary MRFs of size 30x30 u ≈ N (0,1) P ≈ N (0,σ 2 )

95. 4-Neighbourhood MRF σ = 2.5

96. 8-Neighbourhood MRF Test SOCP-Q 50 binary MRFs of size 30x30 u ≈ N (0,1) P ≈ N (0,σ 2 )

97. 8-Neighbourhood MRF σ = 1.125

98. Conclusions Large class of SOCP/QP dominated by LP-S New SOCP relaxations dominate LP-S But better LP relaxations exist