1. An Analysis of Convex Relaxations M. Pawan Kumar Vladimir Kolmogorov Philip Torr for MAP Estimation

2. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Labelling m = {1, 0, 0, 1} Random Variables V = {V 1,...,V 4 } Label Set L = {0, 1} To analyze convex relaxations for MAP estimation

3. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 To analyze convex relaxations for MAP estimation

4. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 + 1 To analyze convex relaxations for MAP estimation

5. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 + 1 + 2 To analyze convex relaxations for MAP estimation

6. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 + 1 + 2 + 1 To analyze convex relaxations for MAP estimation

7. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 + 1 + 2 + 1 + 3 To analyze convex relaxations for MAP estimation

8. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 To analyze convex relaxations for MAP estimation

9. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3 To analyze convex relaxations for MAP estimation

10. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13 Minimum Cost Labelling = MAP estimate Pr(m)  exp(-Cost(m)) To analyze convex relaxations for MAP estimation

11. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13 Which approximate algorithm is the best? NP-hard problem To analyze convex relaxations for MAP estimation

12. Aim 2 5 4 2 6 3 3 7 0 1 1 0 0 2 3 1 1 41 0 V1V1 V2V2 V3V3 V4V4 Label ‘0’ Label ‘1’ Objectives Compare existing convex relaxations – LP, QP and SOCP Develop new relaxations based on the comparison To analyze convex relaxations for MAP estimation

13. Outline Integer Programming Formulation Existing Relaxations Comparison Generalization of Results Two New SOCP Relaxations

14. Integer Programming Formulation 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘0’ Label ‘1’ Unary Cost Unary Cost Vector u = [ 5 Cost of V 1 = 0 2 Cost of V 1 = 1 ; 2 4 ] Labelling m = {1, 0}

15. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘0’ Label ‘1’ Unary Cost Unary Cost Vector u = [ 5 2 ; 2 4 ] T Labelling m = {1, 0} Label vector x = [ -1 V 1  0 1 V 1 = 1 ; 1 -1 ] T Recall that the aim is to find the optimal x Integer Programming Formulation

16. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘0’ Label ‘1’ Unary Cost Unary Cost Vector u = [ 5 2 ; 2 4 ] T Labelling m = {1, 0} Label vector x = [ -11; 1 -1 ] T Sum of Unary Costs = 1 2 ∑ i u i (1 + x i ) Integer Programming Formulation

17. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘0’ Label ‘1’ Pairwise Cost Labelling m = {1, 0} 0 Cost of V 1 = 0 and V 1 = 0 0 00 0 Cost of V 1 = 0 and V 2 = 0 3 Cost of V 1 = 0 and V 2 = 1 10 00 00 10 30 Pairwise Cost Matrix P Integer Programming Formulation

18. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘0’ Label ‘1’ Pairwise Cost Labelling m = {1, 0} Pairwise Cost Matrix P 00 00 0 3 10 00 00 10 30 Sum of Pairwise Costs 1 4 ∑ ij P ij (1 + x i )(1+x j ) Integer Programming Formulation

19. 2 5 4 2 0 1 3 0 V1V1 V2V2 Label ‘0’ Label ‘1’ Pairwise Cost Labelling m = {1, 0} Pairwise Cost Matrix P 00 00 0 3 10 00 00 10 30 Sum of Pairwise Costs 1 4 ∑ ij P ij (1 + x i +x j + x i x j ) 1 4 ∑ ij P ij (1 + x i + x j + X ij )= X = x x T X ij = x i x j Integer Programming Formulation

20. Constraints Uniqueness Constraint ∑ x i = 2 - |L| i  V a Integer Constraints x i  {-1,1} X = x x T Integer Programming Formulation

21. x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  {-1,1} X = x x T Convex Non-Convex Integer Programming Formulation

22. Outline Integer Programming Formulation Existing Relaxations –Linear Programming (LP-S) –Semidefinite Programming (SDP-L) –Second Order Cone Programming (SOCP-MS) Comparison Generalization of Results Two New SOCP Relaxations

23. LP-S x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  {-1,1} X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

24. LP-S x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

25. LP-S X = x x T Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  V b

26. LP-S x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X = x x T Retain Convex Part Schlesinger, 1976 Relax Non-Convex Constraint

27. LP-S x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1], Retain Convex Part Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  V b LP-S

28. Outline Integer Programming Formulation Existing Relaxations –Linear Programming (LP-S) –Semidefinite Programming (SDP-L) –Second Order Cone Programming (SOCP-MS) Comparison Generalization of Results Two New SOCP Relaxations Experiments

29. SDP-L x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  {-1,1} X = x x T Retain Convex Part Lasserre, 2000 Relax Non-Convex Constraint

30. SDP-L x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X = x x T Retain Convex Part Relax Non-Convex Constraint Lasserre, 2000

31. x1x1 x2x2 xnxn 1... 1x1x1 x2x2... xnxn 1xTxT x X = Rank = 1 X ii = 1 Positive Semidefinite Convex Non-Convex SDP-L

32. x1x1 x2x2 xnxn 1... 1x1x1 x2x2... xnxn 1xTxT x X = X ii = 1 Positive Semidefinite Convex SDP-L

33. Schur’s Complement AB BTBT C = I0 B T A -1 I A0 0 C - B T A -1 B IA -1 B 0 I 0 A 0 C -B T A -1 B 0

34. X - xx T 0 1xTxT x X = 10 x I 10 0 X - xx T IxTxT 0 1 Schur’s Complement SDP-L

35. x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X = x x T Retain Convex Part Relax Non-Convex Constraint Lasserre, 2000

36. SDP-L x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] Retain Convex Part X ii = 1 X - xx T 0 Accurate SDP-L Inefficient Lasserre, 2000

37. Outline Integer Programming Formulation Existing Relaxations –Linear Programming (LP-S) –Semidefinite Programming (SDP-L) –Second Order Cone Programming (SOCP-MS) Comparison Generalization of Results Two New SOCP Relaxations

38. SOCP Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X ii = 1 X - xx T 0 Derive SOCP relaxation from the SDP relaxation Further Relaxation

39. 1-D Example X - xx T 0 X - x 2 ≥ 0 For two semidefinite matrices, Frobenius inner product is non-negative A A  0 x 2  X SOC of the form || v || 2  st = 1

40. 2-D Example X 11 X 12 X 21 X 22 1X 12 1 = X = x1x1x1x1 x1x2x1x2 x2x1x2x1 x2x2x2x2 xx T = x12x12 x1x2x1x2 x1x2x1x2 = x22x22

41. 2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2.  0 10 00 X 12 -x 1 x 2 1 - x 2 2 x 1 2  1 -1  x 1  1 C 1.  0 C 1 0

42. 2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2 C 2.  0.  0 00 01 X 12 -x 1 x 2 1 - x 2 2 x 2 2  1 -1  x 2  1 C 2 0

43. 2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2 C 3.  0.  0 11 11 X 12 -x 1 x 2 1 - x 2 2 (x 1 + x 2 ) 2  2 + 2X 12 SOC of the form || v || 2  st C 3 0

44. 2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2 C 4.  0.  0 1 1 X 12 -x 1 x 2 1 - x 2 2 (x 1 - x 2 ) 2  2 - 2X 12 SOC of the form || v || 2  st C 4 0

45. SOCP Relaxation Consider a matrix C 1 = UU T 0 (X - xx T ) ||U T x || 2  X. C 1 C 1.  0 Continue for C 2, C 3, …, C n SOC of the form || v || 2  st Kim and Kojima, 2000

46. SOCP Relaxation How many constraints for SOCP = SDP ? Infinite. For all C 0 Specify constraints similar to the 2-D example xixi xjxj X ij (x i + x j ) 2  2 + 2X ij (x i + x j ) 2  2 - 2X ij

47. SOCP-MS x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] X ii = 1 X - xx T 0 Muramatsu and Suzuki, 2003

48. SOCP-MS x* = argmin 1 2 ∑ u i (1 + x i ) + 1 4 ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  V a x i  [-1,1] (x i + x j ) 2  2 + 2X ij (x i - x j ) 2  2 - 2X ij Specified only when P ij  0 Muramatsu and Suzuki, 2003

49. Outline Integer Programming Formulation Existing Relaxations Comparison Generalization of Results Two New SOCP Relaxations

50. Dominating Relaxation For all MAP Estimation problem (u, P) A dominates B A B ≥ Dominating relaxations are better

51. Equivalent Relaxations A dominates B A B = B dominates A For all MAP Estimation problem (u, P)

52. Strictly Dominating Relaxation A dominates B A B > B does not dominate A For at least one MAP Estimation problem (u, P)

53. SOCP-MS (x i + x j ) 2  2 + 2X ij (x i - x j ) 2  2 - 2X ij Muramatsu and Suzuki, 2003 P ij ≥ 0 (x i + x j ) 2 2 - 1 X ij = P ij < 0 (x i - x j ) 2 2 1 - X ij = SOCP-MS is a QP Same as QP by Ravikumar and Lafferty, 2005 SOCP-MS ≡ QP-RL

54. LP-S vs. SOCP-MS Differ in the way they relax X = xx T X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  V b LP-S (x i + x j ) 2  2 + 2X ij (x i - x j ) 2  2 - 2X ij SOCP-MS F(LP-S) F(SOCP-MS)

55. LP-S vs. SOCP-MS LP-S strictly dominates SOCP-MS LP-S strictly dominates QP-RL Where have we gone wrong? A Quick Recap !

56. Recap of SOCP-MS xixi xjxj X ij 11 11 C = (x i + x j ) 2  2 + 2X ij

57. Recap of SOCP-MS xixi xjxj X ij 1 1 C = (x i - x j ) 2  2 - 2X ij Can we use different C matrices ?? Can we use a different subgraph ??

58. Outline Integer Programming Formulation Existing Relaxations Comparison Generalization of Results –SOCP Relaxations on Trees –SOCP Relaxations on Cycles Two New SOCP Relaxations

59. SOCP Relaxations on Trees Choose any arbitrary tree

60. SOCP Relaxations on Trees Choose any arbitrary C 0 Repeat over trees to get relaxation SOCP-T LP-S strictly dominates SOCP-T LP-S strictly dominates QP-T

61. Outline Integer Programming Formulation Existing Relaxations Comparison Generalization of Results –SOCP Relaxations on Trees –SOCP Relaxations on Cycles Two New SOCP Relaxations

62. SOCP Relaxations on Cycles Choose an arbitrary even cycle P ij ≥ 0 P ij ≤ 0OR

63. SOCP Relaxations on Cycles Choose any arbitrary C 0 Repeat over even cycles to get relaxation SOCP-E LP-S strictly dominates SOCP-E LP-S strictly dominates QP-E

64. SOCP Relaxations on Cycles True for odd cycles with P ij ≤ 0 True for odd cycles with P ij ≤ 0 for only one edge True for odd cycles with P ij ≥ 0 for only one edge True for all combinations of above cases

65. Outline Integer Programming Formulation Existing Relaxations Comparison Generalization of Results Two New SOCP Relaxations –The SOCP-C Relaxation –The SOCP-Q Relaxation

66. The SOCP-C Relaxation Include all LP-S constraintsTrue SOCP ab 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 Submodular Non-submodularSubmodular

67. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 Frustrated Cycle

68. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 LP-S Solution ab 1 0 0 0 0 bc 0 11 0 0 0 ca 0 0 0 0 1 1 1 Objective Function = 0

69. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 LP-S Solution ab 1 0 0 0 0 bc 0 11 0 0 0 ca 0 0 0 0 1 1 1 Define an SOC Constraint using C = 1

70. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 LP-S Solution ab 1 0 0 0 0 bc 0 11 0 0 0 ca 0 0 0 0 1 1 1 (x i + x j + x k ) 2  3 + 2 (X ij + X jk + X ki )

71. The SOCP-C Relaxation ab Include all LP-S constraintsTrue SOCP 0 0 11 0 0 0 bc 1 0 00 0 0 0 ca 0 0 11 0 0 0 010 SOCP-C Solution Objective Function = 0.75 SOCP-C strictly dominates LP-S ab 0.8 0.6 -0.8 -0.6 0.6 -0.6 bc 0.6 -0.6 0 0 ca 0 0 0.8 0.6 -0.6 0.4 -0.4 0.4 -0.3 0.3 -0.3

72. Outline Integer Programming Formulation Existing Relaxations Comparison Generalization of Results Two New SOCP Relaxations –The SOCP-C Relaxation –The SOCP-Q Relaxation

73. The SOCP-Q Relaxation Include all cycle inequalitiesTrue SOCP ab cd Clique of size n Define an SOCP Constraint using C = 1 (Σ x i ) 2 ≤ n + (Σ X ij ) SOCP-Q strictly dominates LP-S SOCP-Q strictly dominates SOCP-C

74. 4-Neighbourhood MRF Test SOCP-C 50 binary MRFs of size 30x30 u ≈ N (0,1) P ≈ N (0,σ 2 )

75. 4-Neighbourhood MRF σ = 2.5

76. 8-Neighbourhood MRF Test SOCP-Q 50 binary MRFs of size 30x30 u ≈ N (0,1) P ≈ N (0,σ 2 )

77. 8-Neighbourhood MRF σ = 1.125

78. Conclusions Large class of SOCP/QP dominated by LP-S New SOCP relaxations dominate LP-S More experimental results in poster

79. Future Work Comparison with cycle inequalities Determine best SOC constraints Develop efficient algorithms for new relaxations

80. Questions ??