Efficiently Solving Convex Relaxations M. Pawan Kumar University of Oxford for MAP Estimation Philip Torr Oxford Brookes University

Aim a bcd Label ‘0’ Label ‘1’ Labelling m = {1, 0, 0, 1} Random Variables V = {a, b, c, d} Label Set L = {0, 1} To solve convex relaxations of MAP estimation Edges E = {(a, b), (b, c), (c, d)}

Aim Label ‘0’ Label ‘1’ Cost(m) = = 13 Approximate using Convex Relaxations Minimum Cost Labelling? NP-hard problem To solve convex relaxations of MAP estimation a bcd

Aim Label ‘0’ Label ‘1’ Objectives Solve tighter convex relaxations – LP and SOCP Handle large number of random variables, e.g. image pixels To solve convex relaxations of MAP estimation a bcd

Outline Integer Programming Formulation Linear Programming Relaxation Additional Constraints Solving the Convex Relaxations Results and Conclusions

Integer Programming Formulation a b Label ‘0’ Label ‘1’ Unary Cost Unary Cost Vector u = [ 5 Cost of a = 0 2 Cost of a = 1 ; 2 4 ] Labelling m = {1, 0}

Label ‘0’ Label ‘1’ Unary Cost Unary Cost Vector u = [ 5 2 ; 2 4 ] T Labelling m = {1, 0} Label vector x = [ -1 a  0 1 a = 1 ; 1 -1 ] T Recall that the aim is to find the optimal x Integer Programming Formulation a b

Label ‘0’ Label ‘1’ Unary Cost Unary Cost Vector u = [ 5 2 ; 2 4 ] T Labelling m = {1, 0} Label vector x = [ -11; 1 -1 ] T Sum of Unary Costs = 1 2 ∑ i u i (1 + x i ) Integer Programming Formulation a b

Label ‘0’ Label ‘1’ Pairwise Cost Labelling m = {1, 0} Pairwise Cost of a and a Cost of a = 0 and b = 0 3 Cost of a = 0 and b = Pairwise Cost Matrix P Integer Programming Formulation a b

Label ‘0’ Label ‘1’ Pairwise Cost Labelling m = {1, 0} Pairwise Cost Matrix P Sum of Pairwise Costs 1 4 ∑ ij P ij (1 + x i )(1+x j ) Integer Programming Formulation a b

Label ‘0’ Label ‘1’ Pairwise Cost Labelling m = {1, 0} Pairwise Cost Matrix P Sum of Pairwise Costs 1 4 ∑ ij P ij (1 + x i +x j + x i x j ) 1 4 ∑ ij P ij (1 + x i + x j + X ij )= X = x x T X ij = x i x j Integer Programming Formulation a b

Constraints Uniqueness Constraint ∑ x i = 2 - |L| i  a Integer Constraints x i  {-1,1} X = x x T Integer Programming Formulation

x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  a x i  {-1,1} X = x x T Convex Non-Convex Integer Programming Formulation

Outline Integer Programming Formulation Linear Programming Relaxation Additional Constraints Solving the Convex Relaxations Results and Conclusions

Linear Programming Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  a x i  {-1,1} X = x x T Retain Convex Part Schlesinger, 1976

Linear Programming Relaxation x* = argmin 1 2 ∑ u i (1 + x i ) ∑ P ij (1 + x i + x j + X ij ) ∑ x i = 2 - |L| i  a Retain Convex Part Schlesinger, 1976 X ij  [-1,1] 1 + x i + x j + X ij ≥ 0 ∑ X ij = (2 - |L|) x i j  b x i  [-1,1]

Dual of the LP Relaxation Wainwright et al., 2001 abc def ghi  = (u, P) abc def ghi abc def ghi 11 22 33 44 55 66 11 22 33 44 55 66  ii   ii  

Dual of the LP Relaxation Wainwright et al., 2001 abc def ghi  = (u, P) abc def ghi abc def ghi 11 22 33 44 55 66 Q(  1 )  ii   ii   Q(  2 ) Q(  3 ) Q(  4 )Q(  5 )Q(  6 ) max   i Q(  i ) Dual of LP

Tree-Reweighted Message Passing Kolmogorov, 2005 abc def ghi abc def ghi 11 22 33 44 55 66 Pick a variable cbaadg a Reparameterize such that u i are min-marginals u1u1 u2u2 u3u3 u4u4 Only one pass of belief propagation

Tree-Reweighted Message Passing Kolmogorov, 2005 abc def ghi abc def ghi 11 22 33 44 55 66 Pick a variable cbaadg a Average the unary costs (u 1 +u 3 )/2 Repeat for all variables (u 1 +u 3 )/2 (u 2 +u 4 )/2 TRW-S

Outline Integer Programming Formulation Linear Programming Relaxation Additional Constraints Solving the Convex Relaxations Results and Conclusions

Cycle Inequalities Chopra and Rao, 1991 a ef bc d a ed xixi xjxj xkxk At least two of them have the same sign x i x j x j x k x k x i X ij X jk X ki X = xx T At least one of them is 1 X ij + X jk + X ki  -1

Cycle Inequalities Chopra and Rao, 1991 a ef bc d X ij + X jk + X kl - X li  -2 xjxj b fe xixi xkxk c xlxl Generalizes to all cycles LP-C

Second-Order Cone Constraints Kumar et al., 2007 a ef bc d x c = xixi xjxj xkxk X c = 1 X ij X ik X jk X ik 1 1 X c = x c x c T X c x c x c T 1 (X c - x c x c T )  0 (x i +x j +x k ) 2 ≤ 3 + X ij + X jk + X ki SOCP-C

Second-Order Cone Constraints Kumar et al., 2007 a ef bc d 1 (X c - x c x c T )  0 SOCP-Q x c = xixi xjxj xkxk X c = 1 X ij X ik X jk X ik 1 1 xlxl X il X jl X kl X il X jl X kl 1

Outline Integer Programming Formulation Linear Programming Relaxation Additional Constraints Solving the Convex Relaxations Results and Conclusions

Modifying the Dual abc def ghi  ii   ii   max   i Q(  i ) 11 22 33 abc def ghi 44 55 66 adg beh cfi +   j s j 11 22 ab de bc ef de gh ef hi 33 44

Modifying TRW-S abc def ghi adg beh cfi ab de bc ef de gh ef hi Pick a variable --- a Pick a cycle/clique with a  ii   ii   max   i Q(  i )+  j s j Can be solved efficiently Run TRW-S for trees with a REPEAT

Properties of the Algorithm Algorithm satisfies the reparametrization constraint Value of dual never decreasesCONVERGENCE Solution satisfies Weak Tree Agreement (WTA) WTA not sufficient for convergence More accurate results than TRW-S

Outline Integer Programming Formulation Linear Programming Relaxation Additional Constraints Solving the Convex Relaxations Results and Conclusions

4-Neighbourhood MRF Test SOCP-C 50 binary MRFs of size 30x30 u ≈ N (0,1) P ≈ N (0,σ 2 ) Test LP-C

4-Neighbourhood MRF σ = 5 LP-C dominates SOCP-C

8-Neighbourhood MRF Test SOCP-Q 50 binary MRFs of size 30x30 u ≈ N (0,1) P ≈ N (0,σ 2 )

8-Neighbourhood MRF σ = 5 /  2 SOCP-Q dominates LP-C

Conclusions Modified LP dual to include more constraints Extended TRW-S to solve tighter dual Experiments show improvement More results in the poster

Future Work More efficient subroutines for solving cycles/cliques Using more accurate LP solvers - proximal projections Analysis of SOCP-C vs. LP-C

Questions?

Timings MethodTime/Iteration BP TRW-S LP-C SOCP-C SOCP-Q Linear in the number of variables!!

Video Segmentation KeyframeUser Segmentation Segment remaining video ….

Video Segmentation Belief Propagation Input

Video Segmentation  -swap Input

Video Segmentation  -expansion Input

Video Segmentation TRW-S Input

Video Segmentation LP-C Input

Video Segmentation SOCP-Q Input 000

4-Neighbourhood MRF σ = 1

4-Neighbourhood MRF σ = 2.5

8-Neighbourhood MRF σ = 1/  2

8-Neighbourhood MRF σ = 2.5 /  2