Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr

Talk Outline Dynamic Graph Cuts –Fast reestimation of cut –Useful for video –Object specific segmentation Estimation of non submodular MRFs –Relaxations beyond linear!!

Example: Video Segmentation

Model Based Segmentation ImageSegmentationPose Estimate [Images courtesy: M. Black, L. Sigal]

Min-Marginals Image MAP SolutionBelief - Foreground Low smoothness High smoothness Moderate smoothness Colour Scale

Uses of Min marginals Estimate of true marginals (uncertainty) Parameter Learning. Get best n solutions easily.

Dynamic Graph Cuts PBPB SBSB cheaper operation computationally expensive operation Simpler problem P B* differences between A and B similar PAPA SASA solve

First segmentation problem MAP solution GaGa Our Algorithm GbGb second segmentation problem Maximum flow residual graph ( G r ) G` difference between G a and G b updated residual graph

The Max-flow Problem - Edge capacity and flow balance constraints Computing the st-mincut from Max-flow algorithms Notation - Residual capacity (edge capacity – current flow) - Augmenting path Simple Augmenting Path based Algorithms - Repeatedly find augmenting paths and push flow. - Saturated edges constitute the st-mincut. [Ford-Fulkerson Theorem] Sink (1) Source (0) a1a1 a2a

9 + α 4 + α Adding a constant to both the t-edges of a node does not change the edges constituting the st-mincut. Key Observation Sink (1) Source (0) a1a1 a2a E (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 E*(a 1,a 2 ) = E(a 1,a 2 ) + α(a 2 +ā 2 ) = E(a 1,a 2 ) + α [a 2 +ā 2 =1] Reparametrization

9 + α 4 All reparametrizations of the graph are sums of these two types. Other type of reparametrization Sink (1) Source (0) a1a1 a2a α 2 + α 1 - α Reparametrization, second type Both maintain the solution and add a constant α to the energy.

Reparametrization Nice result (easy to prove) All other reparametrizations can be viewed in terms of these two basic operations. Proof in Hammer, and also in one of Vlads recent papers.

s G t original graph 0/9 0/7 0/5 0/20/4 0/1 xixi xjxj flow/residual capacity Graph Re-parameterization

s G t original graph 0/9 0/7 0/5 0/20/4 0/1 xixi xjxj flow/residual capacity Graph Re-parameterization t residual graph xixi xjxj 0/12 5/2 3/2 1/0 2/04/0 st-mincut Compute Maxflow GrGr Edges cut

Update t-edge Capacities s GrGr t residual graph xixi xjxj 0/12 5/2 3/2 1/0 2/04/0

Update t-edge Capacities s GrGr t residual graph xixi xjxj 0/12 5/2 3/2 1/0 2/04/0 capacity changes from 7 to 4

Update t-edge Capacities s G` t updated residual graph xixi xjxj 0/12 5/-1 3/2 1/0 2/04/0 capacity changes from 7 to 4 edge capacity constraint violated! (flow > capacity) = 5 – 4 = 1 excess flow (e) = flow – new capacity

add e to both t-edges connected to node i Update t-edge Capacities s G` t updated residual graph xixi xjxj 0/12 3/2 1/0 2/04/0 capacity changes from 7 to 4 edge capacity constraint violated! (flow > capacity) = 5 – 4 = 1 excess flow (e) = flow – new capacity 5/-1

Update t-edge Capacities s G` t updated residual graph xixi xjxj 0/12 3/2 1/0 4/0 capacity changes from 7 to 4 excess flow (e) = flow – new capacity add e to both t-edges connected to node i = 5 – 4 = 1 5/0 2/1 edge capacity constraint violated! (flow > capacity)

Update n-edge Capacities s GrGr t residual graph xixi xjxj 0/12 5/2 3/2 1/0 2/04/0 Capacity changes from 5 to 2

Update n-edge Capacities s t Updated residual graph xixi xjxj 0/12 5/2 3/-1 1/0 2/04/0 G` Capacity changes from 5 to 2 - edge capacity constraint violated!

Update n-edge Capacities s t Updated residual graph xixi xjxj 0/12 5/2 3/-1 1/0 2/04/0 G` Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint

Update n-edge Capacities s t Updated residual graph xixi xjxj 0/11 5/2 2/0 1/0 2/04/0 excess deficiency G` Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint - causes flow imbalance!

Update n-edge Capacities s t Updated residual graph xixi xjxj 0/11 5/2 2/0 1/0 2/04/0 deficiency excess G` Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint - causes flow imbalance! Push excess flow to/from the terminals Create capacity by adding α = excess to both t-edges.

Update n-edge Capacities Updated residual graph Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint - causes flow imbalance! Push excess flow to the terminals Create capacity by adding α = excess to both t-edges. G` xixi xjxj 0/11 5/3 2/0 3/04/1 s t

Update n-edge Capacities Updated residual graph Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint - causes flow imbalance! Push excess flow to the terminals Create capacity by adding α = excess to both t-edges. G` xixi xjxj 0/11 5/3 2/0 3/04/1 s t

Complexity analysis of MRF Update Operations MRF Energy Operation Graph OperationComplexity modifying a unary term modifying a pair-wise term adding a latent variable delete a latent variable Updating a t-edge capacity Updating a n-edge capacity adding a node set the capacities of all edges of a node zero O(1) O(k)* *requires k edge update operations where k is degree of the node

Finding augmenting paths is time consuming. Dual-tree maxflow algorithm [Boykov & Kolmogorov PAMI 2004] -Reuses search trees after each augmentation. -Empirically shown to be substantially faster. Our Idea –Reuse search trees from previous graph cut computation –Saves us search tree creation tree time [O(#edges)] –Search trees have to be modified to make them consistent with new graphs – Constrain the search of augmenting paths New paths must contain at least one updated edge Improving the Algorithm

Reusing Search Trees c = measure of change in the energy –Running time Dynamic algorithm (c + re-create search tree ) Improved dynamic algorithm (c) Video Segmentation Example - Duplicate image frames (No time is needed)

Dynamic Graph Cut vs Active Cuts Our method flow recycling AC cut recycling Both methods: Tree recycling

Experimental Analysis MRF consisting of 2x10 5 latent variables connected in a 4-neighborhood. Running time of the dynamic algorithm

Part II SOCP for MRF

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Labelling m = {1, -1, -1, 1} Random Variables V = {V 1,..,V 4 } Label Set L = {-1,1}

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Cost(m) = 2

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Cost(m) = 2 + 1

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Cost(m) =

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Cost(m) =

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Cost(m) =

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Cost(m) =

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Cost(m) =

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Cost(m) = = 13 Minimum Cost Labelling = MAP estimate Pr(m) exp(-Cost(m))

Aim Accurate MAP estimation of pairwise Markov random fields V1V1 V2V2 V3V3 V4V4 Label -1 Label 1 Objectives Applicable to all types of neighbourhood relationships Applicable to all forms of pairwise costs Guaranteed to converge (Convex approximation)

Motivation Subgraph Matching - Torr , Schellewald et al G1G1 G2G2 Unary costs are uniform V2V2 V3V3 V1V1 MRF A B C D A B C D A B C D

Motivation Subgraph Matching - Torr , Schellewald et al G1G1 G2G2 | d(m i,m j ) - d(V i,V j ) | < 1 2 YESNO Potts Model Pairwise Costs

Motivation V2V2 V3V3 V1V1 MRF A B C D A B C D A B C D Subgraph Matching - Torr , Schellewald et al

Motivation V2V2 V3V3 V1V1 MRF A B C D A B C D A B C D Subgraph Matching - Torr , Schellewald et al

Motivation Matching Pictorial Structures - Felzenszwalb et al Part likelihoodSpatial Prior Outline Texture Image P1P1 P3P3 P2P2 (x,y,, ) MRF

Motivation Image P1P1 P3P3 P2P2 (x,y,, ) MRF Unary potentials are negative log likelihoods Valid pairwise configuration Potts Model Matching Pictorial Structures - Felzenszwalb et al YESNO

Motivation P1P1 P3P3 P2P2 (x,y,, ) Pr(Cow)Image Unary potentials are negative log likelihoods Matching Pictorial Structures - Felzenszwalb et al Valid pairwise configuration Potts Model 1 2 YESNO

Outline Integer Programming Formulation Previous Work Our Approach –Second Order Cone Programming (SOCP) –SOCP Relaxation –Robust Truncated Model Applications –Subgraph Matching –Pictorial Structures

Integer Programming Formulation V1V1 V2V2 Label -1 Label 1 Unary Cost Unary Cost Vector u = [ 5 2 ; 2 4 ] T Labelling m = {1, -1} Label vector x = [ -1 V 1 =-1 1 V 1 = 1 ; 1 -1 ] T Recall that the aim is to find the optimal x

Integer Programming Formulation V1V1 V2V2 Label -1 Label 1 Unary Cost Unary Cost Vector u = [ 5 2 ; 2 4 ] T Labelling m = {1, -1} Label vector x = [ -11; 1 -1 ] T Sum of Unary Costs = 1 2 i u i (1 + x i )

Integer Programming Formulation V1V1 V2V2 Label -1 Label 1 Pairwise Cost Labelling m = {1, -1} 0 Cost of V 1 = -1 and V 1 = Cost of V 1 = -1 and V 2 = -1 3 Cost of V 1 = 0-1and V 2 = Pairwise Cost Matrix P

Integer Programming Formulation V1V1 V2V2 Label -1 Label 1 Pairwise Cost Labelling m = {1, -1} Pairwise Cost Matrix P Sum of Pairwise Costs 1 4 ij P ij (1 + x i )(1+x j )

Integer Programming Formulation V1V1 V2V2 Label 0 Label 1 Pairwise Cost Labelling m = {1, 0} Pairwise Cost Matrix P Sum of Pairwise Costs 1 4 ij P ij (1 + x i +x j + x i x j ) 1 4 ij P ij (1 + x i + x j + X ij )= X = x x T X ij = x i x j

Integer Programming Formulation Constraints Each variable should be assigned a unique label x i = 2 - |L| i V a Marginalization constraint X ij = (2 - |L|) x i j V b

Integer Programming Formulation Chekuri et al., SODA 2001 x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i {-1,1} X = x x T Convex Non-Convex

Key Point In modern optimization the issue is not linear vs non linear but convex vs nonconvex We want to find a convex and good relaxation of the integer program.

Outline Integer Programming Formulation Previous Work Our Approach –Second Order Cone Programming (SOCP) –SOCP Relaxation –Robust Truncated Model Applications –Subgraph Matching –Pictorial Structures

Linear Programming Formulation x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i {-1,1} X = x x T Chekuri et al., SODA 2001 Retain Convex Part Relax Non-convex Constraint

Linear Programming Formulation x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i [-1,1] X = x x T Chekuri et al., SODA 2001 Retain Convex Part Relax Non-convex Constraint

Linear Programming Formulation x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i [-1,1] Chekuri et al., SODA 2001 Retain Convex Part X becomes a variable to be optimized

Feasible Region (IP) x {-1,1}, X = x 2 Linear Programming Formulation Feasible Region for X.

Feasible Region (IP) Feasible Region (Relaxation 1) x {-1,1}, X = x 2 x [-1,1], X = x 2 Linear Programming Formulation Feasible Region for X.

Feasible Region (IP) Feasible Region (Relaxation 1) Feasible Region (Relaxation 2) x {-1,1}, X = x 2 x [-1,1], X = x 2 x [-1,1] Linear Programming Formulation Feasible Region for X.

Linear Programming Formulation Bounded algorithms proposed by Chekuri et al, SODA expansion - Komodakis and Tziritas, ICCV 2005 TRW - Wainwright et al., NIPS 2002 TRW-S - Kolmogorov, AISTATS 2005 Efficient because it uses Linear Programming Not accurate

Semidefinite Programming Formulation x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i {-1,1} X = x x T Lovasz and Schrijver, SIAM Optimization, 1990 Retain Convex Part Relax Non-convex Constraint

x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i [-1,1] X = x x T Semidefinite Programming Formulation Retain Convex Part Relax Non-convex Constraint Lovasz and Schrijver, SIAM Optimization, 1990

Semidefinite Programming Formulation x1x1 x2x2 xnxn x1x1 x2x2... xnxn 1xTxT x X = Rank = 1 X ii = 1 Positive Semidefinite Convex Non-Convex

Semidefinite Programming Formulation x1x1 x2x2 xnxn x1x1 x2x2... xnxn 1xTxT x X = X ii = 1 Positive Semidefinite Convex

Schurs Complement AB BTBT C = I0 B T A -1 I A0 0 C - B T A -1 B IA -1 B 0 I 0 A 0 C -B T A -1 B 0

Semidefinite Programming Formulation X - xx T 0 1xTxT x X = 10 x I 10 0 X - xx T IxTxT 0 1 Schurs Complement

x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i [-1,1] X = x x T Semidefinite Programming Formulation Relax Non-convex Constraint Retain Convex Part Lovasz and Schrijver, SIAM Optimization, 1990

x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i [-1,1] Semidefinite Programming Formulation X ii = 1 X - xx T 0 Retain Convex Part Lovasz and Schrijver, SIAM Optimization, 1990

Feasible Region (IP) x {-1,1}, X = x 2 Semidefinite Programming Formulation Feasible Region for X.

Feasible Region (IP) Feasible Region (Relaxation 1) x {-1,1}, X = x 2 x [-1,1], X = x 2 Semidefinite Programming Formulation Feasible Region for X.

Feasible Region (IP) Feasible Region (Relaxation 1) Feasible Region (Relaxation 2) x {-1,1}, X = x 2 x [-1,1], X = x 2 x [-1,1], X x 2 Semidefinite Programming Formulation Feasible Region for X.

Semidefinite Programming Formulation Formulated by Lovasz and Schrijver, 1990 Finds a full X matrix Max-cut - Goemans and Williamson, JACM 1995 Max-k-cut - de Klerk et al, 2000 Torr AI Stats for labeling problem (2003 TR 2002) Accurate, but not efficient as Semidefinite Programming algorithms slow

Previous Work - Overview LPSDP Examples TRW-S, -expansion Max-k-Cut Torr 2003 AccuracyLowHigh EfficiencyHighLow Is there a Middle Path ???

Outline Integer Programming Formulation Previous Work Our Approach –Second Order Cone Programming (SOCP) –SOCP Relaxation –Robust Truncated Model Applications –Subgraph Matching –Pictorial Structures

Second Order Cone Programming Second Order Cone || v || t OR || v || 2 st x 2 + y 2 z 2

Minimize f T x Subject to || A i x+ b i || <= c i T x + d i i = 1, …, L Linear Objective Function Affine mapping of Second Order Cone (SOC) Constraints are SOC of n i dimensions Feasible regions are intersections of conic regions Second Order Cone Programming

|| v || t tIv vTvT t 0 LP SOCP SDP = 10 vTvT I tI0 0 t 2 - v T v Iv 0 1 Schurs Complement

Outline Integer Programming Formulation Previous Work Our Approach –Second Order Cone Programming (SOCP) –SOCP Relaxation –Robust Truncated Model Applications –Subgraph Matching –Pictorial Structures

First quick definition: Matrix Dot Product AB = ij A ij B ij A 11 A 12 A 21 A 22 B 11 B 12 B 21 B 22 = A 11 B 11 + A 12 B 12 + A 21 B 21 + A 22 B 22

SDP Relaxation x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i [-1,1] X ii = 1 X - xx T 0 We will derive SOCP relaxation from the SDP relaxation Further Relaxation

1-D Example X - xx T 0 X - x 2 0 For two semidefinite matrices, the dot product is non-negative A A 0 x 2 X SOC of the form || v || 2 st, s is a scalar constant.

Feasible Region (IP) Feasible Region (Relaxation 1) Feasible Region (Relaxation 2) x {-1,1}, X = x 2 x [-1,1], X = x 2 x [-1,1], X x 2 SOCP Relaxation For 1D: Same as the SDP formulation Feasible Region for X.

2-D Example X 11 X 12 X 21 X 22 1X 12 1 = X = x1x1x1x1 x1x2x1x2 x2x1x2x1 x2x2x2x2 xx T = x12x12 x1x2x1x2 x1x2x1x2 = x22x22

2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x x 2 2 x x 1 1 C 1. 0 C 1 0

2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2 C X 12 -x 1 x x 2 2 x LP Relaxation -1 x 2 1 C 2 0

2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2 C X 12 -x 1 x x 2 2 (x 1 + x 2 ) X 12 SOC of the form || v || 2 st C 3 0

2-D Example (X - xx T ) 1 - x 1 2 X 12 -x 1 x 2 C X 12 -x 1 x x 2 2 (x 1 - x 2 ) X 12 SOC of the form || v || 2 st C 4 0

General form of SOC constraints Consider a matrix C 1 = UU T 0 (X - xx T ) ||U T x || 2 X. C 1 C 1. 0 Continue for C 2, C 3, …, C n SOC of the form || v || 2 st Kim and Kojima, 2000

SOCP Relaxation How many constraints for SOCP = SDP ? Infinite. For all C 0 We specify constraints similar to the 2-D example

SOCP Relaxation Muramatsu and Suzuki, Constraints hold for the above semidefinite matrices

SOCP Relaxation Muramatsu and Suzuki, a + b + c+ d a 0 b 0 c 0 d 0 Constraints hold for the linear combination

SOCP Relaxation Muramatsu and Suzuki, 2001 a+c+dc-d b+c+d a 0 b 0 c 0 d 0 Includes all semidefinite matrices where Diagonal elements Off-diagonal elements

SOCP Relaxation - A x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i [-1,1] X ii = 1 X - xx T 0

SOCP Relaxation - A x* = argmin 1 2 u i (1 + x i ) P ij (1 + x i + x j + X ij ) x i = 2 - |L| i V a X ij = (2 - |L|) x i j V b x i [-1,1] (x i + x j ) X ij (x i - x j ) X ij Specified only when P ij 0 i.e. sparse!!

Triangular Inequality At least two of x i, x j and x k have the same sign At least one of X ij, X jk, X ik is equal to one X ij + X jk + X ik -1 X ij - X jk - X ik -1 -X ij - X jk + X ik -1 -X ij + X jk - X ik -1 SOCP-B = SOCP-A + Triangular Inequalities

Outline Integer Programming Formulation Previous Work Our Approach –Second Order Cone Programming (SOCP) –SOCP Relaxation –Robust Truncated Model Applications –Subgraph Matching –Pictorial Structures

Robust Truncated Model Pairwise cost of incompatible labels is truncated Potts ModelTruncated Linear Model Truncated Quadratic Model Robust to noise Widely used in Computer Vision - Segmentation, Stereo

Robust Truncated Model Pairwise Cost Matrix can be made sparse P = [ ] Q = [ ] Reparameterization Sparse Q matrix Fewer constraints

Compatibility Constraint Q(m a, m b ) < 0 for variables V a and V b Relaxation Q ij (1 + x i + x j + X ij ) < 0 SOCP-C = SOCP-B + Compatibility Constraints

SOCP Relaxation More accurate than LP More efficient than SDP Time complexity - O( |V| 3 |L| 3 ) Same as LP Approximate algorithms exist for LP relaxation We use |V| 10 and |L| 200

Outline Integer Programming Formulation Previous Work Our Approach –Second Order Cone Programming (SOCP) –SOCP Relaxation –Robust Truncated Model Applications –Subgraph Matching –Pictorial Structures

Subgraph Matching Subgraph Matching - Torr , Schellewald et al G1G1 G2G2 Unary costs are uniform V2V2 V3V3 V1V1 MRF A B C D A B C D A B C D Pairwise costs form a Potts model

Subgraph Matching 1000 pairs of graphs G 1 and G 2 #vertices in G 2 - between 20 and 30 #vertices in G * #vertices in G 2 5% noise to the position of vertices NP-hard problem

Subgraph Matching Method Time (sec) Accuracy (%) LP LBP GBP SDP-A SOCP-A SOCP-B SOCP-C

Outline Integer Programming Formulation Previous Work Our Approach –Second Order Cone Programming (SOCP) –SOCP Relaxation –Robust Truncated Model Applications –Subgraph Matching –Pictorial Structures

Pictorial Structures Image P1P1 P3P3 P2P2 (x,y,, ) MRF Matching Pictorial Structures - Felzenszwalb et al Outline Texture

Pictorial Structures Image P1P1 P3P3 P2P2 (x,y,, ) MRF Unary costs are negative log likelihoods Pairwise costs form a Potts model | V | = 10| L | = 200

Pictorial Structures ROC Curves for 450 +ve and ve images

Pictorial Structures ROC Curves for 450 +ve and ve images

Conclusions We presented an SOCP relaxation to solve MRF More efficient than SDP More accurate than LP, LBP, GBP #variables can be reduced for Robust Truncated Model Provides excellent results for subgraph matching and pictorial structures

Future Work Quality of solution –Additive bounds exist –Multiplicative bounds for special cases ?? –What are good Cs. Message passing algorithm ?? –Similar to TRW-S or -expansion –To handle image sized MRF