Rounding-based Moves for Metric Labeling M. Pawan Kumar Center for Visual Computing Ecole Centrale Paris

Post Metric Labeling Random variables V = {v 1, v 2, …, v n } Label set L = {l 1, l 2, …, l h } Labelings quantatively distinguished by energy E(y) Labeling y ∈ L n Unary potential of variable v a ∈ V ∑ a θ a (y a )

Post Metric Labeling Random variables V = {v 1, v 2, …, v n } Label set L = {l 1, l 2, …, l h } Labelings quantatively distinguished by energy E(y) Labeling y ∈ L n Pairwise potential of variables (v a,v b ) ∑ a θ a (y a )+ ∑ (a,b) w ab d(y a,y b ) w ab is non-negatived(.,.) is a metric distance function min y

Post Existing Work –Move-Making Algorithms (Efficient) –Linear Programming Relaxation (Accurate) Rounding-based Moves –Equivalence –Complete Rounding and Complete Move –Interval Rounding and Interval Move –Hierarchical Rounding and Hierarchical Move Outline

Post Expansion Algorithm Sky House Tree Ground Initialize with TreeExpand GroundExpand HouseExpand Sky Variables take label l α or retain current label Boykov, Veksler and Zabih, ICCV 1999

Post Move-Making Algorithms Iteration t Define S t ⊆ L n containing current labeling y t ∑ a θ a (y a )+ ∑ (a,b) w ab d(y a,y b ) argmin y s.t. y ∈ S t Sometimes it can even be solved exactly Above problem is easier than original problem y t+1 = Start with an initial labeling y 0

Post Existing Work –Move-Making Algorithms (Efficient) –Linear Programming Relaxation (Accurate) Rounding-based Moves –Equivalence –Complete Rounding and Complete Move –Interval Rounding and Interval Move –Hierarchical Rounding and Hierarchical Move Outline

Post Linear Programming Relaxation Chekuri, Khanna, Naor and Zosin, SODA 2001 Binary indicator x a (i) ∈ {0,1} If variable ‘a’ takes the label ‘i’ then x a (i) = 1 ∑ i x a (i) = 1Each variable ‘a’ takes one label Similarly, binary indicator x ab (i,k) ∈ {0,1}

Post Linear Programming Relaxation Minimize a linear function over feasible x Indicators x a (i), x ab (i,k)  {0,1} Relaxed x a (i), x ab (i,k)  [0,1] Rounding Chekuri, Khanna, Naor and Zosin, SODA 2001

Post Existing Work –Move-Making Algorithms (Efficient) –Linear Programming Relaxation (Accurate) Rounding-based Moves –Equivalence –Complete Rounding and Complete Move –Interval Rounding and Interval Move –Hierarchical Rounding and Hierarchical Move Outline

Post Move-Making Bound y*: Optimal Labelingy: Estimated Labeling Σ a θ a (y a ) + Σ (a,b) w ab d(y a,y b ) Σ a θ a (y* a ) + Σ (a,b) w ab d(y* a,y* b ) ≥

Post Move-Making Bound y*: Optimal Labelingy: Estimated Labeling Σ a θ a (y a ) + Σ (a,b) w ab d(y a,y b ) Σ a θ a (y* a ) + Σ (a,b) w ab d(y* a,y* b ) B ≤ For all possible values of θ a (i) and w ab

Post Rounding Approximation x*: LP Optimal Solutionx: Rounded Solution Σ a Σ i θ a (i)x a (i) + Σ (a,b) Σ (i,k) w ab d(i,k)x ab (i,k) ≥ Σ a Σ i θ a (i)x* a (i) + Σ (a,b) Σ (i,k) w ab d(i,k)x* ab (i,k)

Post Rounding Approximation x*: LP Optimal Solutionx: Rounded Solution Σ a Σ i θ a (i)x a (i) + Σ (a,b) Σ (i,k) w ab d(i,k)x ab (i,k) ≤ Σ a Σ i θ a (i)x* a (i) + Σ (a,b) Σ (i,k) w ab d(i,k)x* ab (i,k) A For all possible values of θ a (i) and w ab

Post Equivalence For any known rounding with approximation A there exists a move-making algorithm such that the move-making bound B = A We know how to design such an algorithm

Post Existing Work –Move-Making Algorithms (Efficient) –Linear Programming Relaxation (Accurate) Rounding-based Moves –Equivalence –Complete Rounding and Complete Move –Interval Rounding and Interval Move –Hierarchical Rounding and Hierarchical Move Outline

Post Complete Rounding Treat x* a (i)  [0,1] as probability that y a = l i Cumulative probability z a (i) = Σ j≤i x* a (j) 0z a (1) z a (2) z a (h) = 1 z a (k) z a (i) Generate a random number r  (0,1] Assign the label next to r r

Post Complete Rounding - Example 0z a (1) z a (4) z a (3) z a (2) z b (1) z b (4) z b (3) z b (2) z c (1) z c (4) z c (3) z c (2) r r r

Post Equivalent Move Complete Move !!

Post Complete Move Iteration t Define S t ⊆ L n ∑ a θ a (y a )+ ∑ (a,b) w ab d(y a,y b ) argmin y s.t. y ∈ S t y t+1 = Start with an initial labeling y 0

Post Complete Move Iteration t Define S t = L n ∑ a θ a (y a )+ ∑ (a,b) w ab d(y a,y b ) argmin y s.t. y ∈ S t How do we solve this problem? Above problem is the same as the original problem y t+1 = Start with an initial labeling y 0

Post Complete Move Define S t = L n ∑ a θ a (y a )+ ∑ (a,b) w ab d’(y a,y b ) argmin y s.t. y ∈ S t How do we solve this problem? Above problem is the same as the original problem y t+1 =

Post Complete Move Define S t = L n ∑ a θ a (y a )+ ∑ (a,b) w ab d’(y a,y b ) argmin y s.t. y ∈ S t Obtained by solving a small LP Submodular overestimation d’ of d y t+1 =

Post Submodular Overestimation max i,k d’(l i,l k )/d(l i,l k )min d’ d’(l i,l k ) ≥ d(l i,l k ) s.t. d’(l i,l k+1 ) + d’(l i+1,l k ) ≥ d(l i,l k ) + d(l i+1,l k+1 )

Post Submodular Overestimation bmin d’ d’(l i,l k ) ≥ d(l i,l k ) s.t. d’(l i,l k+1 ) + d’(l i+1,l k ) ≥ d(l i,l k ) + d(l i+1,l k+1 ) bd(l i,l k ) ≥ d’(l i,l k ) Dual provides worst-case instance for complete rounding

Post Existing Work –Move-Making Algorithms (Efficient) –Linear Programming Relaxation (Accurate) Rounding-based Moves –Equivalence –Complete Rounding and Complete Move –Interval Rounding and Interval Move –Hierarchical Rounding and Hierarchical Move Outline

Post Interval Rounding Treat x* a (i)  [0,1] as probability that y a = l i Cumulative probability z a (i) = Σ j≤i x* a (j) 0z a (1) z a (2) z a (h) = 1 z a (k) z a (i) Choose an interval of length h’

Post Interval Rounding Treat x* a (i)  [0,1] as probability that y a = l i Cumulative probability z a (i) = Σ j≤i x* a (j) r Generate a random number r  (0,1] Assign the label next to r if it is within the interval z a (k)-z a (i) 0 Choose an interval of length h’ REPEAT

Post Interval Rounding - Example 0z a (1) z a (4) z a (3) z a (2) z b (1) z b (4) z b (3) z b (2) z c (1) z c (4) z c (3) z c (2)

Post Interval Rounding - Example 0z a (1) z a (2) z b (1) z b (2) z c (1) z c (2) r r r

Post Interval Rounding - Example 0z a (1) z a (4) z a (3) z a (2) z b (1) z b (4) z b (3) z b (2) z c (1) z c (4) z c (3) z c (2)

Post Interval Rounding - Example 0 z c (1) z c (4) z c (3) z c (2)

Post Interval Rounding - Example 0 z c (3) z c (2) r -z c (1)

Post Interval Rounding - Example 0z a (1) z a (4) z a (3) z a (2) z b (1) z b (4) z b (3) z b (2) z c (1) z c (4) z c (3) z c (2)

Post Equivalent Move Interval Move !!

Post Interval Move Iteration t y ∈ S t iff y a = y t a or y a ∈ interval of labels ∑ a θ a (y a )+ ∑ (a,b) w ab d(y a,y b ) argmin y s.t. y ∈ S t y t+1 = Start with an initial labeling y 0 Choose an interval of labels of length h’ How do we solve this problem?

Post Interval Move Iteration t y ∈ S t iff y a = y t a or y a ∈ interval of labels ∑ a θ a (y a )+ ∑ (a,b) w ab d’(y a,y b ) argmin y s.t. y ∈ S t y t+1 = Start with an initial labeling y 0 Choose an interval of labels of length h’ Submodular overestimation d’ of d

Post Existing Work –Move-Making Algorithms (Efficient) –Linear Programming Relaxation (Accurate) Rounding-based Moves –Equivalence –Complete Rounding and Complete Move –Interval Rounding and Interval Move –Hierarchical Rounding and Hierarchical Move Outline

Post Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Hierarchical clustering of labels (e.g. r-HST metrics)

Post Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Assign variables to labels L 1, L 2 or L 3 Move down the hierarchy until the leaf level

Post Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Assign variables to labels l 1, l 2 or l 3

Post Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Assign variables to labels l 4, l 5 or l 6

Post Hierarchical Rounding L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Assign variables to labels l 7, l 8 or l 9

Post Equivalent Move Hierarchical Move !!

Post Hierarchical Move L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Hierarchical clustering of labels (e.g. r-HST metrics)

Post Hierarchical Move L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Obtain labeling y 1 restricted to labels {l 1,l 2,l 3 }

Post Hierarchical Move L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Obtain labeling y 2 restricted to labels {l 4,l 5,l 6 }

Post Hierarchical Move L1L1 L2L2 l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l7l7 l8l8 l9l9 L3L3 Obtain labeling y 3 restricted to labels {l 7,l 8,l 9 }

Post Hierarchical Move L1L1 L2L2 L3L3 VaVa VbVb y 1 (a) y 2 (a) y 3 (a) Move up the hierarchy until we reach the root y 1 (b) y 2 (b) y 3 (b)

Questions?

Post Simple Example - Rounding θ a (1)x a (1) + θ a (2)x a (2) + θ b (1)x b (1) + θ b (2)x b (2)min x≥0 + d(1,1)x ab (1,1) + d(1,2)x ab (1,2) + d(2,1)x ab (2,1) + d(2,2)x ab (2,2) x a (1) + x a (2) = 1s.t. x b (1) + x b (2) = 1 x ab (1,1) + x ab (1,2) = x a (1) x ab (2,1) + x ab (2,2) = x a (2) x ab (1,1) + x ab (2,1) = x b (1) x ab (1,2) + x ab (2,2) = x b (2)

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a is assigned label l 1 ?x* a (1) Probability that V a is assigned label l 2 ?x* a (2)

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 1 and l 1 ? min{x* a (1), x* b (1)}

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 1 and l 1 ? min{x* ab (1,1)+x* ab (1,2), x* ab (1,1) + x* ab (2,1)} x* ab (1,1) + min{x* ab (1,2), x* ab (2,1)}

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 1 and l 2 ? max{0,x* a (1) - x* b (1)}

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 1 and l 2 ? x* ab (1,2) - min{x* ab (1,2), x* ab (2,1)} max{0,x* ab (1,2) - x* ab (2,1)}

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 2 and l 1 ? max{0,x* b (1) - x* a (1)}

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 2 and l 1 ? x* ab (2,1) - min{x* ab (1,2), x* ab (2,1)} max{0,x* ab (2,1) - x* ab (1,2)}

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 2 and l 2 ? 1-max{x* a (1), x* b (1)}

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 2 and l 2 ? min{x* a (2), x* b (2)}

Post Simple Example - Rounding x* a (1) + x* a (2) = 1 x* a (1) 0 x* b (1) + x* b (2) = 1 x* b (1) 0 Generate a uniform random number r  (0,1] Assign the label next to r r r Probability that V a and V b are assigned l 2 and l 2 ? min{x* ab (2,2)+x* ab (1,2), x* ab (2,2) + x* ab (2,1)} x* ab (2,2) + min{x* ab (1,2), x* ab (2,1)}

Post Simple Example - Move θ a (y a ) + θ b (y b ) min y + d(y a,y b ) y a,y b ∈ {1,2} If d is submodular, solve using graph cuts Otherwise

Post Simple Example - Move θ a (y a ) + θ b (y b ) min y + d’(y a,y b ) y a,y b ∈ {0,1} If d is submodular, solve using graph cuts Otherwiseuse submodular overestimation d’ Estimate d’ by minimizing distortion

Post Simple Example - Move bmin d' d’(1,1) ≤ b d(1,1)s.t.d’(1,2) ≤ b d(1,2) d’(2,1) ≤ b d(2,1)d’(2,2) ≤ b d(2,2) d(1,1) ≤ d’(1,1)d(1,2) ≤ d’(1,2) d(2,1) ≤ d’(2,1)d(2,2) ≤ d’(2,2) d’(1,1) + d’(2,2) ≤ d’(2,1) + d’(2,2) Dual LP provides worst-case rounding example LP in the variables d’(i,k)

Post Simple Example - Move d(1,1)β(1,1)+d(1,2)β(1,2)+d(2,1)β(2,1)+d(2,2)β(2,2)min α,β,γ≥0 s.t.d(1,1)α(1,1)+d(1,2)α(1,2)+d(2,1)α(2,1)+d(2,2)α(2,2) = 1 β(1,1) = α(1,1) + γ β(1,2) = α(1,2) - γ β(2,1) = α(2,1) - γ β(2,2) = α(2,2) + γ Set x ab *(i,k) = α(i,k) Set γ = min{x ab *(1,2), x ab *(2,1)}