Efficient Graph Cut Optimization for Full CRFs with Quantized Edges Olga Veksler

Contents Introduction Quantized edge Full-CRF CRF with Potts pairwise potentials sparsely connected fully connected CRF (Full-CRF) Gaussian edge weights mean field optimization Quantized edge Full-CRF optimization two label case multi-label case connection to Gaussian Edge CRF application to semantic segmentation

CRF Energy with Potts Potentials high energy low energy Find labeling x minimizing energy Optimization Solved exactly in binary label case with a graph cut NP hard in multi-label case expansion algorithm approximation (factor of 2) [Boykov et.al.’TPAMI01]

Sparse vs. Fully Connected CRF Sparsely connected CRFs 4, 8, or small neighbourhood connected TRWS [Kolmogorov ‘TPAMI2006] or expansion algorithms work well length regularization [Boykov&Kolmogorov’ICCV2003] Fully connected CRFs [Krahenbul&Koltun’NIPS2011] all pixels are neighbors, n pixels, O(n2) edges naïve application of expansion algorithm, TRWS, etc. is not efficient regularization properties?

Full CRF with Uniform Weights Cardinality regularization labels in {0,1} n pixels in the image, k pixels assigned to label 1

Full CRF with Uniform Weights Cardinality regularization labels in {0,1} n pixels in the image, k pixels assigned to label 1 pairwise energy is

Full CRF with Uniform Weights Cardinality regularization labels in {0,1} n pixels in the image, k pixels assigned to label 1 pairwise energy is w · (n - k) · k k n same pairwise cost Efficient algorithm for each k find the k pixels with lowest cost for label 1 compute total energy chose k corresponding to the smallest energy

Fully Connected CRFs Fully connected CRFs [Krahenbul&Koltun’NIPS2011] assumes Gaussian edge weights efficient mean field inference approximate bilateral filter [Paris&Durand, IJCV’2009] mean field is not a very effective optimization method [Weiss’2001]

CRF+CNN Combination CNN can give blurred not pixel precise results Sharpen with CRF Chen et.al. ICLR’2015 as post processing Or unified framework [Zheng et.al., ICCV’2015]

Quantized Edge Fully Connected CRFs Gaussian Edge Weights [Krahenbul&Koltun’NIPS2011] d superpixels Quantized edge weights m

Quantized Edge Fully Connected CRFs Edge weights depend on superpixel membership do not have to be Gaussian weighted superpixels

Quantized Edge Fully Connected CRFs input image superpixels Interior/exterior weights interior weights exterior weights

Optimization for 2 labels: Superpixel Consider one superpixel internal edges weight w n – k pixels 1 w k pixels w Internal pairwise cost is k·(n - k) w· if vary green superpixel labeling, cost changes only with k

Optimization for 2 labels: Two Superpixels Consider two superpixels external edge weight ww 1 k pixels n – k pixels 1 h pixels m – h pixels ww ww External pairwise cost is k·(m - h) + (n - k) ·h ww · [ ] if vary green superpixel labeling, cost changes only with k If k pixels in green superpixel are assigned to label 1, they must be those that have the smallest cost for label 1

Optimization for 2 labels: Overview Convert binary energy in pixel domain to multi-label energy in smaller superpixel domain new variables are the superpixels new cardinality labels are 0,1,…,superpixelSize assume unary cost for label 0 is 0 old labels {0,1} 8 pixels 4 pixels new variables new labels {0,...,4} {0,...,10} {0,...,8} {0,...,11} 10 pixels 11 pixels

Optimization for 2 labels: Conversion Sort pixels in each superpixel by increasing cost of being assigned to label 1 New variables are the superpixels New labels are 0,1,…,superpixelSize Label k assigned to superpixel means k smallest cost pixels in that superpixel are assigned to label 1 in original problem 111 1 1 1 3 1 sort pixels in each superpixel by cost of being assigned to label 1 1 4 2 s1=2 s2=3 s3=6 s4=11

Unary Cost for Transformed Problem Unary cost for green superpixel to have label k account for unary terms of the original binary problem 1 k pixels n – k pixels w

Unary Cost for Transformed Problem Unary cost for green superpixel to have label k account for unary terms of the original binary problem account for internal pairwise terms of original binary problem k pixels n – k pixels w cost of 1 cost of 1 + w·k·(n - k) cost of 1 cost of 1

Pairwise Cost for Transformed Problem green superpixel to have label k, purple superpixel to have label h models external pairwise cost 1 k pixels n – k pixels h pixels m – h pixels ww k·(m - h) + (n - k) ·h ww · [ ]

Optimization for Transformed Problem Pairwise cost for green superpixel to have label k, purple superpixel to have label h k·(m - h) + (n - k) ·h ww · [ ] Can be rewritten as unary terms + (h - k)2 Can optimize exactly with [Ishikawa’TPAMI04] number of edges is quadratic in the number of labels memory inefficient, time complexity almost as bad as the original binary problem Or with [Ajanthan’CVPR2016] Memory efficient, but time complexity almost as bad as the original binary problem

Optimization: Jump Moves Pairwise cost is quadratic (h - k)2 5 4 3 2 1 7 5 3 2 4 7 +1 jump -1 jump 5 2 1 3 4 7 Jump moves [Veksler’99, Kolmogorov &Shioura’09] each move is optimization of binary energy efficient: number of edges is linear in the number of pixels give exact minimum efficiently if unary terms are also convex Our unary terms are not convex jump moves do not work well in practice

Optmization: Expansion Moves 5 4 3 2 1 7 5 2 3 4 7 2-expansion 5 2 1 7 1 - expansion Expansion moves [Boykov et.al., PAMI’2001] each expansion move is optimization of binary energy efficient: number of edges is linear in the number of pixels Not submodular for quadratic potential but does find the optimum in the overwhelming majority of cases

Multi-Label Quantized Full-CRF Apply expansion algorithm each expansion step is optimization of binary energy already know how to optimize 2-label Edge Quantized Full CRF problem meaning of label 0 is not fixed for expansion algorithm solution construct new superpixels according to the current labeling β α γ ε δ old superpixels new superpixels

Final Algorithm, Multi-Label Case for each α ∊ L perform α-expansion 1. compute new superpixels 2. transform binary expansion energy from pixel domain to multi-label energy in superpixel domain 3. for each β ∊ Ltransformed perform β -expansion until convergence

Connection to Gaussian Full-CRF Quantized edge CRF gets close to Gaussian edge CRF as number of superpixels increases as beta increases

Connection to Gaussian Full-CRF Regularization properties of full Gaussian CRF not well understood “all pixels connected”, “preserves fine structure” ground truth Gaussian Full-CRF results Quantized Edge CRF model helps to understand Gaussian CRF If k pixels in a superpixel split from the rest, shape of the split does not matter equal cost labelings

Optimization Results: Full-CRF, 2 labels validation fold of Pascal 2012 dataset reduced to 70x70 pixels 2 most likely labels global optimum with a graph cut our method is exact in 89% of cases running time in seconds

Optimization Results: Full-CRF, multilabel validation fold of Pascal 2012 dataset 21 labels our method is always better than mean-field, ICM running time in seconds

Full CRFs :Semantic Segmentaiton Test fold of Pascal 2012 dataset 21 labels Overall IOU Unary 67.143 Superpixels 65.89 Mean Field 67.3 Ours 67.75

Full CRFs :Semantic Segmentaiton (a) Input image (b) superpixels (c) unary terms (d) our result (e) ground truth

Summary Quantized Edge Full CRF model Approximation to Gaussian Edge CRF Helps to understand properties of Gaussian Edge CRF Efficient optimization of Quantized Edge full CRF with graph cuts Transform the original problem to a smaller domain Optimization quality significantly better than mean field inference