1. Efficient Graph Cut Optimization for Full CRFs with Quantized Edges
Olga Veksler

2. Contents Introduction Quantized edge Full-CRF
CRF with Potts pairwise potentials
sparsely connected
fully connected CRF (Full-CRF)
Gaussian edge weights
mean field optimization
Quantized edge Full-CRF
optimization
two label case
multi-label case
connection to Gaussian Edge CRF
application to semantic segmentation

3. CRF Energy with Potts Potentials
high energy
low energy
Find labeling x minimizing energy
Optimization
Solved exactly in binary label case with a graph cut
NP hard in multi-label case
expansion algorithm approximation (factor of 2) [Boykov et.al.’TPAMI01]

4. Sparse vs. Fully Connected CRF
Sparsely connected CRFs
4, 8, or small neighbourhood connected
TRWS [Kolmogorov ‘TPAMI2006] or expansion algorithms work well
length regularization [Boykov&Kolmogorov’ICCV2003]
Fully connected CRFs [Krahenbul&Koltun’NIPS2011]
all pixels are neighbors, n pixels, O(n2) edges
naïve application of expansion algorithm, TRWS, etc. is not efficient
regularization properties?

5. Full CRF with Uniform Weights
Cardinality regularization
labels in {0,1}
n pixels in the image, k pixels assigned to label 1

6. Full CRF with Uniform Weights
Cardinality regularization
labels in {0,1}
n pixels in the image, k pixels assigned to label 1
pairwise energy is

7. Full CRF with Uniform Weights
Cardinality regularization
labels in {0,1}
n pixels in the image, k pixels assigned to label 1
pairwise energy is
w ·
(n - k)
· k k n
same pairwise cost
Efficient algorithm
for each k
find the k pixels with lowest cost for label 1
compute total energy
chose k corresponding to the smallest energy

8. Fully Connected CRFs Fully connected CRFs [Krahenbul&Koltun’NIPS2011]
assumes Gaussian edge weights
efficient mean field inference
approximate bilateral filter [Paris&Durand, IJCV’2009]
mean field is not a very effective optimization method [Weiss’2001]

9. CRF+CNN Combination CNN can give blurred not pixel precise results
Sharpen with CRF
Chen et.al. ICLR’2015 as post processing
Or unified framework [Zheng et.al., ICCV’2015]

10. Quantized Edge Fully Connected CRFs
Gaussian Edge Weights [Krahenbul&Koltun’NIPS2011] d
superpixels
Quantized edge weights m

11. Quantized Edge Fully Connected CRFs
Edge weights depend on superpixel membership
do not have to be Gaussian weighted
superpixels

12. Quantized Edge Fully Connected CRFs
input image
superpixels
Interior/exterior weights
interior weights
exterior weights

13. Optimization for 2 labels: Superpixel
Consider one superpixel
internal edges weight w
n – k pixels 1 w
k pixels w
Internal pairwise cost is k·(n - k) w·
if vary green superpixel labeling, cost changes only with k

14. Optimization for 2 labels: Two Superpixels
Consider two superpixels
external edge weight ww 1
k pixels
n – k pixels 1
h pixels
m – h pixels ww ww
External pairwise cost is k·(m - h) + (n - k) ·h
ww · [ ]
if vary green superpixel labeling, cost changes only with k
If k pixels in green superpixel are assigned to label 1, they must be those that have the smallest cost for label 1

15. Optimization for 2 labels: Overview
Convert binary energy in pixel domain to multi-label energy in smaller superpixel domain
new variables are the superpixels
new cardinality labels are 0,1,…,superpixelSize
assume unary cost for label 0 is 0
old labels {0,1}
8 pixels
4 pixels
new variables
new labels
{0,...,4}
{0,...,10}
{0,...,8}
{0,...,11}
10 pixels
11 pixels

16. Optimization for 2 labels: Conversion
Sort pixels in each superpixel by increasing cost of being assigned to label 1
New variables are the superpixels
New labels are 0,1,…,superpixelSize
Label k assigned to superpixel means k smallest cost pixels in that superpixel are assigned to label 1 in original problem
111 1 1 1 3 1
sort pixels in each superpixel by cost of being assigned to label 1 1 4 2
s1=2
s2=3
s3=6
s4=11

17. Unary Cost for Transformed Problem
Unary cost for green superpixel to have label k
account for unary terms of the original binary problem 1
k pixels
n – k pixels w

18. Unary Cost for Transformed Problem
Unary cost for green superpixel to have label k
account for unary terms of the original binary problem
account for internal pairwise terms of original binary problem
k pixels
n – k pixels w
cost of 1
cost of 1
+ w·k·(n - k)
cost of 1
cost of 1

19. Pairwise Cost for Transformed Problem
green superpixel to have label k, purple superpixel to have label h
models external pairwise cost 1
k pixels
n – k pixels
h pixels
m – h pixels ww
k·(m - h) + (n - k) ·h
ww · [ ]

20. Optimization for Transformed Problem
Pairwise cost for
green superpixel to have label k, purple superpixel to have label h
k·(m - h) + (n - k) ·h
ww · [ ]
Can be rewritten as unary terms + (h - k)2
Can optimize exactly with [Ishikawa’TPAMI04]
number of edges is quadratic in the number of labels
memory inefficient, time complexity almost as bad as the original binary problem
Or with [Ajanthan’CVPR2016]
Memory efficient, but time complexity almost as bad as the original binary problem

21. Optimization: Jump Moves
Pairwise cost is quadratic (h - k)2 5 4 3 2 1 7 5 3 2 4 7
+1 jump
-1 jump 5 2 1 3 4 7
Jump moves [Veksler’99, Kolmogorov &Shioura’09]
each move is optimization of binary energy
efficient: number of edges is linear in the number of pixels
give exact minimum efficiently if unary terms are also convex
Our unary terms are not convex
jump moves do not work well in practice

22. Optmization: Expansion Moves5 4 3 2 1 7 5 2 3 4 7
2-expansion 5 2 1 7
1 - expansion
Expansion moves [Boykov et.al., PAMI’2001]
each expansion move is optimization of binary energy
efficient: number of edges is linear in the number of pixels
Not submodular for quadratic potential
but does find the optimum in the overwhelming majority of cases

23. Multi-Label Quantized Full-CRF
Apply expansion algorithm
each expansion step is optimization of binary energy
already know how to optimize 2-label Edge Quantized Full CRF
problem
meaning of label 0 is not fixed for expansion algorithm
solution
construct new superpixels according to the current labeling β α γ ε δ
old superpixels
new superpixels

24. Final Algorithm, Multi-Label Case
for each α ∊ L
perform α-expansion
1. compute new superpixels
2. transform binary expansion energy from pixel domain to multi-label energy in superpixel domain
3. for each β ∊ Ltransformed
perform β -expansion
until convergence

25. Connection to Gaussian Full-CRF
Quantized edge CRF gets close to Gaussian edge CRF
as number of superpixels increases
as beta increases

26. Connection to Gaussian Full-CRF
Regularization properties of full Gaussian CRF not well understood
“all pixels connected”, “preserves fine structure”
ground truth
Gaussian Full-CRF results
Quantized Edge CRF model helps to understand Gaussian CRF
If k pixels in a superpixel split from the rest, shape of the split does not matter
equal cost labelings

27. Optimization Results: Full-CRF, 2 labels
validation fold of Pascal 2012 dataset
reduced to 70x70 pixels
2 most likely labels
global optimum with a graph cut
our method is exact in 89% of cases
running time in seconds

28. Optimization Results: Full-CRF, multilabel
validation fold of Pascal 2012 dataset
21 labels
our method is always better than mean-field, ICM
running time in seconds

29. Full CRFs :Semantic Segmentaiton
Test fold of Pascal 2012 dataset
21 labels
Overall IOU
Unary
Superpixels
Mean Field
Ours

30. Full CRFs :Semantic Segmentaiton
(a) Input image
(b) superpixels
(c) unary terms
(d) our result
(e) ground truth

31. Summary Quantized Edge Full CRF model
Approximation to Gaussian Edge CRF
Helps to understand properties of Gaussian Edge CRF
Efficient optimization of Quantized Edge full CRF with graph cuts
Transform the original problem to a smaller domain
Optimization quality significantly better than mean field inference