1. Efficient Inference for Fully-Connected CRFs with Stationarity
Yimeng Zhang, Tsuhan Chen
CVPR 2012

2. Summary
Explore object-class segmentation with fully-connected CRF models
Only restriction on pairwise terms is `spatial stationarity’ (i.e. depend on relative locations)
Show how efficient inference can be achieved by
Using a QP formulation
Using FFT to calculate gradients in complexity (linear in) O(NlogN)

3. Fully-connected CRF model
General pairwise CRF model:
Image I
Class labeling, X:
Label set, L:
V = set of pixels, N_i = neighbourhood of pixel i, Z(I) = partition function, psi = potential functions

4. Fully-connected CRF model
General pairwise CRF model:
In fully-connected CRF, for all i, N_i = V

5. Unary Potential
Unary potential generates a score for each object class per pixel (TextonBoost)

6. Pairwise Potential
Pairwise potential measures compatibility of the labels at each pair of pixels
Combines spatial and colour contrast factors

7. Pairwise Potential
Colour contrast:
Spatial term:

8. Pairwise Potential
Learning the spatial term

9. MAP inference using QP relaxation
Introduce a binary indicator variable for each pixel and label
MAP inference expressed as a quadratic integer program, and relaxed to give the QP

10. MAP inference using QP relaxation
QP relaxation has been proved to be tight in all cases (Ravikumar ICML 2006 [24])
Moreover, it is convex whenever matrix of edge-weights is negative-definite
Additive bound for non-convex case
QP requires O(KN) variables, LP requires (K^2E)

11. MAP inference using QP relaxation
Gradient
Derive fixed-point update by forming Lagrangian and setting its derivative to 0

12. Illustration of QP updates

13. Efficiently evaluating the gradient
Required summation
Would be a convolution without the color term
With color term is requires 5D-filtering
Can be approximated by clustering into C color clusters, => C convolutions across

14. Efficiently evaluating the gradient
Hence, for the case x_i = x_j, we need to evaluate
Instead, evaluate for C clusters (C = 10 to 15)
where
Finally, interpolate

15. Update complexity
FFTs of each spatial filters can be calculated in advance (K^2 filters)
At each update, we require C FFTs calculating, O(CNlogN)
K^2 convolutions are needed, each requiring a multiplication, O(K^2CN)
Terms can be added in Fourier domain, => only KC inverse FFTs needed, O(KCNlogN)
Run-time per iteration < 0.1s for 213x320 pixels (+ downsampling by factor of 5)

16. MSRC synthetic experiment
Unary terms randomized
Spatial distributions set to ground-truth

17. MSRC synthetic experiment
Running times

18. Sowerby synthetic experiment

19. MSRC full experiment Use TextonBoost unary potentials
Compare with several other CRFs with same unaries
Grid only
Grid + P^N (Kohli, CVPR 2008)
Grid + P^N + Cooccurrence (Ladickỳ, ECCV 2010)
Fully-connected + Gaussian spatial (Krähenbühl, NIPS 2011)

20. MSRC full experiment
Qualitative comparison

21. MSRC full experiment Quantitative comparison Overall Per-class
Timing: 2-8s per image