1. Associative Hierarchical CRFs for Object Class Image Segmentation Ľubor Ladický 1 1 Oxford Brookes University 2 Microsoft Research Cambridge Based on the work done with : Chris Russell 1, Pushmeet Kohli 1,2, Paul Sturgess 1, Karteek Alahari 1, Philip H.S. Torr 1

2. Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

3. Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

4. Object-class Segmentation problem MSRC ImageOur Result Aims to assign a label for each pixel of an image Classifier trained on the training set Performance evaluated on never seen test set

5. Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

6. Datasets MSRC Dataset 591 images 320 x 213 21 classes MSRC Image Ground Truth

7. Datasets VOC2009 Dataset 1499 training + 750 test images 500 x 375 20 foreground classes + 1 background class VOC Image Ground Truth

8. Datasets CamVid Dataset 367 training + 233 test video frames 960 x 720 11 classes CamVid Image Ground Truth

9. Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

10. Pairwise CRF over Pixels CRF Shotton ECCV06

11. Pairwise CRF over Pixels No quantization errors Lacks long range interactions Results oversmoothed Better performance on CamVid & MSRC datasets

12. Pairwise CRF over Segments Yang et al. CVPR07, Batra et al. CVPR08, Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Felzenschwalb, Huttenlocher, IJCV2004

13. Pairwise CRF over Segments Allows long range interactions Better performance for VOC dataset Can not recover from incorrect segmentation Impossible to obtain perfect unsupervised segmentation

14. Pairwise CRF over Segments Allows long range interactions Better performance for VOC dataset Can not recover from incorrect segmentation Impossible to obtain perfect unsupervised segmentation

15. Detection-driven Segmentation Larlus et al. CVPR08, Felzenszwalb et al. CVPR08, Vedaldi et al. ICCV09 GrabCut Detector

16. Detection-driven Segmentation The best performance for VOC dataset Can not be used for background classes Can not recover from incorrect detection

17. Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

18. Robust P N approach Kohli, Ladický, Torr CVPR08

19. Robust P N approach Segment consistency as a weak constraint Robust to misleading segmentations Allows multiple overlapping segmentations General formulation not used in application Unary and pairwise potentials only at the pixel level

20. Robust P N reformulation Ladický, Russell, Kohli, Torr ICCV09

21. Old formulation of Robust P N potential is equivalent to pairwise formulation where Robust P N reformulation Ladický, Russell, Kohli, Torr ICCV09

22. Associative Hierarchical CRF Allows unary potentials for region variables Allows pairwise potentials for region variables Allows multiple layers and multiple hierarchies Zhu et al. NIPS2008, Lim et al. ICCV2009, Hinton 2002

23. Analysis of the new model Ladický, Russell, Kohli, Torr ICCV09 Let's have one segmentation and potentials only over segment level

24. Analysis of the new model Let's have one segmentation and potentials only over segment level Energy of two pixels from the same clique is symmetric and semi-metric Minimum will be segment-consistent The cost of every segment consistent labelling is the same as the cost of the pairwise CRF labelling over segments Equivalent to pairwise CRF over segments Ladický, Russell, Kohli, Torr ICCV09

25. Associative Hierarchical CRF Merges information over multiple scales Allows multiple hierarchies Allows long range interactions Easy to learn weights Interlayer connection limited(?) to associative relationship Ladický, Russell, Kohli, Torr ICCV09

26. Comparison to other methods Ladický, Russell, Kohli, Torr ICCV09 Robust P N Segment CRFHierarchical CRF

27. Detectors in Associative Hierarchical CRFs Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted) GrabCutDetector Detector potential takes the form where

28. Detectors in Associative Hierarchical CRFs Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted) Special 2-label case of Robust P N Allows multiple overlapping detections Can recover from incorrect detection Same inference methods may be applied as for Associative Hierarchical CRFs We can distinguish between different instances of objects in the final result

29. Label co-occurrence in CRFs Ladický, Russell, Kohli, Torr CVPR10 (submitted) Possible labelling ? Why not ?

30. Label co-occurrence in CRFs Ladický, Russell, Kohli, Torr CVPR10 (submitted) Our requirements Global Energy function – no hard decisions Invariance to number of pixels taking given label Efficiency – should be tractable Standard approaches Hard decisions in the preprocessing step (Csurka et al. BMVC08) Unary potential (Torralba et al. ICCV03) Pairwise potential (Rabinovich et al. ICCV07, CVPR08) Only solution : E C = C(L(x))

31. Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

32. Multi-feature Boosted Unary Pixel Potential Ladický, Russell, Kohli, Torr ICCV09 Sturgess, Alahari, Ladický, Torr BMVC09 Unary likelihoods based on spatial configuration (Shotton et al. ECCV06) Classifier trained using boosting

33. Multi-feature Boosted Unary Segment Potential Ladický, Russell, Kohli, Torr ICCV09 Classifier trained using boosting

34. Other Potentials Ladický, Russell, Kohli, Torr ICCV09 Ladický, Russell, Kohli, Torr CVPR10 (submitted) Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted) Intensity dependent pairwise pixel potential Colour EMD-distance dependent pairwise segment potential Detector potentials based on off-the-shelf detectors (Felzenszwalb et al. CVPR08, Vedaldi et al. ICCV09) Generatively trained Co-occurence potential

35. Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

36. Inference over Hierarchical CRF Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

37. Inference over Hierarchical CRF Problem is NP-hard Any message passing algorithm (TRW-S, BP,..) or ICM can be applied to pairwise model αβ-swap (potentials must be semi-metric) Ishikawa construction over (α-F-β transition) αexpansion (potentials must be metric) Reparametrization of interlayer connection to metric potential Ishikawa construction over (α-F-old transition) For more details read our technical report Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

38. Inference over Hierarchical CRF Problem is NP-hard Any message passing algorithm (TRW-S, BP,..), ICM or LP relaxation can be applied to pairwise model αβ-swap (potentials must be semi-metric) Ishikawa construction over (α-F-β transition) αexpansion (potentials must be metric) Reparametrization of interlayer connection to metric potential Ishikawa construction over (α-F-old transition) For more details read our technical report Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

39. Inference over Hierarchical CRF Problem is NP-hard Any message passing algorithm (TRW-S, BP,..), ICM or LP relaxation can be applied to pairwise model αβ-swap (potentials must be semi-metric) Ishikawa construction over (α-F-β transition) αexpansion (potentials must be metric) Reparametrization of interlayer connection to metric potential Ishikawa construction over (α-F-old transition) For more details read our technical report Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

40. Inference over Hierarchical CRF Problem is NP-hard Any message passing algorithm (TRW-S, BP,..), ICM or LP relaxation can be applied to pairwise model αβ-swap (potentials must be semi-metric) Ishikawa construction over (α-F-β transition) α-expansion (potentials must be metric) Reparametrization of interlayer connection to metric potential Ishikawa construction over (α-F-old transition) Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

41. Inference over Hierarchical CRF Russell, Ladický, Kohli, Torr AISTATS10 (submitted)

42. Inference for Co-occurence Integer programming formulation possible using one variable for each subset of the label set LP relaxation applicable by relaxing IP program αβ-swap possible if the cooccurence cost C(L(x)) is monotonically increasing with respect to L(x) α-expansion approximate move possible if the cooccurence cost C(L(x)) is monotonically increasing with respect to L(x) Ladický, Russell, Kohli, Torr CVPR10 (submitted)

43. Overview Object-class Segmentation problem Datasets MSRC dataset VOC2009 dataset CamVid dataset Standard approaches Pairwise CRF over Pixels Pairwise CRF over Segments Detection-driven Segmentation Model Robust-P N model Associative Hierarchical CRF Detectors in AHCRF Label co-occurrence in CRFs Potential training Inference Results Discussion General overviewOur approach

44. VOC2009 Results

47. MSRC Results

48. Qualitative comparison of results without and with co-occurence Ladický, Russell, Kohli, Torr CVPR10 (submitted) MSRC Image Without COWith CO Without CO

49. CamVid Results Sturgess, Alahari Ladický, Torr BMVC09 Ladický, Alahari, Sturgess, Russell, Torr CVPR10 (submitted) Our Result CamVid Image Ground Truth Brostow et al.

50. Qualitative comparisons Quantitative comparison of methods on VOC2009 dataset Quantitative comparison of methods on MSRC dataset Quantitative comparison of methods on CamVid dataset

51. Take home message

52. Use our model

53. Take home message Use our model Use your favourite potentials

54. Take home message Use our model Use your favourite potentials Use your friend's favourite potentials

55. Take home message Use our model Use your favourite potentials Use your friend's favourite potentials Use your friend's friend's favourite potentials

56. Take home message Use our model Use your favourite potentials Use your friend's favourite potentials Use your friend's friend's favourite potentials Vision solved

57. Take home message Use our model Use your favourite potentials Use your friend's favourite potentials Use your friend's friend's favourite potentials Vision solved (..almost)

58. Thank you Questions?