1. Graph Cut based Inference with Co-occurrence Statistics Ľubor Ladický, Chris Russell, Pushmeet Kohli, Philip Torr

2. Image labelling Problems Image Denoising Geometry Estimation Object Segmentation Assign a label to each image pixel Building Sky Tree Grass

3. Standard CRF Energy Pairwise CRF models Data termSmoothness term

4. Standard CRF Energy Pairwise CRF models Restricted expressive power Data termSmoothness term

5. Structures in CRF Taskar et al. 02 – associative potentials Kohli et al. 08 – segment consistency Woodford et al. 08 – planarity constraint Vicente et al. 08 – connectivity constraint Nowozin & Lampert 09 – connectivity constraint Roth & Black 09 – field of experts Ladický et al. 09 – consistency over several scales Woodford et al. 09 – marginal probability Delong et al. 10 – label occurrence costs

6. Pairwise CRF models Standard CRF Energy for Object Segmentation Cannot encode global consistency of labels!! Local context

7. Image from Torralba et al. 10 Detection Suppression If we have 1000 categories (detectors), and each detector produces 1 fp every 10 images, we will have 100 false alarms per image… pretty much garbage… [Torralba et al. 10, Leibe & Schiele 09, Barinova et al. 10]

8. Thing – Thing Stuff - Stuff Stuff - Thing [ Images from Rabinovich et al. 07 ] Encoding Co-occurrence Co-occurrence is a powerful cue [Heitz et al. '08] [Rabinovich et al. ‘07]

9. Thing – Thing Stuff - Stuff Stuff - Thing [ Images from Rabinovich et al. 07 ] Encoding Co-occurrence Co-occurrence is a powerful cue [Heitz et al. '08] [Rabinovich et al. ‘07] Proposed solutions : 1. Csurka et al. 08 - Hard decision for label estimation 2. Torralba et al. 03 - GIST based unary potential 3. Rabinovich et al. 07 - Full-connected CRF

10. So... What properties should these global co-occurence potentials have ?

11. Desired properties 1. No hard decisions

12. Desired properties 1. No hard decisions Incorporation in probabilistic framework Unlikely possibilities are not completely ruled out

13. Desired properties 1. No hard decisions 2. Invariance to region size

14. Desired properties 1. No hard decisions 2. Invariance to region size Cost for occurrence of {people, house, road etc.. } invariant to image area

15. Desired properties 1. No hard decisions 2. Invariance to region size The only possible solution : Local context Global context Cost defined over the assigned labels L(x) L(x)={,, }

16. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred L(x)={ building, tree, grass, sky } L(x)={ aeroplane, tree, flower, building, boat, grass, sky } 

17. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred 4. Efficiency

18. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred 4. Efficiency a) Memory requirements as O(n) with the image size and number or labels b) Inference tractable

19. Torralba et al.(2003) – Gist-based unary potentials Rabinovich et al.(2007) - complete pairwise graphs Csurka et al.(2008) - hard estimation of labels present Previous work

20. Zhu & Yuille 1996 – MDL prior Bleyer et al. 2010 – Surface Stereo MDL prior Hoiem et al. 2007 – 3D Layout CRF MDL Prior Delong et al. 2010 – label occurence cost Related work C(x) = K |L(x)| C(x) = Σ L K L δ L (x)

21. Zhu & Yuille 1996 – MDL prior Bleyer et al. 2010 – Surface Stereo MDL prior Hoiem et al. 2007 – 3D Layout CRF MDL Prior Delong et al. 2010 – label occurence cost Related work C(x) = K |L(x)| C(x) = Σ L K L δ L (x) All special cases of our model

22. Inference Pairwise CRF Energy

23. Inference IP formulation (Schlesinger 73)

24. Inference Pairwise CRF Energy with co-occurence

25. Inference IP formulation with co-occurence

26. Inference IP formulation with co-occurence Pairwise CRF cost Pairwise CRF constaints

27. Inference IP formulation with co-occurence Co-occurence cost

28. Inference IP formulation with co-occurence Inclusion constraints

29. Inference IP formulation with co-occurence Exclusion constraints

30. Inference LP relaxation Relaxed constraints

31. Inference LP relaxation Very Slow! 80 x 50 subsampled image takes 20 minutes

32. Inference: Our Contribution Pairwise representation One auxiliary variable Z  2 L Infinite pairwise costs if x i  Z [see technical report] *Solvable using standard methods: BP, TRW etc.

33. Inference: Our Contribution Pairwise representation One auxiliary variable Z  2 L Infinite pairwise costs if x i  Z [see technical report] *Solvable using standard methods: BP, TRW etc. Relatively faster but still computationally expensive!

34. Inference using Moves Graph Cut based move making algorithms [Boykov et al. 01] α-expansion transformation function Series of locally optimal moves Each move reduces energy Optimal move by minimizing submodular function Space of Solutions (x) : L N Move Space (t) : 2 N Search Neighbourhood Current Solution N Number of Variables L Number of Labels

35. Inference using Moves Graph Cut based move making algorithms [Boykov, Veksler, Zabih. 01] α-expansion transformation function

36. Inference using Moves Label indicator functions Co-occurence representation

37. Inference using Moves Move Energy Cost of current label set

38. Inference using Moves Move Energy Decomposition to α-dependent and α-independent part α-independentα-dependent

39. Inference using Moves Move Energy Decomposition to α-dependent and α-independent part Either α or all labels in the image after the move

40. Inference using Moves Move Energy submodularnon-submodular

41. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) – E'(t) = E(t) for current solution – E'(t)  E(t) for any other labelling

42. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) – E'(t) = E(t) for current solution – E'(t)  E(t) for any other labelling Occurrence - tight

43. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) – E'(t) = E(t) for current solution – E'(t)  E(t) for any other labelling Co-occurrence overestimation

44. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) – E'(t) = E(t) for current solution – E'(t)  E(t) for any other labelling General case [See the paper]

45. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) – E'(t) = E(t) for current solution – E'(t)  E(t) for any other labelling Quadratic representation

46. Application: Object Segmentation Standard MRF model for Object Segmentation Label based Costs Cost defined over the assigned labels L(x)

47. Training of label based potentials Indicator variables for occurrence of each label Label set costs Approximated by 2 nd order representation

48. Experiments Methods – Segment CRF – Segment CRF + Co-occurrence Potential – Associative HCRF [Ladický et al. ‘09] – Associative HCRF + Co-occurrence Potential Datasets MSRC-21 Number of Images: 591 Number of Classes: 21 Training Set: 50% Test Set: 50% PASCAL VOC 2009 Number of Images: 1499 Number of Classes: 21 Training Set: 50% Test Set: 50%

49. MSRC - Qualitative

50. VOC 2010-Qualitative

51. Quantitative Results MSRC-21 PASCAL VOC 2009

52. Incorporated label based potentials in CRFs Proposed feasible inference Open questions – Optimal training method for co-occurence – Bounds of graph cut based inference Questions ? Summary and further work