Presentation is loading. Please wait.

Presentation is loading. Please wait.

What, Where & How Many? Combining Object Detectors and CRFs

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "What, Where & How Many? Combining Object Detectors and CRFs"— Presentation transcript:

1 What, Where & How Many? Combining Object Detectors and CRFs
Ľubor Ladický, Paul Sturgess, Karteek Alahari, Christopher Russell, Philip H.S. Torr 1

2 Complete Scene Understanding
Involves Localization of all instances of foreground objects (“things”) Localization of all background classes (“stuff”) Pixel-wise segmentation 3D reconstruction Pose detection Action recognition Event recognition METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 2

3 Semantic Segmentation
Complete Scene Understanding Involves Localization of all instances of foreground objects (“things”) Localization of all background classes (“stuff”) Pixel-wise segmentation 3D reconstruction Pose detection Action recognition Event recognition Semantic Segmentation METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 3

4 Complete Scene Understanding
Involves Localization of all instances of foreground objects (“things”) Localization of all background classes (“stuff”) Pixel-wise segmentation METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 4

5 We're interested in whole scene understanding
Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Adelson, Forsyth et al. 96 5

6 We're interested in whole scene understanding
Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Adelson, Forsyth et al. 96 6

7 We're interested in whole scene understanding
Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Adelson, Forsyth et al. 96 7

8 We're interested in whole scene understanding
Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Adelson, Forsyth et al. 96 8

9 We're interested in whole scene understanding
Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Adelson, Forsyth et al. 96 9

10 We're interested in whole scene understanding
Semantic Scene Understanding We're interested in whole scene understanding Given an image, label all the stuff Stuff : Material defined by a homogeneous or repetitive pattern, with no specific spatial extent / shape. METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Adelson, Forsyth et al. 96 10

11 We're interested in whole scene understanding
Semantic Scene Understanding We're interested in whole scene understanding Given an image, label all the stuff Stuff : Material defined by a homogeneous or repetitive pattern, with no specific spatial extent / shape. METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Adelson, Forsyth et al. 96 11

12 Our plan is to combine Semantic Scene Understanding
State of the art sliding window object detection State of the art segmentation techniques car METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 12

13 Algorithms for Object Localization
Sliding window detectors HOG descriptor (Dalal & Triggs CVPR05) Based on histograms of features (Vedaldi et al. ICCV09) Part-based models (Felzenszwalb et al. CVPR09) METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 13

14 Algorithms for Object Localization
Sliding window detectors HOG descriptor (Dalal & Triggs CVPR05) Based on histograms of features (Vedaldi et al. ICCV09) Part-based models (Felzenszwalb et al. CVPR09) METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 14

15 Algorithms for Object Localization
Sliding window detectors HOG descriptor (Dalal & Triggs CVPR05) Based on histograms of features (Vedaldi et al. ICCV09) Part-based models (Felzenszwalb et al. CVPR09) METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 15

16 Algorithms for Object Localization
METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Non-maxima suppression Greedily Using field of indicator variables (Ramanan et al ICCV09) One binary variable yi { 0, 1 } per location/scale 16

17 Algorithms for Object Localization
car METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Non-maxima suppression Greedily Using field of indicator variables (Ramanan et al ICCV09) One binary variable yi { 0, 1 } per location/scale 17

18 Sliding window detectors
car Sliding window + Segmentation OBJCUT (Kumar et al. 05) Updating colour model (GrabCut - Rother et al. 04) METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 18

19 Sliding window detectors not good for “stuff”
METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Try to detect “sky” ! 19

20 Sliding window detectors not good for “stuff”
sky METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Sky is irregular shape not suited to the sliding window approach 20

21 State-of-the-art for object detection (“things”)
Sliding window detectors State-of-the-art for object detection (“things”) Do not work for background classes (“stuff”) No distinct shape Cannot be enclosed in a box Cannot recover from incorrect detections METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 21

22 Pairwise CRF over pixels
Algorithms for Object-class Segmentation Pairwise CRF over pixels CRF construction Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Final segmentation Shotton et al. ECCV06 22

23 Training of Potentials
Algorithms for Object-class Segmentation Pairwise CRF over pixels CRF construction Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Training of Potentials Shotton et al. ECCV06 23

24 Training of Potentials
Algorithms for Object-class Segmentation Pairwise CRF over pixels CRF construction Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Training of Potentials MAP Final segmentation Shotton et al. ECCV06 24

25 Pairwise CRF over Super-pixels / Segments
Algorithms for Object-class Segmentation Pairwise CRF over Super-pixels / Segments Unsupervised segmentation Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Felzenszwalb, Huttenlocher, IJCV2004 25

26 Pairwise CRF over Super-pixels / Segments
Algorithms for Object-class Segmentation Pairwise CRF over Super-pixels / Segments Unsupervised segmentation Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Felzenszwalb, Huttenlocher, IJCV2004 26

27 Training of potentials
Algorithms for Object-class Segmentation Pairwise CRF over Super-pixels / Segments Unsupervised segmentation Input image Training of potentials METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Batra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08, Rabinovich et al. ICCV07, Boix et al. CVPR10 27

28 Training of potentials
Algorithms for Object-class Segmentation Pairwise CRF over Super-pixels / Segments Unsupervised segmentation Input image Training of potentials METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features MAP Batra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08, Rabinovich et al. ICCV07, Boix et al. CVPR10 Final segmentation 28

29 Associative Hierarchical CRF
Algorithms for Object-class Segmentation Associative Hierarchical CRF Multiple segmentations or hierarchies Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Ladický et al. ICCV09 29

30 Associative Hierarchical CRF
Algorithms for Object-class Segmentation Associative Hierarchical CRF Multiple segmentations or hierarchies CRF construction Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Ladický et al. ICCV09 30

31 Associative Hierarchical CRF
Algorithms for Object-class Segmentation Associative Hierarchical CRF Multiple segmentations or hierarchies CRF construction Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features MAP Final segmentation Ladický et al. ICCV09, Russell et al. UAI10 31

32 State-of-the-art performance on MSRC & CamVid
Associative Hierarchical CRF State-of-the-art performance on MSRC & CamVid Merges information at multiple scales Can recover from incorrect segmentations METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Ladický et al. ICCV09, Sturgess et al., BMVC09 32

33 State-of-the-art performance on MSRC & CamVid
Associative Hierarchical CRF State-of-the-art performance on MSRC & CamVid Merges information at multiple scales Can recover from incorrect segmentations However, No concept of “things” Cannot distinguish between multiple instances METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features car Ladický et al. ICCV09, Sturgess et al., BMVC09 33

34 CRF Formulation with Detectors
CRF formulation altered with a potential for each detection METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 34

35 AH-CRF energy without detectors
CRF Formulation with Detectors CRF formulation altered with a potential for each detection AH-CRF energy without detectors METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features CRF graph over pixels 35

36 CRF Formulation with Detectors
CRF formulation altered with a potential for each detection AH-CRF energy without detectors Set of pixels of d-th detection Classifier response Detected label METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features CRF graph over pixels 36

37 CRF Formulation with Detectors
Joint CRF formulation should contain Possibility to reject detection hypothesis Recover the status of the detection (0 / 1) Thus, potential is a minimum over indicator variable yd { 0, 1 } Indicator variables METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features CRF graph over pixels 37

38 CRF Formulation with Detectors
Detection potential should decrease the energy of labelling agreeing with the detection hypothesis Partial disagreement penalized but not directly rejects the hypothesis Indicator variables METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features CRF graph over pixels 38

39 CRF Formulation with Detectors
Detection potential should decrease the energy of labelling agreeing with the detection hypothesis Partial disagreement penalized but not directly rejects the hypothesis Detection hypothesis status Partial inconsistency cost Detection strength METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features CRF graph over pixels 39

40 CRF Formulation with Detectors
Detection potential should decrease the energy of labelling agreeing with the detection hypothesis Partial disagreement penalized but not directly rejects the hypothesis Linear thresholded Linear METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features CRF graph over pixels 40

41 This higher order potential can be transformed to
Inference over CRF with Detectors This higher order potential can be transformed to Solvable with all graph cut-based methods which take the form of Robust PN (Kohli et al. CVPR08) METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 41

42 Result without detections
Results on CamVid dataset Result without detections Set of detections Final Result METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Brostow et al. Sturgess et al. Brostow et al. ECCV08, Sturgess et al. BMVC09 42

43 Result without detections
Results on CamVid dataset Result without detections Set of detections Final Result METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features Also provides number of object instances (using yd’s) 43

44 Results on VOC2009 dataset CRF without detectors CRF with detectors
Input image Input image METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 44

45 Novel formulation of CRF with detectors
Summary Novel formulation of CRF with detectors Can recover from incorrect detections Possible to obtain instances of objects Efficiently solvable METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 45

46 Future and ongoing work
Automatic non-maximum suppression Part-based models in CRF New things & stuff dataset available soon! Questions? METHOD time: complementary set of features Wide variety of object classes found in road scenes HO high quality object-class boundaries Multiclass classifier with feature selection High quality ground truth Discussion Next->features 46

Download ppt "What, Where & How Many? Combining Object Detectors and CRFs"

Similar presentations

Latent SVMs for Human Detection with a Locally Affine Deformation Field Ľubor Ladický 1 Phil Torr 2 Andrew Zisserman 1 1 University of Oxford 2 Oxford.

Latent SVMs for Human Detection with a Locally Affine Deformation Field Ľubor Ladický 1 Phil Torr 2 Andrew Zisserman 1 1 University of Oxford 2 Oxford.
O BJ C UT M. Pawan Kumar Philip Torr Andrew Zisserman UNIVERSITY OF OXFORD.

O BJ C UT M. Pawan Kumar Philip Torr Andrew Zisserman UNIVERSITY OF OXFORD.
The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton.

The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton.
Pose Estimation and Segmentation of People in 3D Movies Karteek Alahari, Guillaume Seguin, Josef Sivic, Ivan Laptev Inria, Ecole Normale Superieure ICCV.

Pose Estimation and Segmentation of People in 3D Movies Karteek Alahari, Guillaume Seguin, Josef Sivic, Ivan Laptev Inria, Ecole Normale Superieure ICCV.
Foreground Focus: Finding Meaningful Features in Unlabeled Images Yong Jae Lee and Kristen Grauman University of Texas at Austin.

Foreground Focus: Finding Meaningful Features in Unlabeled Images Yong Jae Lee and Kristen Grauman University of Texas at Austin.
Ľubor Ladický1 Phil Torr2 Andrew Zisserman1

Ľubor Ladický1 Phil Torr2 Andrew Zisserman1
Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction Ľubor Ladický, Paul Sturgess, Christopher Russell, Sunando Sengupta, Yalin.

Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction Ľubor Ladický, Paul Sturgess, Christopher Russell, Sunando Sengupta, Yalin.
Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based.

Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based.
Part 4: Combined segmentation and recognition by Rob Fergus (MIT)

Part 4: Combined segmentation and recognition by Rob Fergus (MIT)
Learning to Combine Bottom-Up and Top-Down Segmentation Anat Levin and Yair Weiss School of CS&Eng, The Hebrew University of Jerusalem, Israel.

Learning to Combine Bottom-Up and Top-Down Segmentation Anat Levin and Yair Weiss School of CS&Eng, The Hebrew University of Jerusalem, Israel.
Scene Labeling Using Beam Search Under Mutex Constraints ID: O-2B-6 Anirban Roy and Sinisa Todorovic Oregon State University 1.

Scene Labeling Using Beam Search Under Mutex Constraints ID: O-2B-6 Anirban Roy and Sinisa Todorovic Oregon State University 1.
Many slides based on P. FelzenszwalbP. Felzenszwalb General object detection with deformable part-based models.

Many slides based on P. FelzenszwalbP. Felzenszwalb General object detection with deformable part-based models.
Efficient Inference for Fully-Connected CRFs with Stationarity

Efficient Inference for Fully-Connected CRFs with Stationarity
Global spatial layout: spatial pyramid matching Spatial weighting the features Beyond bags of features: Adding spatial information.

Global spatial layout: spatial pyramid matching Spatial weighting the features Beyond bags of features: Adding spatial information.
Object-centric spatial pooling for image classification Olga Russakovsky, Yuanqing Lin, Kai Yu, Li Fei-Fei ECCV 2012.

Object-centric spatial pooling for image classification Olga Russakovsky, Yuanqing Lin, Kai Yu, Li Fei-Fei ECCV 2012.
Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs Roozbeh Mottaghi 1, Sanja Fidler 2, Jian Yao 2, Raquel Urtasun 2, Devi Parikh 3 1 UCLA.

Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs Roozbeh Mottaghi 1, Sanja Fidler 2, Jian Yao 2, Raquel Urtasun 2, Devi Parikh 3 1 UCLA.
GraphCut-based Optimisation for Computer Vision Ľubor Ladický.

GraphCut-based Optimisation for Computer Vision Ľubor Ladický.
Enhancing Exemplar SVMs using Part Level Transfer Regularization 1.

Enhancing Exemplar SVMs using Part Level Transfer Regularization 1.
Contour Based Approaches for Visual Object Recognition Jamie Shotton University of Cambridge Joint work with Roberto Cipolla, Andrew Blake.

Contour Based Approaches for Visual Object Recognition Jamie Shotton University of Cambridge Joint work with Roberto Cipolla, Andrew Blake.
More sliding window detection: Discriminative part-based models Many slides based on P. FelzenszwalbP. Felzenszwalb.

More sliding window detection: Discriminative part-based models Many slides based on P. FelzenszwalbP. Felzenszwalb.

Similar presentations

Ads by Google