1 Outline Overview Integrating Vision Models CCM: Cascaded Classification Models Learning Spatial Context TAS: Things and Stuff Descriptive Querying of Images LOOPS: Localizing Object Outlines using Probabilistic Shape Future Directions [Heitz et al. NIPS 2008b]

2 Image Queries on Objects Categorization Localization Descriptive What color is his tail? Where is his head? Is he… standing? sitting? bent over? What is he doing?

3 Related Work Refined Localization Coarse Fine Boosted Detectors “Parts” Models OUR WORK Localization Models [ Torralba et al., PAMI 2007 ] [ Opelt, ECCV 2006 ] [ Fei-Fei and Perona, CVPR 2005 ] [ Fergus et al., CVPR 2003 ] [ Leibe et al, ECCV 2004 ] [ Bar-Hillel et al, CVPR 2005 ] [ Winn & Shotton, CVPR 2006 ] [ Kumar et al., CVPR 2005 ] [ Cootes et al., CVIU 1995 ] [ Borenstein et al., CVPR 2004 ]

4 Shape Representation: Landmarks Set of “keypoint” landmarks Shape defined by connecting piecewise  linear contour Internal landmarks are allowed (but not shown here)

5 Training Data Images + Outlines

6 State-of-the-art Alternatives kAS Detector: Edge-based object detector Pro: No outline required Great at detection Con: No single outline OBJ CUT: Object-based segmentation Pro: Produces outlines Con: Appearance model based on internal texture [ Kumar et al., CVPR 2005 ] [ Prasad et al., CVPR 2006 ] [ Ferrari et al., CVPR 2007 ]

7 LOOPS Pipeline Localized Test Outlines LOOPS Model -1 std +1 std produce corresponded training data register model to images shape based classification Consistent Outlines learn shape & landmark detectors Descriptive Classification DownUp UP DOWN Localizing Object Outlines using Probabilistic Shape Images + Outlines

8 Corresponded Outlines produce corresponded training data Consistent Outlines Localizing Object Outlines using Probabilistic Shape Images + Outlines [ Hill & Taylor, BMVC 1996 ] Based on existing work in medical imaging Intuition: Arc length and curvature should remain consistent

9 Learning Shape & Detectors LOOPS Model -1 std +1 std Consistent Outlines learn shape & landmark detectors Localizing Object Outlines using Probabilistic Shape

10 Multivariate Gaussian over landmark locations Shape Model Neck Legs

11 Build on state-of-the-art discriminative methods for detecting “parts” or “objects” Build a detector for each landmark Landmark Detectors

12 Registration Localized Test Outlines LOOPS Model -1 std +1 std register model to images Localizing Object Outlines using Probabilistic Shape

13 “Registering” the Model to an Image Task: Assign each landmark l  L to a pixel p l  P ? ? L – Assignment of Landmarks to Pixels L* = argmax Score(L | I) = argmax ShapeScore(L) + ImageScore(L | I)

14 The LOOPS MRF shape score landmark detectors pairwise image score Registering = MAP Inference over L

15 Outlining Image Detectors Only Full LOOPS

16 Results GiraffeLlama Rhino

kAS Detector OBJ CUT LOOPS

kAS Detector OBJ CUT LOOPS

19 LOOPS Pipeline Localized Test Outlines LOOPS Model -1 std +1 std produce corresponded training data register model to images shape based classification Consistent Outlines learn shape & landmark detectors Descriptive Classification DownUp UP DOWN Localizing Object Outlines using Probabilistic Shape Images + Outlines

20 Descriptive Classification Localized Test Outlines shape based classification Descriptive Classification DownUp UP DOWN Localizing Object Outlines using Probabilistic Shape

21 RANDOM Descriptive Queries Goal: Classify based on shape characteristics Is the giraffe Or # Training Instances (per class) “True” shape Boosting Close this gap Accuracy

22 Mammals