1. 1 Outline Overview Integrating Vision Models CCM: Cascaded Classification Models Learning Spatial Context TAS: Things and Stuff Descriptive Querying of Images LOOPS: Localizing Object Outlines using Probabilistic Shape Future Directions [Heitz et al. NIPS 2008b]

2. 2 Image Queries on Objects Categorization Localization Descriptive What color is his tail? Where is his head? Is he… standing? sitting? bent over? What is he doing?

3. 3 Related Work Refined Localization Coarse Fine Boosted Detectors “Parts” Models OUR WORK Localization Models [ Torralba et al., PAMI 2007 ] [ Opelt, ECCV 2006 ] [ Fei-Fei and Perona, CVPR 2005 ] [ Fergus et al., CVPR 2003 ] [ Leibe et al, ECCV 2004 ] [ Bar-Hillel et al, CVPR 2005 ] [ Winn & Shotton, CVPR 2006 ] [ Kumar et al., CVPR 2005 ] [ Cootes et al., CVIU 1995 ] [ Borenstein et al., CVPR 2004 ]

4. 4 Shape Representation: Landmarks Set of “keypoint” landmarks Shape defined by connecting piecewise  linear contour Internal landmarks are allowed (but not shown here)

5. 5 Training Data Images + Outlines

6. 6 State-of-the-art Alternatives kAS Detector: Edge-based object detector Pro: No outline required Great at detection Con: No single outline OBJ CUT: Object-based segmentation Pro: Produces outlines Con: Appearance model based on internal texture [ Kumar et al., CVPR 2005 ] [ Prasad et al., CVPR 2006 ] [ Ferrari et al., CVPR 2007 ]

7. 7 LOOPS Pipeline Localized Test Outlines LOOPS Model -1 std +1 std produce corresponded training data register model to images shape based classification Consistent Outlines learn shape & landmark detectors Descriptive Classification DownUp UP DOWN Localizing Object Outlines using Probabilistic Shape Images + Outlines

8. 8 Corresponded Outlines produce corresponded training data Consistent Outlines Localizing Object Outlines using Probabilistic Shape Images + Outlines [ Hill & Taylor, BMVC 1996 ] Based on existing work in medical imaging Intuition: Arc length and curvature should remain consistent

9. 9 Learning Shape & Detectors LOOPS Model -1 std +1 std Consistent Outlines learn shape & landmark detectors Localizing Object Outlines using Probabilistic Shape

10. 10 Multivariate Gaussian over landmark locations Shape Model Neck Legs

11. 11 Build on state-of-the-art discriminative methods for detecting “parts” or “objects” Build a detector for each landmark Landmark Detectors

12. 12 Registration Localized Test Outlines LOOPS Model -1 std +1 std register model to images Localizing Object Outlines using Probabilistic Shape

13. 13 “Registering” the Model to an Image Task: Assign each landmark l  L to a pixel p l  P ? ? L – Assignment of Landmarks to Pixels L* = argmax Score(L | I) = argmax ShapeScore(L) + ImageScore(L | I)

14. 14 The LOOPS MRF shape score landmark detectors pairwise image score Registering = MAP Inference over L

15. 15 Outlining Image Detectors Only Full LOOPS

16. 16 Results GiraffeLlama Rhino

17. kAS Detector OBJ CUT LOOPS

18. kAS Detector OBJ CUT LOOPS

19. 19 LOOPS Pipeline Localized Test Outlines LOOPS Model -1 std +1 std produce corresponded training data register model to images shape based classification Consistent Outlines learn shape & landmark detectors Descriptive Classification DownUp UP DOWN Localizing Object Outlines using Probabilistic Shape Images + Outlines

20. 20 Descriptive Classification Localized Test Outlines shape based classification Descriptive Classification DownUp UP DOWN Localizing Object Outlines using Probabilistic Shape

21. 21 RANDOM Descriptive Queries Goal: Classify based on shape characteristics Is the giraffe Or 12345678910 0.4 0.6 0.8 1 # Training Instances (per class) “True” shape Boosting Close this gap Accuracy

22. 22 Mammals