1. Part 4: combined segmentation and recognition Li Fei-Fei

2. Aim Given an image and object category, to segment the object Segmentation should (ideally) be shaped like the object e.g. cow-like obtained efficiently in an unsupervised manner able to handle self-occlusion Segmentation Object Category Model Cow Image Segmented Cow

3. In this section: brief paper reviews Jigsaw approach: Borenstein & Ullman, 2001, 2002 Concurrent recognition and segmentation: Yu and Shi, 2002 Image parsing: Tu et al. 2003 Interleaved segmentation: Liebe & Schiele, 2004, 2005 OBJCUT: Kumar et al. 2005 LOCUS: Winn and Jojic, 2005

4. Jigsaw approach: Borenstein and Ullman, 2001, 2002

5. Jigsaw approach Each patch has foreground/background mask

6. Object-Specific Figure-Ground Segregation Stella X. Yu and Jianbo Shi, 2002

7. Object-Specific Figure-Ground Segregation Some segmentation/detection results Yu and Shi, 2002

8. Image parsing: Tu, Zhu and Yuille 2003

10. Perceptual and Sensory Augmented Computing Interleaved Object Categorization and Segmentation Implicit Shape Model - Recognition Backprojected Hypotheses Interest Points Matched Codebook Entries Probabilistic Voting Voting Space (continuous) Backprojection of Maxima Segmentation Refined Hypotheses (uniform sampling) Liebe and Schiele, 2003, 2005

11. Perceptual and Sensory Augmented Computing Interleaved Object Categorization and Segmentation Interpretation of p(figure) map  per-pixel confidence in object hypothesis  Use for hypothesis verification p(figure) p(ground) Segmentation p(figure) p(ground) Original image Liebe and Schiele, 2003, 2005

12. Perceptual and Sensory Augmented Computing Interleaved Object Categorization and Segmentation Cows: Results Segmentations from interest points Single-frame recognition - No temporal continuity used! Liebe and Schiele, 2003, 2005

13. OBJCUT: shape prior -- Layered Pictorial Structures (LPS) Generative model Composition of parts + spatial layout Layer 2 Layer 1 Parts in Layer 2 can occlude parts in Layer 1 Spatial Layout (Pairwise Configuration) Kumar, et al. 2004, 2005

14. OBJCUT Probability of labelling in addition has Unary potential which depend on distance from Θ (shape parameter) D (pixels) m (labels) Θ (shape parameter) Image Plane Object Category Specific MRF x y mxmx mymy Unary Potential Φ x (m x |Θ) Kumar, et al. 2004, 2005

15. In the absence of a clear boundary between object and background SegmentationImage OBJCUT: Results Using LPS Model for Cow

16. LOCUS model Deformation field D Position & size T Class shape π Class edge sprite μ o,σ o Edge image e Image Object appearance λ 1 Background appearance λ 0 Mask m Shared between images Different for each image Winn and Jojic, 2005

17. Summary Strength –Explains every pixel of the image –Useful for image editing, layering, etc. Issues –Invariance issues (especially) scale, view-point variations