Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based image retrieval Datasets & Conclusions

Aim Given an image and object category, to segment the object Segmentation should (ideally) be shaped like the object e.g. cow-like obtained efficiently in an unsupervised manner able to handle self-occlusion Segmentation Object Category Model Cow Image Segmented Cow Slide from Kumar ‘05

Examples of bottom-up segmentation Borenstein and Ullman, ECCV 2002 Example: Normalized Cuts, Shi & Malik, 1997 Difficult without top-down cues

Random Fields for segmentation I = Image pixels (observed) h = foreground/background labels (hidden) – one label per pixel  = Parameters Prior LikelihoodPosteriorJoint 1.Generative approach models joint  Markov random field (MRF) 2. Discriminative approach models posterior directly  Conditional random field (CRF)

I (pixels) Image Plane i j h (labels)  {foreground, background} hihi hjhj Unary Potential  i ( I |h i,  i ) Pairwise Potential (MRF)  ij (h i, h j |  ij ) MRF PriorLikelihood Generative Markov Random Field Prior has no dependency on I

Conditional Random Field Lafferty, McCallum and Pereira 2001 PairwiseUnary Dependency on I allows introduction of pairwise terms that make use of image. For example, neighboring labels should be similar only if pixel colors are similar  Contrast term Discriminative approach I (pixels) Image Plane i j hihi hjhj e.g Kumar and Hebert 2003

I (pixels) Image Plane i j hihi hjhj Figure from Kumar et al., CVPR 2005 OBJCUT Ω (shape parameter) Kumar, Torr & Zisserman 2005 PairwiseUnary Ω is a shape prior on the labels from a Layered Pictorial Structure (LPS) model Segmentation by: - Match LPS model to image (get number of samples, each with a different pose -Marginalize over the samples using a single graph cut [Boykov & Jolly, 2001] Label smoothness Contrast Distance from Ω Color Likelihood

OBJCUT: Shape prior - Ω - Layered Pictorial Structures (LPS) Generative model Composition of parts + spatial layout Layer 2 Layer 1 Parts in Layer 2 can occlude parts in Layer 1 Spatial Layout (Pairwise Configuration) Kumar, et al. 2004, 2005

In the absence of a clear boundary between object and background SegmentationImage OBJCUT: Results Using LPS Model for Cow

Layout Consistent Random Field Layout consistency Part detector Winn and Shotton 2006 I = Image pixels (observed) h = foreground/background labels (hidden) – one label per pixel  = Parameters Variant of conditional random field

[Lepetit et al. CVPR 2005] Decision forest classifier Features are differences of pixel intensities Classifier Winn and Shotton 2006 Layout CRF: Part detector

Layout consistency (8,3)(9,3)(7,3) (8,2)(9,2)(7,2) (8,4)(9,4)(7,4) Neighboring pixels (p,q) ? (p,q+1) (p,q) (p+1,q+1) (p-1,q+1) Layout consistent Winn and Shotton 2006

Stability of part labelling Part color key

Other recognition & segmentation papers Object-Specific Figure-Ground Segregation Stella X. Yu and Jianbo Shi, 2002 Image parsing: Tu, Zhu and Yuille 2003 Todorovic and Ahuja, CVPR 2006 LOCUS model: See Jon’s talk tomorrow Kannan, Jojic and Frey 2004; Winn and Jojic, 2005 Implicit Shape Model - Liebe and Schiele, 2003 See CVPR 2007 course slides for more details 3D Layout CRF, Hoiem et al. CVPR 2007 Figure from Borenstein and Ullman, ECCV 2002

Summary Strength –Explains every pixel of the image –Useful for image editing, layering, etc. Issues –Invariance issues (especially) scale, view-point variations –Inference difficulties