1 Building a Dictionary of Image Fragments Zicheng Liao Ali Farhadi Yang Wang Ian Endres David Forsyth Department of Computer Science, University of Illinois at Urbana-Champaign

2 Outline ► Introduction ► Related Work ► ► Building a Dictionary of Fragments ► ► Applications ► ► Conclusion

3 Introduction

4 Introduction ► Image fragment – regions that could represent a single object, an object in context or a piece of a scene – form a natural representation of objects. ► ► Key step - to build a large, rich dictionary of image fragments automatically.

5 Introduction ► Four steps:  1.G  1.Generate a set of fragment proposals from image sets.   2.Verifythe qualities of the generated fragment proposals with a discriminative method. The selected fragments are grouped and indexed by the labels of their source images.   3.Use a clean-up procedure to remove anomalies from the dictionary within each category.   4.Matte the resulting fragments out of the training images to get the best possible boundaries.

6 Related Work ► ► Automated object segmentation   (a) that it is useful to work with more than one segmentation of a particular image, then choose good fragments   (b) that these multiple segmentations can yield estimates of support   (c) that it is possible to identify segments that seem to form a single object, without knowing what the object is

7 Related Work ► ► Exemplar-based image classification   There exist methods for image classification using region-based exemplar matching and image-based exemplar matching

8 Building a Dictionary of Fragments ► 1. ► 1. Generating fragment proposals ► ► 2. Fragment verification ► ► 3. Dictionary clean-up ► ► 4. Matting dictionary fragments

9 Building a Dictionary of Fragments

10 Building a Dictionary of Fragments- Generating fragment proposals ► Proposing regions: ► Proposing regions: This step aims to generate a large and diverse set of proposals that are likely to be object regions.

11 Building a Dictionary of Fragments- Generating fragment proposals ► ► Ranking proposals:   The next step of is to rank all the proposals in an image so that object regions are ranked higher than non-object regions.   Use a rank-SVM formulation based on various features computed from proposal regions, such as color, texture, geometric surfaces, boundaries, etc.

12 Building a Dictionary of Fragments- Fragment Verification

13 Building a Dictionary of Fragments- Fragment Verification ► ► Flickr: These images are downloaded from Flickr. They are mainly about humans, objects, activities, pets, and familiar scenes (indoor and outdoor). ► ► Caltech256: This dataset is widely used for object recognition in the computer vision literature. ► ► PASCAL VOC2010: This is another widely used object recognition dataset. The images are more complex than those in Caltech256.

14 Building a Dictionary of Fragments- Fragment Verification

15 Building a Dictionary of Fragments- Fragment Verification ► ► A good fragment can be a whole object, a meaningful component of a larger object, or a scene that consists of part of an scene. ► ► A good fragment ’ s effective size should cover neither too little nor too much of the entire image domain. If a fragment covers too little of the image, it may contain little discriminative information and is unlikely to be useful for image compositing. On the other hand, if a fragment covers too much of the image, its distinction from the whole image is small.

16 Building a Dictionary of Fragments- Dictionary Cleanup ► ► Fragments associated with the same tag (e.g. “ cat ” ) in the source images are grouped as “ cat ” fragments. ► ► In this step we want to remove such within- class fragment anomalies from the dictionary.

17 Building a Dictionary of Fragments- Dictionary Cleanup ► ► How to cleanup   For each new incoming fragment, we use an adapted asymmetric region-to-image matching algorithm to measure its distance to the fragment set and count the top k (5 in experiment) best matches.

18 Building a Dictionary of Fragments- Dictionary Cleanup

19 Building a Dictionary of Fragments- Matting Dictionary Fragments ► ► The closed-form matting algorithm simplifies the matting equation with a local window color smoothness assumption, and transforms the problem into a quadratic optimization problem, which can be solved efficiently via a sparse linear solver.

20 Applications ► 1. ► 1. Image Classification ► ► 2. Object Localization ► ► 3. Image Composition

21 Applications- Applications- Image Classification ► ► Fragments may have a small advantage because contextual information creates noise, or because our fragment selection procedure will prefer images with high contrast between fragment and background.

22 Applications- Applications- Image Classification

23 Applications- Applications- Object Localization ► ► The spatial support of fragments can be used to accurately localize objects in a query image.

24 Applications- Applications- Object Localization

25 Applications-Image Composition

26 Applications-Image Composition

27 Conclusions ► ► Use the highly localized information of fragment-based matching to do object detection.

28 ► Thanks for your listening.