1. Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based image retrieval Datasets & Conclusions

2. Databases Caltech 101 Caltech 256 Pascal Visual Object Classes (VOC) LabelMe Slides from Andrew Zisserman

3. Caltech 101 Pictures of objects belonging to 101 categories. About 40 to 800 images per category. Most categories have about 50 images. The size of each image is roughly 300 x 200 pixels. Collected in September 2003 by Fei-Fei Li, Marco Andreetto, and Marc 'Aurelio Ranzato. Train on 5, 10, 15, 20 or 30 images Test on rest – report results per class

4. Caltech 101 images

5. Smallest category size is 31 images: Too easy? –left-right aligned –Rotation artifacts –Soon will saturate performance Caltech-101: Drawbacks

6. Caltech-256 Smallest category size now 80 images About 30K images Harder –Not left-right aligned –No artifacts –Performance is halved –More categories New and larger clutter category

7. Caltech 256 images baseball-bat basketball-hoop dog kayac traffic light

8. The PASCAL Visual Object Classes (VOC) Dataset and Challenge Mark Everingham Luc Van Gool Chris Williams John Winn Andrew Zisserman

9. The PASCAL VOC Challenge Challenge in visual object recognition funded by PASCAL network of excellence Publicly available dataset of annotated images. Development kit available. Main competitions in classification (is there an X in this image) and detection (where are the X’s) “Taster competitions” in segmentation and 2-D human “pose estimation” (2007-present)

10. Dataset Content 20 classes: aeroplane, bicycle, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, train, TV Real images downloaded from flickr, not filtered for “quality” Complex scenes, scale, pose, lighting, occlusion,...

11. Annotation Complete annotation of all objects Annotated in one session with written guidelines Truncated Object extends beyond BB Occluded Object is significantly occluded within BB Pose Facing left Difficult Not scored in evaluation

12. Examples Aeroplane Bus BicycleBirdBoatBottle CarCatChairCow

13. History New dataset annotated annually –Annotation of test set is withheld until after challenge ImagesObjectsClassesEntries 20052,2322,871412 Collection of existing and some new data. 20065,3049,5071025 Completely new dataset from flickr (+MSRC) 20079,96324,6402028 Increased classes to 20. Introduced tasters. 20088,77620,73920 Added “occlusion” flag. Reuse of taster data. Release detailed results to support “meta-analysis”

14. Main Challenge Tasks Classification –Is there a dog in this image? –Evaluation by precision/recall Detection –Localize all the people (if any) in this image –Evaluation by precision/recall based on bounding box overlap

15. Person detection Example Precision/Recall: 2007

16. Russell, Torralba, Freman, 2005 LabelMe

17. CMU/MIT frontal facesvasc.ri.cmu.edu/idb/html/face/frontal_images cbcl.mit.edu/software-datasets/FaceData2.html PatchesFrontal faces Graz-02 Databasewww.emt.tugraz.at/~pinz/data/GRAZ_02/Segmentation masksBikes, cars, people UIUC Image Databasel2r.cs.uiuc.edu/~cogcomp/Data/Car/Bounding boxesCars TU Darmstadt Databasewww.vision.ethz.ch/leibe/data/Segmentation masksMotorbikes, cars, cows LabelMe datasetpeople.csail.mit.edu/brussell/research/LabelMe/intro.htmlPolygonal boundary>500 Categories Caltech 101www.vision.caltech.edu/Image_Datasets/Caltech101/Caltech101.htmlSegmentation masks101 categories Caltech 256 COIL-100 http://www.vision.caltech.edu/Image_Datasets/Caltech256/ www1.cs.columbia.edu/CAVE/research/softlib/coil-100.html Bounding Box Patches 256 Categories 100 instances NORBwww.cs.nyu.edu/~ylclab/data/norb-v1.0/Bounding box50 toys Databases for object localization Databases for object recognition On-line annotation tools ESP gamewww.espgame.orgGlobal image descriptionsWeb images LabelMepeople.csail.mit.edu/brussell/research/LabelMe/intro.htmlPolygonal boundaryHigh resolution images The next tables summarize some of the available datasets for training and testing object detection and recognition algorithms. These lists are far from exhaustive. Links to datasets Collections PASCALhttp://www.pascal-network.org/challenges/VOC/Segmentation, boxesvarious

18. Topics not covered Context –Scene –Inter-object relations Video –Tracking & detection Multiple viewpoints

19. Summary Methods reviewed here –Bag of words –Bag of words with location –Parts and structure –Discriminative methods –Combined Segmentation and recognition –Recognition for retrieval Resources online: http://cs.nyu.edu/~fergus/icml_tutorial –Slides –Code –Links to datasets