1. Visual Object Recognition Rob Fergus Courant Institute, New York University http://cs.nyu.edu/~fergus/icml_tutorial/

2. Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based image retrieval Datasets & Conclusions

3. Li Fei-Fei, Princeton Rob Fergus, NYU Antonio Torralba, MIT Recognizing and Learning Object Categories: Year 2007 http://people.csail.mit.edu/torralba/shortCourseRLOC

4. Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based image retrieval Datasets & Conclusions

5. So what does object recognition involve?

6. Classification: are there street-lights in the image?

7. Detection: localize the street-lights in the image

8. Object categorization mountain building tree banner vendor people street lamp

9. Scene and context categorization outdoor city …

10. Application: Assisted driving meters Ped Car Lane detection Pedestrian and car detection Collision warning systems with adaptive cruise control, Lane departure warning systems, Rear object detection systems,

11. Application: Computational photography

12. Application: Improving online search Query: STREET Organizing photo collections

13. Challenges 1: view point variation Michelangelo 1475-1564

14. Challenges 2: scale

15. Challenges 3: illumination slide credit: S. Ullman

16. Challenges 4: background clutter Bruegel, 1564

17. Challenges 5: occlusion http://lh5.ggpht.com/_wJc6t2hDl2M/RrL7Gh6sS7I/AAAAAAAAAYY/n3xaHc2opls/DSC00633.JPG

18. Challenges 6: deformation Xu, Beihong 1943 http://img.timeinc.net/time/asia/magazine/2007/1112/racehorse_1112.jpg

19. History: single object recognition Object 1 Object 2 Object 3

20. David Lowe [1985] Single object recognition history: Geometric methods Rothwell et al. [1992]

21. Single object recognition history: Appearance-based methods Murase & Nayer 1995 Schmid & Mohr 1997 Lowe, et al. 1999, 2003 Mahamud and Herbert, 2000 Ferrari et al. 2004 Rothganger et al. 2004 Moreels and Perona, 2005 …

22. Instance 1 Instance 2 Instance 3 Challenges 7: intra-class variation Shoe class

23. History: early object categorization

24. Fischler, Elschlager, 1973 Turk and Pentland, 1991 Belhumeur, Hespanha, & Kriegman, 1997 Rowley & Kanade, 1998 Schneiderman & Kanade 2004 Viola and Jones, 2000 Heisele et al., 2001 Amit and Geman, 1999 LeCun et al. 1998 Belongie and Malik, 2002 DeCoste and Scholkopf, 2002 Simard et al. 2003 Poggio et al. 1993 Argawal and Roth, 2002 Schneiderman & Kanade, 2004 …..

26. Three main issues Representation –How to represent an object category Learning –How to form the classifier, given training data Recognition –How the classifier is to be used on novel data

27. Representation –Generative / discriminative / hybrid

28. Representation –Generative / discriminative / hybrid –Appearance only or location and appearance

29. Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –Invariances View point Illumination Occlusion Scale Deformation Clutter etc.

30. Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –Invariances –Part-based or global with sub-window

31. Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –Invariances –Parts or global w/sub- window –Use set of features or each pixel in image

32. –Unclear how to model categories, so learn rather than manually specify Learning

33. –Unclear how to model categories, so learn rather than manually specify –Methods of training: generative vs. discriminative Learning

34. –Unclear how to model categories, so learn rather than manually specify –Methods of training: generative vs. discriminative –Level of supervision Manual segmentation; bounding box; image labels; noisy labels Learning Contains a motorbike

35. Learning –Unclear how to model categories, so learn rather than manually specify –Methods of training: generative vs. discriminative –Level of supervision Manual segmentation; bounding box; image labels; noisy labels -- Training images: Issue of over-fitting (typically limited training data) Negative images for discriminative methods

36. Learning –Unclear how to model categories, so learn rather than manually specify –Methods of training: generative vs. discriminative –Level of supervision Manual segmentation; bounding box; image labels; noisy labels -- Training images: Issue of over-fitting (typically limited training data) Negative images for discriminative methods --Priors

37. –Scale / orientation range to search over –Speed –Context Recognition

38. Hoiem, Efros, Herbert, 2006 –Context enables pruning of detector output Recognition