1. 11/26/2015 Copyright G.D. Hager Class 2 - Schedule 1.Optical Illusions 2.Lecture on Object Recognition 3.Group Work 4.Sports Videos 5.Short Lecture on Rigid Transformations 6.Lab time

2. 11/26/2015 Copyright G.D. Hager Object Recognition Techniques

3. 11/26/2015 Copyright G.D. Hager Li Fei-Fei, UIUC Rob Fergus, MIT Antonio Torralba, MIT Recognizing and Learning Object Categories ICCV 2005 Beijing, Short Course, Oct 15

4. 11/26/2015 Copyright G.D. Hager

5. 11/26/2015 Copyright G.D. Hager perceptible vision materialthing

6. 11/26/2015 Copyright G.D. Hager

7. 11/26/2015 Copyright G.D. Hager

8. 11/26/2015 Copyright G.D. Hager Plato said… Ordinary objects are classified together if they `participate' in the same abstract Form, such as the Form of a Human or the Form of Quartz. Forms are proper subjects of philosophical investigation, for they have the highest degree of reality. Ordinary objects, such as humans, trees, and stones, have a lower degree of reality than the Forms. Fictions, shadows, and the like have a still lower degree of reality than ordinary objects and so are not proper subjects of philosophical enquiry.

9. 11/26/2015 Copyright G.D. Hager Bruegel, 1564

10. 11/26/2015 Copyright G.D. Hager How many object categories are there? Biederman 1987

11. 11/26/2015 Copyright G.D. Hager Problems of Computer Vision: Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

12. 11/26/2015 Copyright G.D. Hager Problems of Computer Vision: Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

13. 11/26/2015 Copyright G.D. Hager Problems of Computer Vision: Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

14. 11/26/2015 Copyright G.D. Hager Object Recognition: The Problem Given: A database D of “known” objects and an image I: 1. Determine which (if any) objects in D appear in I 2. Determine the pose (rotation and translation) of the object Segmentation (where is it 2D) Recognition (what is it) The object recognition conundrum Pose Est. (where is it 3D)

15. 11/26/2015 Copyright G.D. Hager Object Recognition Issues How general is the problem? –2D vs. 3D –range of viewing conditions –available context –segmentation cues What sort of data is best suited to the problem? –local 2D features –3D surfaces –images How many objects are involved? –small: brute force search –large: ??

16. 11/26/2015 Copyright G.D. Hager Object Recognition Approaches Geometry-based –Interpretation trees: use features compute “local constraints” valid under Euclidean or similarity group –Invariants: use features compute “global indices” that do not change over viewing conditions (i.e. invariant in the projective group) Image-based: –store information about views and match to views intensities histograms Semi-local: –use features detected using a stable (but not invariant) interest operator –use stable (but not invariant) measures on groups of features to index views

17. 11/26/2015 Copyright G.D. Hager So what does object recognition involve?

18. 11/26/2015 Copyright G.D. Hager Verification: is that a bus?

19. 11/26/2015 Copyright G.D. Hager Detection: are there cars?

20. 11/26/2015 Copyright G.D. Hager Identification: is that a picture of Mao?

21. 11/26/2015 Copyright G.D. Hager Object categorization sky building flag wall banner bus cars bus face street lamp

22. 11/26/2015 Copyright G.D. Hager Scene and context categorization outdoor city traffic …

23. 11/26/2015 Copyright G.D. Hager Challenges 1: view point variation Michelangelo 1475-1564

24. 11/26/2015 Copyright G.D. Hager Challenges 2: illumination slide credit: S. Ullman

25. 11/26/2015 Copyright G.D. Hager Challenges 3: occlusion Magritte, 1957

26. 11/26/2015 Copyright G.D. Hager Challenges 4: scale

27. 11/26/2015 Copyright G.D. Hager Challenges 5: deformation Xu, Beihong 1943

28. 11/26/2015 Copyright G.D. Hager Challenges 6: background clutter Klimt, 1913

29. 11/26/2015 Copyright G.D. Hager History: single object recognition

30. 11/26/2015 Copyright G.D. Hager History: single object recognition Lowe, et al. 1999, 2003 Mahamud and Herbert, 2000 Ferrari, Tuytelaars, and Van Gool, 2004 Rothganger, Lazebnik, and Ponce, 2004 Moreels and Perona, 2005 …

31. 11/26/2015 Copyright G.D. Hager Challenges 7: intra-class variation

32. 11/26/2015 Copyright G.D. Hager History: early object categorization

33. 11/26/2015 Copyright G.D. Hager Turk and Pentland, 1991 Belhumeur et al. 1997 Schneiderman et al. 2004 Viola and Jones, 2000 Amit and Geman, 1999 LeCun et al. 1998 Belongie and Malik, 2002 Schneiderman et al. 2004 Argawal and Roth, 2002 Poggio et al. 1993

34. 11/26/2015 Copyright G.D. Hager

35. 11/26/2015 Copyright G.D. Hager OBJECTS ANIMALS INANIMATE PLANTS MAN-MADENATURAL VERTEBRATE ….. MAMMALS BIRDS GROUSEBOARTAPIR CAMERA

36. 11/26/2015 Copyright G.D. Hager

37. 11/26/2015 Copyright G.D. Hager Scenes, Objects, and Parts Features Parts Objects Scene E. Sudderth, A. Torralba, W. Freeman, A. Willsky. ICCV 2005.

38. 11/26/2015 Copyright G.D. Hager Object categorization: the statistical viewpoint vs. Bayes rule: posterior ratio likelihood ratioprior ratio

39. 11/26/2015 Copyright G.D. Hager Object categorization: the statistical viewpoint posterior ratio likelihood ratioprior ratio Discriminative methods model posterior Generative methods model likelihood and prior

40. 11/26/2015 Copyright G.D. Hager Discriminative Direct modeling of Zebra Non-zebra Decision boundary

41. 11/26/2015 Copyright G.D. Hager Model and Generative LowMiddle HighMiddle  Low

42. 11/26/2015 Copyright G.D. Hager Three main issues Representation –How to represent an object category Learning –How to form the classifier, given training data Recognition –How the classifier is to be used on novel data

43. 11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid

44. 11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid –Appearance only or location and appearance

45. 11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –Invariances View point Illumination Occlusion Scale Deformation Clutter etc.

46. 11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –invariances –Part-based or global w/sub-window

47. 11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –invariances –Parts or global w/sub-window –Use set of features or each pixel in image

48. 11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning Learning

49. 11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –Methods of training: generative vs. discriminative Learning

50. 11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –What are you maximizing? Likelihood (Gen.) or performances on train/validation set (Disc.) –Level of supervision Manual segmentation; bounding box; image labels; noisy labels Learning Contains a motorbike

51. 11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –What are you maximizing? Likelihood (Gen.) or performances on train/validation set (Disc.) –Level of supervision Manual segmentation; bounding box; image labels; noisy labels –Batch/incremental (on category and image level; user- feedback ) Learning

52. 11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –What are you maximizing? Likelihood (Gen.) or performances on train/validation set (Disc.) –Level of supervision Manual segmentation; bounding box; image labels; noisy labels –Batch/incremental (on category and image level; user- feedback ) –Training images: Issue of overfitting Negative images for discriminative methods Priors Learning

53. 11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –What are you maximizing? Likelihood (Gen.) or performances on train/validation set (Disc.) –Level of supervision Manual segmentation; bounding box; image labels; noisy labels –Batch/incremental (on category and image level; user- feedback ) –Training images: Issue of overfitting Negative images for discriminative methods –Priors Learning

54. 11/26/2015 Copyright G.D. Hager –Scale / orientation range to search over –Speed Recognition

55. 11/26/2015 Copyright G.D. Hager State of The Art Current systems deal with simple nearly 2D situations or very restricted input On ~100 categories, reported accuracies in the 70-90% range, but with huge computational loads Not clear we have the right approach (yet)!

56. 11/26/2015 Copyright G.D. Hager Example: Fergus 2003

57. 11/26/2015 Copyright G.D. Hager Results, Fergus 2003

58. 11/26/2015 Copyright G.D. Hager Summary Object recognition/categorization is a rapidly evolving area Current systems are getting to the point they may be useful in real applications. Much more remains to be done in understanding how to move to the next level of performance.