1. Features-based Object Recognition P. Moreels, P. Perona California Institute of Technology

2. I : Individual object recognition Problem setup Coarse-to-fine hypothesis search. Probabilistic decomposition Experiments II : Searching for Grandma : Look at the eyes ! Problem setup Architecture of the system Demo Outline

3. The recognition continuum variability Individual objects Categories means of transportation BMW logo cars

4. Individual Object Recognition Autonomous navigation Security & Identification

5. … Problem setup New scene (test image) Models from database Find models and their pose (location, orientation…)

6. Models from database New scene (test image) Hypotheses – models & position 11 22 33 Θ = affine transformation variable H

7. Image characterization by features Features = privileged points of interest in an image Reduce the volume of information Interest points = high information content (e.g. high gradient) Hopefully provide invariance

8. Foreground vs. background features 1 2 3 4 5 6 7 8 Mutual euclidean distances in appearance space

9. Features assignments = models from database New scene (test image)... Interpretation... variable V

10. Scene interpretation scene model background

11. Coarse-to-fine example Fleuret & Geman (01) Face identification in complex scenes all scales, all tilts position in 8x8 cell all scales, right tilts position in 2x2 cell large scales, right tilts position in 2x2 cell

12. Coarse-to-fine strategy Search over possible locations is expensive Coarse-to-fine: progressive search refinement

13. Progressively narrow down focus on correct region of hypothesis space Use first information that is easy to obtain Simple building blocks organized in a cascade Probabilistic interpretation of each step Coarse-to-Fine detection

14. Prior knowledge Which objects are likely to be there, which pose are they likely to have ? unlikely situations

15. Coarse data: model voting Database Search tree  model #10  model #12  model #25  model #12 Query features Candidate matches obtained via indexing in a kd-tree. Counts number of votes obtained by each model Types of matches: –Matches with correct model –Stray matches with incorrect model –Features that did not match well with any model Candidate matches

16. (x 1,y 1,s 1,  1 ) (x 2,y 2,s 2,  2 ) Transform predicted by this match:  x = x 2 -x 1  y = y 2 -y 1  s = s 2 / s 1  =  2 -  1 Each match is represented by a dot in the space of 2D similarities (Hough space) xx yy ss  Use of rich descriptors (SIFT) Lowe’99,’04

17. Prediction of position of model center after transform The space of transform parameters is discretized into ‘bins’ Coarse bins to limit boundary issues and have a low false- alarm rate for this stage We count the number of votes collected by each bin. Model: statistical estimation => learns fraction of matches in correct bin and neighbors Coarse Hough transform Model Test scene correct transformation

18. Correspondence or clutter ? RANSAC ‘RANdom SAmple Consensus’ – robust statistic for parameter estimation 2D affine transform : 6 parameters  each sample contains 3 candidate correspondences. Model: –Probability p of detecting a feature –clutter: Poisson d d d  Output of RANSAC : pose transformation + set of features correspondences

19. Consistency Consistency between observations and hypothesis foreground features ‘null’ assignments geometry appearance Consistency - appearanceConsistency - geometry

20. Learning foreground & background densities Ground truth pairs of matches are collected Gaussian densities, centered on the nomimal value that appearance / pose should have according to H Learning background densities is easy: match to random images. Moreels&Perona, IJCV, 2007

21. Score of an extended hypothesis Hypothesis: model + position observed features geometry + appearance database of models constant Consistency Prior on model and poses Assignments Votes per modelVotes per model pose bin Prior on assignments (before actual observations)

22. Experiments

23. UIUC database – Models 8 objects, 20 views/object

24. UIUC database – Test scenes 52 test scenes

25. Giuseppe Toys database – Models 61 objects, 1-2 views/object

26. Giuseppe Toys database – Test scenes 141 test scenes

27. a. Object found, correct pose  Detection b. Object found, incorrect pose  False alarm c. Wrong object found  False alarm d. Object not found  Non detection Performance evaluation Test image hand-labeled before the experiments

28. Results – UIUC database Test scenes Models from database ROC Threshold on probabilistic score.

29. Results – Giuseppe Toys database Lowe’99,’04 Lower false alarm rate - more systematic verification of geometry consistency - more consistent verification of geometric consistency undetected objects: features with poor Appearance distinctiveness Index to incorrect models - +

30. Results – Giuseppe Toys database

31. Coarse-to-fine strategy Probabilistic interpretation of each step Higher performance, comparable speed Conclusions

32. Result from ‘Google Images’ Looking for Grandma (or Brad Pitt) in collaboration with Alex Holub

33. Our result Looking for Grandma (or Brad Pitt)

34. Uses Viola-Jones face detection Face detection Confidence scores (M.Everingham) used to reject false alarms

35. Face warped to offset orientation and scale Patches extracted around face features  Face representation  Face representation by a long vector (eg. 19x11x11=2299) Dimensionality reduction to 50 dimensions via PCA

36. Similarity matrix in feature space

37. What face features are most useful ?

38. What size of patch is most useful ?

39. Distance optimization Goal: - small distance within images of same individual - larger distance between images of different person Optimization of following cost function Gradient descent used for optimization

40. Results – Tom Cruise

41. Results – Ronald Reagan

42. To conclude - Matlab demo