1. CS 1699: Intro to Computer Vision Detection II: Deformable Part Models Prof. Adriana Kovashka University of Pittsburgh November 12, 2015

2. Today: Object category detection Window-based approaches: – Review: Viola-Jones detector – Dalal-Triggs pedestrian detector Part-based approaches: – Implicit shape model – Deformable parts model

3. Object Category Detection Focus on object search: “Where is it?” Build templates that quickly differentiate object patch from background patch Dog or Non-Dog? … Derek Hoiem

4. Viola-Jones detector: features Feature output is difference between adjacent regions Efficiently computable with integral image: any sum can be computed in constant time “Rectangular” filters Value at (x,y) is sum of pixels above and to the left of (x,y) Integral image Adapted from Kristen Grauman and Lana Lazebnik Value = ∑ (pixels in white area) – ∑ (pixels in black area)

5. Considering all possible filter parameters: position, scale, and type: 180,000+ possible features associated with each 24 x 24 window Which subset of these features should we use to determine if a window has a face? Use AdaBoost both to select the informative features and to form the classifier Viola-Jones detector: features Kristen Grauman

6. Viola-Jones detector: AdaBoost Want to select the single rectangle feature and threshold that best separates positive (faces) and negative (non- faces) training examples, in terms of weighted error. Outputs of a possible rectangle feature on faces and non-faces. … Resulting weak classifier: For next round, reweight the examples according to errors, choose another filter/threshold combo. Kristen Grauman

7. Cascading classifiers for detection Form a cascade with low false negative rates early on Apply less accurate but faster classifiers first to immediately discard windows that clearly appear to be negative Kristen Grauman DEMO

8. Dalal-Triggs pedestrian detector 1.Extract fixed-sized (64x128 pixel) window at each position and scale 2.Compute HOG (histogram of gradient) features within each window 3.Score the window with a linear SVM classifier 4.Perform non-maxima suppression to remove overlapping detections with lower scores Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

9. Resolving detection scores Non-max suppression Score = 0.1 Score = 0.8 Adapted from Derek Hoiem

10. Slide by Pete Barnum, Kristen Grauman Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05 Map each grid cell in the input window to a histogram counting the gradients per orientation. Train a linear SVM using training set of pedestrian vs. non-pedestrian windows. Code available: http://pascal.inrialpes.fr/soft/olt/

11. Histogram of gradient orientations – Votes weighted by magnitude – Bilinear interpolation between cells Orientation: 9 bins (for unsigned angles) Histograms in 8x8 pixel cells Slide by Pete Barnum Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

12. Histograms of oriented gradients (HOG) N. Dalal and B. Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005Histograms of Oriented Gradients for Human Detection 10x10 cells 20x20 cells Image credit: N. Snavely

13. Histograms of oriented gradients (HOG) N. Dalal and B. Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005Histograms of Oriented Gradients for Human Detection Image credit: N. Snavely

14. Histograms of oriented gradients (HOG) N. Dalal and B. Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005Histograms of Oriented Gradients for Human Detection

15. Histograms of oriented gradients (HOG) N. Dalal and B. Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005Histograms of Oriented Gradients for Human Detection

16. Slide by Pete Barnum Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05 pos w neg w

17. pedestrian Slide by Pete Barnum Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

18. Detection examples

19. Are we done? Single rigid template usually not enough to represent a category – Many objects (e.g. humans) are articulated, or have parts that can vary in configuration – Many object categories look very different from different viewpoints, or from instance to instance Slide by N. Snavely

20. Deformable objects Images from Caltech-256 Slide Credit: Duan Tran

21. Deformable objects Images from D. Ramanan’s dataset Slide Credit: Duan Tran

22. Parts-based Models Define object by collection of parts modeled by 1.Appearance 2.Spatial configuration Slide credit: Rob Fergus

23. How to model spatial relations? One extreme: fixed template Derek Hoiem

24. Part-based template Object model = sum of scores of features at fixed positions +3+2-2-2.5= -0.5 +4+1+0.5+3+0.5= 10.5 > 7.5 ? ? Non-object Object Derek Hoiem

25. How to model spatial relations? Another extreme: bag of words = Derek Hoiem

26. How to model spatial relations? Star-shaped model Root Part Derek Hoiem

27. How to model spatial relations? Star-shaped model = X X X ≈ Root Part Derek Hoiem

28. Parts-based Models Articulated parts model – Object is configuration of parts – Each part is detectable and can move around Adapted from Derek Hoiem, images from Felzenszwalb

29. Implicit shape models Visual vocabulary is used to index votes for object position [a visual word = “part”] B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004Combined Object Categorization and Segmentation with an Implicit Shape Model visual codeword with displacement vectors training image annotated with object localization info Lana Lazebnik

30. Implicit shape models Visual vocabulary is used to index votes for object position [a visual word = “part”] B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004Combined Object Categorization and Segmentation with an Implicit Shape Model test image Lana Lazebnik

31. Implicit shape models: Training 1.Build vocabulary of patches around extracted interest points using clustering Lana Lazebnik

32. Implicit shape models: Training 1.Build vocabulary of patches around extracted interest points using clustering 2.Map the patch around each interest point to closest word Lana Lazebnik

33. Implicit shape models: Training 1.Build vocabulary of patches around extracted interest points using clustering 2.Map the patch around each interest point to closest word 3.For each word, store all positions it was found, relative to object center Lana Lazebnik

34. Implicit shape models: Testing 1.Given new test image, extract patches, match to vocabulary words 2.Cast votes for possible positions of object center 3.Search for maxima in voting space Lana Lazebnik

35. Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial K. Grauman, B. Leibe Original image Example: Results on Cows

36. Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial K. Grauman, B. Leibe Original imageInterest points Example: Results on Cows

37. Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial K. Grauman, B. Leibe Original imageInterest points Matched patches Example: Results on Cows

38. Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial K. Grauman, B. Leibe 1 st hypothesis Example: Results on Cows

39. Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial K. Grauman, B. Leibe 2 nd hypothesis Example: Results on Cows

40. Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial K. Grauman, B. Leibe Example: Results on Cows 3 rd hypothesis

41. Perceptual and Sensory Augmented Computing Visual Object Recognition Tutorial K. Grauman, B. Leibe Detection Results Qualitative Performance  Recognizes different kinds of objects  Robust to clutter, occlusion, noise, low contrast

42. Discriminative part-based models P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, PAMI 32(9), 2010Object Detection with Discriminatively Trained Part Based Models Root filter Part filters Deformation weights Lana Lazebnik

43. Discriminative part-based models P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, PAMI 32(9), 2010Object Detection with Discriminatively Trained Part Based Models Multiple components Lana Lazebnik

44. Discriminative part-based models P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, PAMI 32(9), 2010Object Detection with Discriminatively Trained Part Based Models Lana Lazebnik

45. Scoring an object hypothesis The score of a hypothesis is the sum of appearance scores minus the sum of deformation costs Appearance weights Subwindow features Deformation weights Displacements Adapted from Lana Lazebnik

46. Detection Lana Lazebnik

47. Training Training data consists of images with labeled bounding boxes Need to learn the filters and deformation parameters Lana Lazebnik

48. Training Our classifier has the form w are model parameters, z are latent hypotheses Latent SVM training: Initialize w and iterate: Fix w and find the best z for each training example Fix z and solve for w (standard SVM training) Lana Lazebnik

49. Car model Component 1 Component 2 Lana Lazebnik

50. Car detections Lana Lazebnik

51. Person model Lana Lazebnik

52. Person detections Lana Lazebnik

53. Bottle model Lana Lazebnik

54. Cat model Lana Lazebnik

55. Cat detections Lana Lazebnik

56. Detection state of the art Object detection system overview. Our system (1) takes an input image, (2) extracts around 2000 bottom-up region proposals, (3) computes features for each proposal using a large convolutional neural network (CNN), and then (4) classifies each region using class-specific linear SVMs. R-CNN achieves a mean average precision (mAP) of 53.7% on PASCAL VOC 2010. For comparison, Uijlings et al. (2013) report 35.1% mAP using the same region proposals, but with a spatial pyramid and bag-of-visual- words approach. The popular deformable part models perform at 33.4%. R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, CVPR 2014.Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation Lana Lazebnik

57. Summary Window-based approaches – Assume object appears in roughly the same configuration in different images – Look for alignment with a global template Part-based methods – Allow parts to move somewhat from their usual locations – Look for good fits in appearance, for both the global template and the individual part templates Next time: Analyzing and debugging detection methods