1. Object Recognition by Discriminative Combinations of Line Segments and Ellipses Alex Chia ^˚ Susanto Rahardja ^ Deepu Rajan ˚ Maylor Leung ˚ ^ Institute for Infocomm Research (I²R), Singapore ˚ Nanyang Technological University, Singapore

2. Horse-side Image classification –Separate images containing an object category from other images Goals 2

3. Category-Level Object Detection –Localize all instances of an object category from an image People Face Cow-side Goals – cont. 3

4. Region based approach –Exploits image pixel brightness or color values –Other classes (e.g. horse) are more defined by their shape Region based approach –Exploits image pixel brightness or color values –Not suitable for complex classes characterized by thin skeletal structures (e.g. bicycle) Existing Approaches 4

5. Contour based approach –Exploits spatial configuration or statistic of edge pixels –Edge based rich local descriptors –Contour fragments –Shape primitives Contour based approach –Exploits spatial configuration or statistic of edge pixels –Edge based rich local descriptors –Contour fragments –Shape primitives Existing Approaches – cont. Contour based approach –Shape primitives I.Support abstract reasoning (unlike edge based local descriptors) II.Efficient storage demands (unlike contour fragments) III.Efficient comparison across single and multiple scales (unlike contour fragments) 5

6. Detect object instances and classify images Boost discriminative codeword combinations Construct shape tokens Our contour based approach - outline Detect object instances and classify images Evaluate performance Learn category-specific codebook of shape tokens Boost discriminative codeword combinations Construct shape tokens Extract line segments and ellipses Learn category-specific codebook of shape tokens Dataset Training images Testing images Extract line segments and ellipses Learning phase Evaluation phase 6

7. Constructing shape tokens Pair a reference primitive to its connected neighbor –Tokens: Ellipse-line, Line-line, Ellipse-ellipse Geometrical and spatial properties –Length, orientation, distance between midpoints, relative primitive positions θrθr θnθn h lrlr lnln wrwr lrlr wrwr θrθr lnln θnθn h 7

8. Difference in widths A token is compared only to similar typed tokens Differences in their attributes Difference in spatial separation of primitives Difference in orientationDifference in widths Difference in lengths Difference in spatial separation of primitives Difference in orientation Difference in relative primitive positions Comparing shape tokens 8

9. Clustering for its relative position –Mean-shift clustering Extracting tokens from within the bounding boxes of training objects Learning category-specific codebook Clustering for its scale normalized appearance descriptors –Adapted bisecting 2-medoid clustering Normalized appearance descriptor Normalized translational vector 10

10. Medoid in each mean-shift as candidate codeword Appearance distance allowance Indicate range of appearance candidate represents = Mean appearance distance + Std. dev. Scale normalized circular window Indicate where candidate is found relative to object centroid center and radius of window: Medoid in each mean-shift as candidate codeword Appearance distance allowance Indicate range of appearance candidate represents Learning category-specific codebook – cont. Mean-shift sub-cluster feature space x x x x x x x + = Mean appearance distance + Std. dev. Medoid in each mean-shift as candidate codeword 11

11. Learning category-specific codebook – cont. Score each candidate by appearance + geometric qualities Number of unique training objects Candidates from all sub-clusters Candidates from 350 most populated sub-clusters Appearance qualities Geometric quality 12

12. Learning category-specific codebook – cont. Radial ranking method to select candidate into codebook 13

13. Learning category-specific codebook – cont. Candidates from all sub-clusters Candidates from 350 most populated sub-clusters Candidates from 350 selected sub-clusters FaceBike-frontBottle Horse-sideCow-side 14

14. Matching codeword combination Every codeword in combination finds image tokens within (appearance constraint) Centroid predictions by all codewords in combination concur (geometric constraint) Learning discriminative codeword combinations Each codeword parameterized by Appearance distance allowance Scale normalized circular window with radius and center Matching codeword combination Every codeword in combination finds image tokens within (appearance constraint) Centroid predictions by all codewords in combination concur (geometric constraint) 15

15. For a scale ‘s’ and location ‘x’, all codewords find matching tokens within its estimated window, will predict centroid locations which concur Learning discriminative codeword combinations – cont. Basic idea for finding matched codeword combinations x x x = (0,0)+ + x x Given codeword i and codeword j, for a scale ‘s’ and location ‘x’ in an image 16

16. Learning discriminative codeword combinations – cont. Finding token t* within estimated window that has the least appearance distance to codeword x x = (0,0) + x x x x [0, 2] if matching token found within window = x x x otherwise x x x x x x Response of codeword i at scale ‘s’ and location ‘x’ of image 17

17. Simple example (2 codewords) –Matching of codewords ‘i’ and ‘j’ at scale s and location x –Generalized form and pipi pjpj pipi pjpj p, [0, 2] {-1 or +1} where, and … pipi pjpj pipi pjpj Binary decision tree Learning discriminative codeword combinations – cont. Visual aspects of tokens Spatial layout of tokens Relationships of tokens and, [0, 2] Direction of inequality Structural constraints of object class AppearanceGeometric+ + constraints of object class AppearanceGeometricStructural+ + constraints of object class AppearanceGeometricStructural+ + Predicted label 18 predicted label i p and predicted label i p j p and predicted label and j p i p k p  iii pxs , i p

18. Input Output …… … Learning discriminative codeword combinations – cont. …… … … … …… … Matrix of values … Vector of z labels Weight vector … Boosting Output …… … Detection confidence: 19

19. False positives per image Recall Shotton et. al. I Shotton-et. al. II (Retrained test) Bai et. al. Our method 0.8738 0.8903 0.8032 0.9218 Detection RP-AUC False positive rate True positive rate Shotton et. al. I Shotton-et. al. II (Retrained test) Our method 0.9251 0.9400 0.9500 Classification ROC-AUC Experimental Results – Weizmann horse J. Shotton et.al., TPAMI, 2008. X. Bai et. al., ICCV, 2009.

20. 1004000.98260.99530.93250.9310 1004000.99831.00000.99961.0000 1002170.99740.99660.98950.9850 1004000.98830.99920.97970.9912 32140.96430.90000.68430.6925 34160.96880.97270.82560.7233 29130.94680.91720.60420.6398 19120.95840.93750.74210.6344 90530.94450.93660.82990.6959 54640.97730.98020.90090.9468 34160.98440.97270.83350.8575 45650.99440.99920.99450.9975 44220.99180.95660.73680.7852 55960.97560.98160.93610.9680 39180.93520.93210.57300.4271 30200.95250.96000.96190.9035 31200.98000.98250.89640.9158 Average across categories0.97300.96590.84830.8291 Object category Number of object images Image classification results ROC-AUC Object detection results RP-AUC TrainingTesting Our methodShotton et. al.Our methodShotton et. al. Object category Number of object images Image classification results ROC-AUC Object detection results RP-AUC TrainingTesting Our methodShotton et. al.Our methodShotton et. al. Plane Motorbike Face Car-rear Car-2/3-rear Car-front Bike-rear Bike-front Bike-side Bottle Cow-front Cow-side Horse-front Horse-side Person Mug Cup Average across categories0.97300. 96590.84830.8291 Experimental Results – Graz-17 J. Shotton et. al, TPAMI, 2008. Additional comparisons with other methods provided in paper 21

21. Presented a contour based recognition approach which exploits simple and generic shape primitives Proposed a method to learn discriminative primitive combinations which have variable number of primitives Demonstrated with extensive experiments across 17 categories the effectiveness of our approach Summary

22. Thank you ysachia@i2r.a-star.edu.sg