Object Recognition by Discriminative Combinations of Line Segments and Ellipses Alex Chia ^˚ Susanto Rahardja ^ Deepu Rajan ˚ Maylor Leung ˚ ^ Institute for Infocomm Research (I²R), Singapore ˚ Nanyang Technological University, Singapore

Horse-side Image classification –Separate images containing an object category from other images Goals 2

Category-Level Object Detection –Localize all instances of an object category from an image People Face Cow-side Goals – cont. 3

Region based approach –Exploits image pixel brightness or color values –Other classes (e.g. horse) are more defined by their shape Region based approach –Exploits image pixel brightness or color values –Not suitable for complex classes characterized by thin skeletal structures (e.g. bicycle) Existing Approaches 4

Contour based approach –Exploits spatial configuration or statistic of edge pixels –Edge based rich local descriptors –Contour fragments –Shape primitives Contour based approach –Exploits spatial configuration or statistic of edge pixels –Edge based rich local descriptors –Contour fragments –Shape primitives Existing Approaches – cont. Contour based approach –Shape primitives I.Support abstract reasoning (unlike edge based local descriptors) II.Efficient storage demands (unlike contour fragments) III.Efficient comparison across single and multiple scales (unlike contour fragments) 5

Detect object instances and classify images Boost discriminative codeword combinations Construct shape tokens Our contour based approach - outline Detect object instances and classify images Evaluate performance Learn category-specific codebook of shape tokens Boost discriminative codeword combinations Construct shape tokens Extract line segments and ellipses Learn category-specific codebook of shape tokens Dataset Training images Testing images Extract line segments and ellipses Learning phase Evaluation phase 6

Constructing shape tokens Pair a reference primitive to its connected neighbor –Tokens: Ellipse-line, Line-line, Ellipse-ellipse Geometrical and spatial properties –Length, orientation, distance between midpoints, relative primitive positions θrθr θnθn h lrlr lnln wrwr lrlr wrwr θrθr lnln θnθn h 7

Difference in widths A token is compared only to similar typed tokens Differences in their attributes Difference in spatial separation of primitives Difference in orientationDifference in widths Difference in lengths Difference in spatial separation of primitives Difference in orientation Difference in relative primitive positions Comparing shape tokens 8

Clustering for its relative position –Mean-shift clustering Extracting tokens from within the bounding boxes of training objects Learning category-specific codebook Clustering for its scale normalized appearance descriptors –Adapted bisecting 2-medoid clustering Normalized appearance descriptor Normalized translational vector 10

Medoid in each mean-shift as candidate codeword Appearance distance allowance Indicate range of appearance candidate represents = Mean appearance distance + Std. dev. Scale normalized circular window Indicate where candidate is found relative to object centroid center and radius of window: Medoid in each mean-shift as candidate codeword Appearance distance allowance Indicate range of appearance candidate represents Learning category-specific codebook – cont. Mean-shift sub-cluster feature space x x x x x x x + = Mean appearance distance + Std. dev. Medoid in each mean-shift as candidate codeword 11

Learning category-specific codebook – cont. Score each candidate by appearance + geometric qualities Number of unique training objects Candidates from all sub-clusters Candidates from 350 most populated sub-clusters Appearance qualities Geometric quality 12

Learning category-specific codebook – cont. Radial ranking method to select candidate into codebook 13

Learning category-specific codebook – cont. Candidates from all sub-clusters Candidates from 350 most populated sub-clusters Candidates from 350 selected sub-clusters FaceBike-frontBottle Horse-sideCow-side 14

Matching codeword combination Every codeword in combination finds image tokens within (appearance constraint) Centroid predictions by all codewords in combination concur (geometric constraint) Learning discriminative codeword combinations Each codeword parameterized by Appearance distance allowance Scale normalized circular window with radius and center Matching codeword combination Every codeword in combination finds image tokens within (appearance constraint) Centroid predictions by all codewords in combination concur (geometric constraint) 15

For a scale ‘s’ and location ‘x’, all codewords find matching tokens within its estimated window, will predict centroid locations which concur Learning discriminative codeword combinations – cont. Basic idea for finding matched codeword combinations x x x = (0,0)+ + x x Given codeword i and codeword j, for a scale ‘s’ and location ‘x’ in an image 16

Learning discriminative codeword combinations – cont. Finding token t* within estimated window that has the least appearance distance to codeword x x = (0,0) + x x x x [0, 2] if matching token found within window = x x x otherwise x x x x x x Response of codeword i at scale ‘s’ and location ‘x’ of image 17

Simple example (2 codewords) –Matching of codewords ‘i’ and ‘j’ at scale s and location x –Generalized form and pipi pjpj pipi pjpj p, [0, 2] {-1 or +1} where, and … pipi pjpj pipi pjpj Binary decision tree Learning discriminative codeword combinations – cont. Visual aspects of tokens Spatial layout of tokens Relationships of tokens and, [0, 2] Direction of inequality Structural constraints of object class AppearanceGeometric+ + constraints of object class AppearanceGeometricStructural+ + constraints of object class AppearanceGeometricStructural+ + Predicted label 18 predicted label i p and predicted label i p j p and predicted label and j p i p k p  iii pxs , i p

Input Output …… … Learning discriminative codeword combinations – cont. …… … … … …… … Matrix of values … Vector of z labels Weight vector … Boosting Output …… … Detection confidence: 19

False positives per image Recall Shotton et. al. I Shotton-et. al. II (Retrained test) Bai et. al. Our method Detection RP-AUC False positive rate True positive rate Shotton et. al. I Shotton-et. al. II (Retrained test) Our method Classification ROC-AUC Experimental Results – Weizmann horse J. Shotton et.al., TPAMI, X. Bai et. al., ICCV, 2009.

Average across categories Object category Number of object images Image classification results ROC-AUC Object detection results RP-AUC TrainingTesting Our methodShotton et. al.Our methodShotton et. al. Object category Number of object images Image classification results ROC-AUC Object detection results RP-AUC TrainingTesting Our methodShotton et. al.Our methodShotton et. al. Plane Motorbike Face Car-rear Car-2/3-rear Car-front Bike-rear Bike-front Bike-side Bottle Cow-front Cow-side Horse-front Horse-side Person Mug Cup Average across categories Experimental Results – Graz-17 J. Shotton et. al, TPAMI, Additional comparisons with other methods provided in paper 21

Presented a contour based recognition approach which exploits simple and generic shape primitives Proposed a method to learn discriminative primitive combinations which have variable number of primitives Demonstrated with extensive experiments across 17 categories the effectiveness of our approach Summary

Thank you