Global and Efficient Self-Similarity for Object Classification and Detection CVPR 2010 Thomas Deselaers and Vittorio Ferrari

Conventional Image Descriptors Measure direct image properties gradients colors 2

Self-Similarity vs Conventional Descriptors [Shechtman, Irani CVPR 07] Assumption of conventional image descriptors There is a direct visual property shared by images of objects of the same class (e.g. colors, gradients, …). This property can be used to compare images. Self-similarity: Indirect property: geometric layout of repeating patches within an image More general property 3

Local Self-Similarity Descriptors 4 [Shechtman, Irani CVPR 07]

Using Local Self-Similarity Descriptors Applications: object recognition, image retrieval, action recognition Ensemble matching [Shechtman CVPR 07] Nearest neighbor matching [Boiman CVPR 08] Bag of local self-similarities [Gehler ICCV09, Vedaldi ICCV09, Hörster ACMM08, Lampert CVPR09, Chatfield ICCV09 WS] 1.Compute LSS descriptors for an image 2.Assign the LSS descriptors to a codebook 3.Represent the image as a histogram of LSS descriptors 5

Self-Similarity goes Global Capture long-range self-similarities and their spatial arrangement 6

Self-Similarity goes Global Capture long-range self-similarities and their spatial arrangement 7

compute self-similarity between all pairs of pixels Global Self-Similarity Tensor 4D self-similarity tensor Note: local self-similarities included 8

Problems with the GSS Tensor Computation time: Memory requirement: Aim: Reduce both 9 11 ∼ 80GB ∼ 20h

Outline Efficient global self-similarity tensor Global self-similarity descriptors – Bag of correlation surfaces – Self-similarity hypercubes Detection with self-similarity hypercubes – Efficient sliding window – Efficient subwindow search Experiments – Global self-similarity better than local self-similarity – Complementary to conventional descriptors – Object detection possible 10

Efficient Global Self-Similarity Tensor Find an efficient approximation to Quantize patches according to codebook If two patches are assigned to the same prototype, they are similar Reduces runtime to speedup:

Efficient Global Self-Similarity Two patches are only similar if they are assigned to the same prototype Reduces memory to reduction: 12

Patch Prototype Codebooks Remember: Self-similarity encodes image content indirectly Image-specific codebooks can be smaller than conventional ones see paper for more generic codebooks and extensive evaluation 13

Self-similarity hypercubes: now Bag of correlation surfaces: only in the paper Global Self-Similarity Descriptors So far: Compact GSS computed efficiently Now: Descriptors that can be used in machine learning classifiers Fixed dimensionality Compact representation 14

Self-Similarity Hybercubes SSH of size 15

SSHs for Detection Computing SSH naïvely requires operations Sliding windows has to evaluate many windows 16 operations

Efficient Computation of SSHs Compute integral self-similarity tensor: operations to compute SSH for an image window 17  ∼ 5000x speedup  can be obtained using 16 lookups in

Efficient Subwindow Search for SSH 18 Derive an upper bound on the score of a set of windows Section 5.2 in our paper Similar to [Lampert PAMI09]

Experiments: Object classification PASCAL 07 objects – 9608 cropped images of objects from PASCAL 07 – 20 classes Task: Classify each test image into one of 20 classes Model: Linear SVM Train: train+val Test: test 19

Classification on the PASCAL 07 objects set + GSS outperform LSS + Self-Similarity is truly complementary to conventional descriptors 20 classification accuracy [%]

Experiments: Object detection ETHZ Shape Classes – 255 images – 5 classes (apple logos, bottles, giraffes, mugs, swans) Task: Detect objects in images Detector: Linear SVM, sliding windows 21 e.g. [Ferrari CVPR07, Maji CVPR09]

Detection Results + SSH outperforms BOLSS + it is possible to use GSS for detection with good results 22 BoLSSSSH apple logos bottles giraffes mugs swans Average DR at FPPI 0.4 apple logos bottles giraffes mug s swans } } SSH BoLSS FPPI 0.4 Comparison results (avg): [Ferrari CVPR07]: 71.9 [Maji CVPR09]: 93.2 … many more DR at 0.5 PASCAL overlap

Runtimes for Computing Descriptors 200x200 image GSS tensor – directly: 5512s ( ∼ 1.5 hours) – using our method: 81s ( ∼ 1.5 minutes) Computing descriptors: few seconds Our method: 70x speedup For Reference: – GIST: 0.4s – BOLSS: 0.7s 23

Runtimes for Detection Given the prototype assignment map (80s) (once only) SSH sliding window: 30s/img (once per class) For Comparison – Computing direct GSS tensor for windows: 4 years/img Speedup: ∼ 1 million ⇒ Using our methods, GSS can be used for object detection For Reference: – Felzenszwalb PAMI 09: 5s. 24 June 2014

Global and Efficient Self-Similarity for Object Classification and Detection CVPR 2010 Thomas Deselaers and Vittorio Ferrari Feasible

Conclusion self-similarity should be considered globally – Global self-similarity performs better than local self-similarity truly complementary to conventional descriptors global self-similarity is feasible – efficient computation of self-similarity – two descriptors based on self-similarity global self-similarity for detection code will be available soon 26

Thank you for your attention