Matthew Brown University of British Columbia (prev.) Microsoft Research [ Collaborators: † Simon Winder, *Gang Hua, † Rick Szeliski † =MS Research, *=MS Live Labs]

 Panoramic Stitching Digital Image Pro, Windows Live Photogallery, Expression, HDView  3D Modelling Photosynth  Virtual Earth Location Recognition  Image Search Lincoln [ yellow = product, white = technology preview, grey = research ]

[ ]

[ Slide credit: Noah Snavely] Scene reconstruction Photo Explorer Input photographs Relative camera positions and orientations Point cloud Sparse correspondence [ ]  Photosynth is based on Photo Tourism [Snavely, Seitz, Szeliski SIGGRAPH 2006 ]  Photo Tourism uses SIFT for correspondence

[ Seitz et al CVPR 2006, Goesele et al ICCV 2007 ]

[ Photo Tourism – Snavely, Seitz, Szeliski - SIGGRAPH 2006 ] 3D Point Cloud

[ Photo Tourism – Snavely, Seitz, Szeliski - SIGGRAPH 2006 ] 3D Point Cloud

3D Point Cloud [ Photo Tourism – Snavely, Seitz, Szeliski - SIGGRAPH 2006 ]

3D Point Cloud [ Photo Tourism – Snavely, Seitz, Szeliski - SIGGRAPH 2006 ]

 † = for simplicity + efficiency  * = measured by ROC curve  Q: Form of the descriptor function f(.)? Find a function of a local image patch descriptor = f ( ) s.t. a nearest neighbour classifier † is optimal*

Algorithm Normalized Image Patch Descriptor Vector Gradients Quantized to k Orientations Normalize Summation [ SIFT – Lowe ICCV 1999 ]

Algorithm Normalized Image Patch Descriptor Vector Gradients Quantized to k Orientations Normalize (plus PCA) Summation [ GLOH – Mikolajzcyk Schmid PAMI 2005 ]

Algorithm Normalized Image Patch Descriptor Vector Create Edge Map Normalize Summation [ Shape Context – Belongie Malik Puzicha NIPS 2000 ]

Algorithm Normalized Image Patch Descriptor Vector Feature Detector Normalize Summation TSN [ Geometric Blur – Berg Malik CVPR 2001 ]

Normalized Image Patch Descriptor Vector TSN Parameters  Propose a framework for descriptor algorithms  Learn parameters to find best performance  Train on a ground truth data set based on accurate 3D matches

Normalized Image Patch (w x h) Descriptor Vector TSN  Transformation block Local gradients Steerable filters Isotropic filters  Haar wavelets  Local classifier  Quantized intensities (w x h x k)  Output: one length k vector per source pixel

Normalized Image Patch (w x h) Descriptor Vector SNT (w x h x k) (m x k)  Spatial summation block with m regions  Output: m length k vectors S1 S2 S3 S4

Normalized Image Patch (w x h) Descriptor Vector SNT (w x h x k) (m x k)  Normalization Block Unit normalization SIFT normalization with clipping

STN

S2T1aN2 Parameters Training Pairs Incorrect Match % Correct Match % Update Parameters (Powell) Descriptor Distances

S2T1aN2 Parameters Test Pairs Incorrect Match % Correct Match % Final Error Rate Descriptor Distances 95%

 Polar lattice S2 always has lower error rate than rectangular S1  Gradient and DOG with S2 beat our SIFT reference (4% vs 6% error)

 Steerable filters produce great results if phase information is kept

 SIFT normalization is important  Best result: 4 th order steerable filters with phase information combined with polar S4-25 Gaussian summation block (2% error vs SIFT at 6%)  Very large numbers of dimensions

w PCA

w LDA

 LDA on pixels ≈ SIFT (6%)  PCA gave small improvement  Normalised patches Gradient patches

Effect of # of Training Pairs  LDA on pixels ≈ SIFT (6%)  PCA gave small improvement  Need ~100,000 training examples

 LDA on T1-T3 < 4.5%  Optimal #dimensions ~20-30  Post-normalisation important T1 T3

LDA using T blocks T1–T4  LDA on T1-T3 < 4.5%  Optimal #dimensions ~20-30  Post-normalisation important

LDA using CVPR 07 descriptors  Overall best results  #dimensions reduced from 100’s to 10’s  Need more challenging dataset!

Algorithm Normalized Image Patch Descriptor Vector Feature Detector Normalize Summation TSN “complex” “simple”

 Used learning to obtain good descriptors  Achieved error rates 1/3 of SIFT  Produced useful ground truth data set  Future Work  Use multi-view stereo ground truth  Multi-level simple-complex architecture  + non-parametric T blocks  Learn interest point detectors [ refs: 1) Winder, Brown CVPR ) Hua, Brown, Winder ICCV 2007 ]

[ ]