Real-Time Feature Matching using Adaptive and Spatially Distributed Classification Trees Aurélien BOFFY, Yanghai TSIN, Yakup GENC

S I E M E N S C O R P O R A T E R E S E A R C H 2 SCR© S I E M E N S C O R P O R A T E R E S E A R C H 2 SCR© Real-time wide-baseline feature matching  Object detection  Pose estimation  Tracking  3D reconstruction  Augmented reality  …

S I E M E N S C O R P O R A T E R E S E A R C H 3 SCR© S I E M E N S C O R P O R A T E R E S E A R C H 3 SCR© Previous work  Lucas – Kanade method (1981)  Harris detector (1988)  small motion assumption - short baseline  SIFT (Lowe, 1999)  Affine invariant features Hessian-Affine (Mikolajczyk and Schmid, 2002) Maximally Stable Extrema Regions (MSER) (Matas et al., 2002) etc.  too slow for a real-time application  Randomized Trees for Real-Time Keypoint Recognition V. Lepetit, P. Lagger and P. Fua,CVPR 2005

S I E M E N S C O R P O R A T E R E S E A R C H 4 SCR© S I E M E N S C O R P O R A T E R E S E A R C H 4 SCR© Limitations of the method of Lepetit et al. Results with the method of Lepetit et al. : Results with our improvements : Sensivity to viewpoint changes Sensivity to illumination conditions

S I E M E N S C O R P O R A T E R E S E A R C H 5 SCR© 1.The original method of Lepetit and colleagues 2. Our improvements 2.1. Adaptive trees 2.2. Spatially distributed trees 3. Experimental results S I E M E N S C O R P O R A T E R E S E A R C H 5 SCR©

S I E M E N S C O R P O R A T E R E S E A R C H 6 SCR© 1.The original method of Lepetit and colleagues 2. Our improvements 2.1. Adaptive trees 2.2. Spatially distributed trees 3. Experimental results S I E M E N S C O R P O R A T E R E S E A R C H 6 SCR©

S I E M E N S C O R P O R A T E R E S E A R C H 7 SCR© Feature matching as a classification problem Real-time wide-baseline feature matching Input for the training phase: a single image the object of interestSelection of N prominent keypoints

S I E M E N S C O R P O R A T E R E S E A R C H 8 SCR© Keypoint selection

S I E M E N S C O R P O R A T E R E S E A R C H 9 SCR© P ( f  C j | f reaches this leaf ) Classification trees

S I E M E N S C O R P O R A T E R E S E A R C H 10 SCR© Tree construction

S I E M E N S C O R P O R A T E R E S E A R C H 11 SCR© 1.The original method of Lepetit and colleagues 2. Our improvements 2.1. Adaptive trees 2.2. Spatially distributed trees 3. Experimental results S I E M E N S C O R P O R A T E R E S E A R C H 11 SCR©

S I E M E N S C O R P O R A T E R E S E A R C H 12 SCR© object of interest input image inliers and outliers

S I E M E N S C O R P O R A T E R E S E A R C H 13 SCR© Correctly classified feature class feature class probability feature class probability Reinforcement of the inliers

S I E M E N S C O R P O R A T E R E S E A R C H 14 SCR© object of interest input image Recovering the lost matches ?

S I E M E N S C O R P O R A T E R E S E A R C H 15 SCR© probability feature class probability misclassified feature class Recovering the lost matches

S I E M E N S C O R P O R A T E R E S E A R C H 16 SCR© Geometric modeling imperfections for non-planar patches can be compensated for.  Even for a complex 3D object, only a single image (plus the 3D-coordinates of the selected features) can be used as input to construct the trees. How does the proposed method work? Appearance modeling inaccuracy (limited resolution of the input model, shadows, reflection, etc.) can also be compensated for.  Extension of the working ranges of viewpoints and illumination conditions.

S I E M E N S C O R P O R A T E R E S E A R C H 17 SCR© 1.The original method of Lepetit and colleagues 2. Our improvements 2.1. Adaptive trees 2.2. Spatially distributed trees 3. Experimental results S I E M E N S C O R P O R A T E R E S E A R C H 17 SCR©

S I E M E N S C O R P O R A T E R E S E A R C H 18 SCR© object of interest Spatially distributed trees

S I E M E N S C O R P O R A T E R E S E A R C H 19 SCR© 1.The original method of Lepetit and colleagues 2. Our improvements 2.1. Adaptive trees 2.2. Spatially distributed trees 3. Experimental results S I E M E N S C O R P O R A T E R E S E A R C H 19 SCR©

S I E M E N S C O R P O R A T E R E S E A R C H 20 SCR© More stable results Original method With our improvements

S I E M E N S C O R P O R A T E R E S E A R C H 21 SCR© Original method With our improvements More robust to illumination changes

S I E M E N S C O R P O R A T E R E S E A R C H 22 SCR© Adaptation to sudden illumination changes

S I E M E N S C O R P O R A T E R E S E A R C H 23 SCR© Adaptation of the classification trees to the real viewing conditions by analyzing the matching performance on the frames actually captured by the camera. Spatially distribution of the trees, so that each of them models a certain viewing volume more precisely.  More stable results  Extended detectable range  More robust to illumination changes Contributions

S I E M E N S C O R P O R A T E R E S E A R C H 24 SCR© Thank you

S I E M E N S C O R P O R A T E R E S E A R C H 25 SCR© The problem of occlusion If the camera is continuously moving, erroneous update usually does not happen repeatedly for the same class in the same leaf node. If the true feature is exposed later, the trees can adapt back to the real correspondence.

S I E M E N S C O R P O R A T E R E S E A R C H 26 SCR© Influence of adaptive trees with offline and/or online adaptation