Eduard Serradell Domingo July 28 th, 2011 Simultaneous Point Matching and Recovery of Rigid and Nonrigid Shapes Thesis director Francesc Moreno Noguer Tutor Alberto Sanfeliu Cortés T HESIS P ROPOSAL

Objective Simultaneously solve the correspondence problem and recover rigid and nonrigid shapes. Given two point clouds extracted from different views of the same object, the objective is to simultaneously solve for point correspondence and recover the mapping between the two rigid or nonrigid model representations.

Motivation  Given {u m } from a model point set and {v t } from a target point set, find: model point set target point set

Motivation CORRESPONDENCES  Given {u m } from a model point set and {v t } from a target point set, find: umum vtvt

Motivation CORRESPONDENCES TRANSFORM ESTIMATION H est  Given {u m } from a model point set and {v t } from a target point set, find: umum vtvt

Motivation CORRESPONDENCES TRANSFORM ESTIMATION H est  Given {u m } from a model point set and {v t } from a target point set, find: umum vtvt SIMULTANEOUSLY

Motivation  Common problems: correspondences that do not fit the model umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: partial matching parts of the scene are occluded 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D umum vtvt

Motivation  Common problems: points that do not belong to the model and hinder the recognition umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: error in the position of points umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: regular structures are indistinguishable algorithms fall into local minima umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: umum vtvt model shapes can undergo rigid or nonrigid deformations 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: model shapes can undergo rigid or nonrigid deformations umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: umum vtvt the transform can be embedded in 2D or 3D or projective as in monocular view case (2D-3D) Coordinate System 1 Coordinate System 2 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: umum vtvt u m = R · v t + t Coordinate System 1 Coordinate System 2 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: World Coordinate System Camera Coordinate System the transform can be embedded in 2D or 3D or projective as in monocular view case (2D-3D) 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

Motivation  Common problems: World Coordinate System Camera Coordinate System x cam = A [R| t] x world 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

State-of-Art 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Rigid 2D-2D / 3D-3DRigid 2D-3D Nonrigid 2D-2D / 3D-3D Nonrigid 2D-3D

Nonrigid 2D-2D / 3D-3D Rigid 2D-3D 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model State-of-Art Rigid 2D-2D / 3D-3D

State-of-Art RIGID 2D-2D / 3D-3D MATCHING [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography RANSAC (global solution / high complexity)

State-of-Art RIGID 2D-2D / 3D-3D MATCHING [1992] Besl & McKay, A Method for Registration of 3D Shapes [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography ICP : Iterative Closest Point (requires good initialization)

State-of-Art RIGID 2D-2D / 3D-3D MATCHING [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography [1992] Besl & McKay, A Method for Registration of 3D Shapes [2005] Chum & Matas. Matching With PROSAC - Progressive Sample Consensus PROSAC: RANSAC + Appearance

State-of-Art RIGID 2D-2D / 3D-3D MATCHING [1992] Besl & McKay, A Method for Registration of 3D Shapes [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography [2005] Chum & Matas. Matching With PROSAC - Progressive Sample Consensus Unsolved when: Weak detection, -outliers, -occlusions, -image noise, -repetitive patterns, -highly textured scenes, -oblique angles -… PROSAC complexity similar to RANSAC (too high)

Rigid 2D-2D / 3D-3D Nonrigid 2D-3D Nonrigid 2D-2D / 3D-3D 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model State-of-Art Rigid 2D-3D

State-of-Art Known correspondences : Perspective-n-Point (PnP) (old problem) RIGID 2D-3D MATCHING [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem

State-of-Art RIGID 2D-3D MATCHING [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography RANSAC: Random Sampled Consensus & DLT: Direct Linear Transform (high complexity)

State-of-Art RIGID 2D-3D MATCHING [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography [2002] David et al,SoftPOSIT: Simultaneous Pose and Correspondence Determination SoftPOSIT: Unknown correspondences

State-of-Art RIGID 2D-3D MATCHING [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography [2002] David et al,SoftPOSIT: Simultaneous Pose and Correspondence Determination [2008] Moreno-Noguer et al, Pose Priors for Simultaneously Solving Alignment and Correspondence. Blind PnP: Unknown correspondences Pose Priors (geometrical consistency) Kalman Filter to propagate pose uncertainty

State-of-Art RIGID 2D-3D MATCHING [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography [2002] David et al,SoftPOSIT: Simultaneous Pose and Correspondence Determination [2008] Moreno-Noguer et al, Pose Priors for Simultaneously Solving Alignment and Correspondence. Modeling the uncertainty : Kalman Filter linearizes the uncertainty model Work with Bayesian non-parametric models (Gaussian Processes)

State-of-Art 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Nonrigid 2D-3D Rigid 2D-2D / 3D-3D Rigid 2D-3D Nonrigid 2D-2D / 3D-3D

State-of-Art NONRIGID 2D-2D / 3D-3D MATCHING [2003] Chui & Rangarajan. A New Point Matching Algorithm for Non-Rigid Registration [1998] Gold et al.. New Algorithms for 2D and 3D Point Matching: Point Estimation and Correspondence Soft-assign + Thin-plate splines (requires good initialization) (smooth deformations) Soft-assign (requires good initialization)

State-of-Art [2003] Hannel et al.. An Extension of the ICP Algorithm for Modeling Nonrigid Objects with Mobile Robots NONRIGID 2D-2D / 3D-3D MATCHING [2003] Chui & Rangarajan. A New Point Matching Algorithm for Non-Rigid Registration [2008] Li et al, Global Correspondence Optimization for Non-Rigid Registration of Depth Scans [1998] Gold et al.. New Algorithms for 2D and 3D Point Matching: Point Estimation and Correspondence Nonrigid ICP Variants (Require a good initialization)

State-of-Art [2003] Hannel et al.. An Extension of the ICP Algorithm for Modeling Nonrigid Objects with Mobile Robots NONRIGID 2D-2D / 3D-3D MATCHING [2003] Chui & Rangarajan. A New Point Matching Algorithm for Non-Rigid Registration [2008] Li et al, Global Correspondence Optimization for Non-Rigid Registration of Depth Scans [2010] Deng et al, Retinal Fundus Image Registration via Vascular Structure Graph Matching [1998] Gold et al.. New Algorithms for 2D and 3D Point Matching: Point Estimation and Correspondence [2002] Belongui, Shape matching and Object Recognition Using Shape Contexts Shape appearance + Thin-plate splines (Smooth deformations) Graph Matching + Thin-plate splines (Smooth deformations)

State-of-Art [2003] Hannel et al.. An Extension of the ICP Algorithm for Modeling Nonrigid Objects with Mobile Robots NONRIGID 2D-2D / 3D-3D MATCHING [2003] Chui & Rangarajan. A New Point Matching Algorithm for Non-Rigid Registration [2008] Li et al, Global Correspondence Optimization for Non-Rigid Registration of Depth Scans [2010] Myronenko & Song, Point-Set Registration: Coherent Point Drift [2010] Deng et al, Retinal Fundus Image Registration via Vascular Structure Graph Matching [1998] Gold et al.. New Algorithms for 2D and 3D Point Matching: Point Estimation and Correspondence [2002] Belongui, Shape matching and Object Recognition Using Shape Contexts Unsolved problems Harsh deformations (Gaussian Processes vs TPS) Avoid local minima (Global Search vs ICP et al.)

Rigid 2D-2D / 3D-3D Rigid 2D-3D Nonrigid 2D-2D / 3D-3D State-of-Art 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Nonrigid 2D-3D

State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures

State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion Discriminative methods: Database learning & Nearest Neigbour Selection

State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures [2010] Sanchez et al., Simultaneous Pose, Correspondence and Non-Rigid Shape [2007] Salzmann et al., Surface Deformation Models for Non-Rigid 3D Shape Recovery [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion Generative methods: PCA model of the surface

State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures [2009] Groher, Deformable 2D-3D Registration of Vascular Structures in a One View Scenario [2010] Sanchez et al., Simultaneous Pose, Correspondence and Non-Rigid Shape [2007] Salzmann et al., Surface Deformation Models for Non-Rigid 3D Shape Recovery [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion Thin-plate Splines (Medical Imaging!)

State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures [2010] Salzmann & Urtasun, Combining Discriminative and Generative Methods for 3D Deformable Surface and Articulated Pose Reconstruction [2010] Sanchez et al., Simultaneous Pose, Correspondence and Non-Rigid Shape [2007] Salzmann et al., Surface Deformation Models for Non-Rigid 3D Shape Recovery [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion [2009] Groher, Deformable 2D-3D Registration of Vascular Structures in a One View Scenario Combining Discriminative & Generative methods

State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures [2010] Salzmann & Urtasun, Combining Discriminative and Generative Methods for 3D Deformable Surface and Articulated Pose Reconstruction [2010] Sanchez et al., Simultaneous Pose, Correspondence and Non-Rigid Shape [2007] Salzmann et al., Surface Deformation Models for Non-Rigid 3D Shape Recovery [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion [2009] Groher, Deformable 2D-3D Registration of Vascular Structures in a One View Scenario Potential improvements Better parameterization for articulated structures Harsh deformations (Gaussian Processes vs TPS)

Nonrigid 2D-3D Nonrigid 2D-2D / 3D-3D 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Contributions Rigid 2D-3D Rigid 2D-2D / 3D-3D

Contributions Rigid registration Homography (2D-to-2D) estimation

Rigid registration  Feature Point Detectors assigns multiple correspondences PROSAC picks just the one with better similarity score !!! Contributions Homography (2D-to-2D) estimation

Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point Contributions Homography (2D-to-2D) estimation

Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point  Iterative approach Contributions Homography (2D-to-2D) estimation

Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point  Iterative approach Contributions Homography (2D-to-2D) estimation

Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point  Iterative approach Contributions Homography (2D-to-2D) estimation

Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point  Iterative approach & backtracking when necessary Contributions Homography (2D-to-2D) estimation

Contributions Rigid registration  Some results : PROSACBlind Homography

Contributions Rigid registration  E. Serradell, M. Ozuysal, V. Lepetit, P. Fua and F. Moreno-Noguer: Combining Geometric and Appearance Priors for Robust Homography Estimation. In ECCV 2010 Homography (2D-to-2D) estimation

Rigid 2D-2D / 3D-3D Rigid 2D-3D Nonrigid 2D-2D / 3D-3D Contributions 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Nonrigid 2D-3D

Contributions Nonrigid registration  Nonrigid 2D-to-3D registration  Project in collaboration with  New parameterization for articulated models 2D X-ray ImageCT 3D Volume

Contributions Nonrigid registration  Recursive parameterization of the nodes of the articulated structure 2D X-ray Image CT 3D Volume Camera (known)

Contributions Nonrigid registration  Recursive parameterization of the nodes of the articulated structure  Generative model: Probabilistic PCA 2D features synthetic samples Camera (known)

Contributions Nonrigid registration  Recursive parameterization of the nodes of the articulated structure  Generative model: Probabilistic PCA 2D features generative model Camera (known)

Contributions Nonrigid registration  Recursive parameterization of the nodes of the articulated structure  Generative model: Probabilistic PCA  Iterative update 1.- Kalman Filter model projection 2.- Assign correspondences 2D features generative model Camera (known)

Contributions Nonrigid registration  E. Serradell, A. Romero, R. Leta, C. Gatta and F. Moreno-Noguer: Simultaneous Correspondence and Non-Rigid 3D Reconstruction of the Coronary Tree from Single X-ray Images. In ICCV D X-ray ImageCT 3D Volume shape prior recovered model

Contributions 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Nonrigid 2D-3D Rigid 2D-2D / 3D-3D Rigid 2D-3D Nonrigid 2D-2D / 3D-3D

Contributions Nonrigid registration  Nonrigid 2D-to-2D or 3D-to-3D registration Optical Microscope Image Stack Electron Microscope Image Stack m10 -7 m Partial matching

Contributions Nonrigid registration  Nonrigid 2D-to-2D or 3D-to-3D registration  Extract neuronal tree  Graph Matching Optical Microscope Image Stack Electron Microscope Image Stack

Contributions Nonrigid registration  Nonrigid 2D-to-2D or 3D-to-3D registration  Extract neuronal tree  Graph Matching  Two step process 1.- Affine transform (Kalman Filter approach) Optical Microscope Image Stack Electron Microscope Image Stack y = A x + b

Contributions Nonrigid registration  Nonrigid 2D-to-2D or 3D-to-3D registration  Extract neuronal tree  Graph Matching  Two step process 1.- Affine transform (Kalman Filter approach) 2.- Nonrigid transform (Gaussian Processes for regression) Optical Microscope Image Stack Electron Microscope Image Stack y = A x + b + f(x)

Contributions Nonrigid registration  Some results original graphs affine transform nonlinear refining

Contributions Nonrigid registration  Some results original graphs affine transform nonlinear refining affine transform nonlinear refining

Contributions Nonrigid registration  E. Serradell, J. Kybic, F. Moreno-Noguer and P. Fua: Robust Elastic 2D/3D Geometric Graph Matching, submitted to SPIE Medical Imaging Optical Microscope Image Stack Electron Microscope Image Stack

Contributions 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Rigid 2D-2D / 3D-3DRigid 2D-3D Nonrigid 2D-2D / 3D-3D Nonrigid 2D-3D

Global solution to point registration  Valid for 2D-2D,3D-3D,2D-3D / rigid and nonrigid models  Using Gaussian Processes … some preliminary results: Contributions initial shape recovered shape

1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis

1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis DoneOn-goingTo Do

1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis Research Stay at CVLAB (EPFL) DoneOn-goingTo Do

1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis Project in collaboration with CVC (UAB), MAiA (UB) and Hospital Sant Pau DoneOn-goingTo Do

1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis Research Stay at CVLAB (EPFL) DoneOn-goingTo Do

1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis Research Stay at CVLAB (EPFL) DoneOn-goingTo Do

1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis DoneOn-goingTo Do

Published papers  E. Serradell, M. Ozuysal, V. Lepetit, P. Fua and F. Moreno-Noguer: Combining Geometric and Appearance Priors for Robust Homography Estimation. In ECCV 2010*  E. Serradell, A. Romero, R. Leta, C. Gatta and F. Moreno-Noguer: Simultaneous Correspondence and Non-Rigid 3D Reconstruction of the Coronary Tree from Single X-ray Images. In ICCV 2011* * ICCV, ECCV acceptance rate < 30% Submitted papers  E. Serradell, J. Kybic, F. Moreno-Noguer and P. Fua: Robust Elastic 2D/3D Geometric Graph Matching, submitted to SPIE Medical Imaging Summary of achievements

THANKS!