Matching and Recognition in 3D

Moving from 2D to 3D – Some Things are Easier No occlusion (but sometimes missing data instead)No occlusion (but sometimes missing data instead) Segmenting objects often simplerSegmenting objects often simpler

Moving from 2D to 3D – Many Things are Harder Rigid transform has 6 degrees of freedom vs. 3Rigid transform has 6 degrees of freedom vs. 3 – Brute-force algorithms much less practical Rotations do not commuteRotations do not commute – Difficult to parameterize, search over No natural parameterization for surfaces in 3DNo natural parameterization for surfaces in 3D – Hard to do FFT, convolution, PCA – Exception: range images

Matching / Recognition in 3D Project into 2D, do image matchingProject into 2D, do image matching Structural methods (i.e., part decomposition, graph matching)Structural methods (i.e., part decomposition, graph matching) Shape similarity methodsShape similarity methods Statistical methodsStatistical methods Feature-based methodsFeature-based methods

3D Medial Axis and Shock Scaffolds Medial axis: locus of points equidistant from 2 surfacesMedial axis: locus of points equidistant from 2 surfaces Shock scaffolds [Leymarie & Kimia]: do matching on “sheets” and “lines”Shock scaffolds [Leymarie & Kimia]: do matching on “sheets” and “lines”

Shape Similarity Key difficulty – locating objects under any rigid-body transformationKey difficulty – locating objects under any rigid-body transformation Translation: relatively easy (match centroids)Translation: relatively easy (match centroids) Rotation:Rotation: – Align objects to each other – Align objects to canonical coordinate frame – Rotation-invariant methods

Iterative Closest Points (ICP) Besl & McKay, 1992Besl & McKay, 1992 Start with rough guess for alignmentStart with rough guess for alignment Iteratively refine transformIteratively refine transform

ICP Assume closest points correspond to each other, compute the best transform…Assume closest points correspond to each other, compute the best transform…

ICP … and iterate to find alignment… and iterate to find alignment Converges to some local minimumConverges to some local minimum Correct if starting position “close enough“Correct if starting position “close enough“

Aligning Scans Start with manual initial alignmentStart with manual initial alignment [Pulli]

Aligning Scans Improve alignment using ICP algorithmImprove alignment using ICP algorithm [Pulli]

Aligning Objects With Moments For each point on object, computeFor each point on object, compute Canonical orientation based on eigenvectors (ordered by eigenvalue)Canonical orientation based on eigenvectors (ordered by eigenvalue)

Problem with PCA-Based Alignment If eigenvalues are close, axes unstableIf eigenvalues are close, axes unstable

Rotation-Invariant Descriptors Decompose model into spherical shellsDecompose model into spherical shells Decompose each shell into spherical harmonicsDecompose each shell into spherical harmonics Keep amplitude, throw away phaseKeep amplitude, throw away phase 3D Model Shape Descriptor Rotation Independent Components

Statistical Methods for Matching Shape EGI: extended Gaussian imagesEGI: extended Gaussian images For each direction, what fraction of normals point in that directionFor each direction, what fraction of normals point in that direction Not rotation invariant, but tends to be peakyNot rotation invariant, but tends to be peaky

Shape Distributions Osada, Funkhouser, Chazelle, and DobkinOsada, Funkhouser, Chazelle, and Dobkin Compact representation for entire 3D objectCompact representation for entire 3D object Invariant under translation, rotation, scaleInvariant under translation, rotation, scale Application: search engine for 3D shapesApplication: search engine for 3D shapes

Computing Shape Distributions Pick n random pairs of points on the objectPick n random pairs of points on the object Compute histogram of distancesCompute histogram of distances Normalize for scaleNormalize for scale 3D Model Shape Distribution Random sampling

Comparing Shape Distributions Similarity Measure 3DModelShapeDistribution

Shape Distributions for Simple Shapes

Robustness Results 7 Mugs 7 Missiles

Classification Results

Features on Surfaces Can construct edge and corner detectorsCan construct edge and corner detectors Analogue of 1 st derivative: surface normalAnalogue of 1 st derivative: surface normal Analogue of 2 nd derivative: curvatureAnalogue of 2 nd derivative: curvature – Curvature at each point in each direction – Minimum and maximum: “principal curvatures” – Can threshold or do nonmaximum suppression

3D Identification Using Spin Images Spin images: Johnson and HebertSpin images: Johnson and Hebert “Signature” that captures local shape“Signature” that captures local shape More expressive than curvatureMore expressive than curvature

Computing Spin Images Start with a point on a 3D modelStart with a point on a 3D model Find (averaged) surface normal at that pointFind (averaged) surface normal at that point Define coordinate system centered at this point, oriented according to surface normal and two (arbitrary) tangentsDefine coordinate system centered at this point, oriented according to surface normal and two (arbitrary) tangents Express other points (within some distance) in terms of the new coordinatesExpress other points (within some distance) in terms of the new coordinates

Computing Spin Images Compute histogram of locations of other points, in new coordinate system, ignoring rotation around normal:Compute histogram of locations of other points, in new coordinate system, ignoring rotation around normal:

Computing Spin Images

Spin Image Parameters Size of neighborhoodSize of neighborhood – Determines whether local or global shape is captured – Big neighborhood: more discriminatory power – Small neighborhood: resistance to clutter Size of bins in histogram:Size of bins in histogram: – Big bins: less sensitive to noise – Small bins: captures more detail, less storage

Spin Image Results Range Image Model in Database

Spin Image Results Detected Models