1. Reverse Engineering
Niloy J. Mitra

2. Capturing Geometry: 3D Scanners
laser scanner
3D geometry

3. Shape Acquisition

4. Shape Acquisition

5. Shape Acquisition

6. Shape Acquisition

7. Shape Acquisition

8. Partial Shape Similarity
partial similarity

9. Total vs Partial Matching
Total matching is easy
PCA (Principal Component Analysis)

10. Total vs Partial Matching
Partial matching is difficult
Which region matches which other region(s)?
Space of rigid transforms  rotation + translation
Brute force approach  not feasible
Instead of exhaustive searching, use local geometry to guide where to search
Easy to verify a transform

11. Descriptor Based Alignment
Feature based alignment
Combinatorial search, need multiple objects together
Descriptor based alignment
Fails for partial similarity
[Osada et al. `02]
[Kazhan et al. `03]

12. Registration What is registration?
Finding a one-to-one mapping between two or more coordinate systems such that corresponding features of models in the different systems are mapped to each other
Using the mapping to align a model(s)
Pair-wise model alignment
Transformation to a canonical pose/coordinate system
A one-to-one mapping between the coordinates in one space and those in another such that the points in the two spaces that correspond to the same feature are mapped to each other. (from Maurer).
Audette 2000
M. Kazhdan

13. Registration OR What is the resulting alignment/pose used for?
Object recognition in scenes
Stitching together parts of a model captured from different views
Alignment for pose-dependent shape descriptors OR
Chang and Krumm, 1999
Funkhouser, COS 597D Class Notes

14. Registration Choice of Transformation
Rigid: mutual distances of points within a model are conserved during transformation
R is a rotation matrix and t is a translation vector
Non-rigid
Account for surface deformations in the transformation
Affine transformation
Audette 2000

15. Normalization
Use PCA to place models into a canonical coordinate frame
Covariance Matrix
Computation
Principal Axis
Alignment
In the past, this problem has been addressed by trying to obtain the optimal alignment without explicitly trying all possible rotations. In particular, the models are normalized for rotation by placing each model into its own canonical coordinate frame. Two models are then compared by comparing them at their normalized rotation, and assuming that this provides the optimal pair-wise alignment.
Traditionally, Principal Component Analysis has been used to provide the canonical coordinate frame. In particular, given a model, the covariance ellipsoid is computed, and the model is then rotated so that the principal axes of the ellipsoid align with x-, y-, and z-axes.
M. Kazhdan

16. Steps for finding principal axes
Translate point set {pi} to origin by center of mass:
Result is new point set {qi}

17. Steps for finding principal axes
Calculate second-order covariance matrix:

18. Steps for finding principal axes
Decompose symmetric covariance matrix:
Matrix U contains 3 principal axes (eigenvectors) as rows: A, B, C
Matrix S contains eigenvalues

19. Aligning Scans Start with manual initial alignment [Pulli]
The first thing to do for a new scan is to align or register it with the previous scans. We did an interactive rough alignment by moving the latest scan, shown in red, to approximately match the other scans.
If we had tracked the gantry location, this manual alignment would have been unnecessary.
Start with manual initial alignment
[Pulli]
S. Rusinkiewicz

20. Aligning Scans Improve alignment using ICP algorithm [Pulli]
This is followed by automated accurate alignment with one of the neighboring scans using a variant of ICP, that is, Iterative Closest Points method.
Improve alignment using ICP algorithm
[Pulli]
S. Rusinkiewicz

21. Pairwise Rigid Registration Goal
Align two partially- overlapping meshes given initial guess for relative transform
So, let’s formalize the rigid registration problem as solving for a rigid-body transform (i.e., translation and rotation, or 6 total degrees of freedom in 3D) that minimizes the distance between two partially-overlapping meshes. We’ll focus on iterative algorithms that converge to a local minimum, so we’ll assume that we have an initial guess already.
S. Rusinkiewicz

22. Aligning 3D Data
If correct correspondences are known, can find correct relative rotation/translation
The basic idea behind ICP is that, if we somehow knew correspondences, we could solve for the translation that minimizes pairwise distance.
S. Rusinkiewicz

23. Aligning 3D Data
How to find correspondences: User input? Feature detection? Signatures?
Alternative: assume closest points correspond
What ICP does is make a seemingly-radical guess: that closest points are the correspondences. You take these, run the point pairs through the minimization algorithms, and get… the wrong answer. However, if you started out reasonably close, this process got you closer…
S. Rusinkiewicz

24. Aligning 3D Data … and iterate to find alignment
Iterative Closest Points (ICP) [Besl & McKay 92]
Converges if starting position “close enough”
… which means that you can repeat the process and converge to the right answer.
S. Rusinkiewicz

25. Basic ICP Select e.g. 1000 random points
Match each to closest point on other scan, using data structure such as k-d tree
Reject pairs with distance > k times median
Construct error function:
Minimize (closed form solution in [Horn 87])
So, here’s a slightly formalized version of that algorithm.
S. Rusinkiewicz

26. ICP Variants
Variants on the following stages of ICP have been proposed:
Selecting source points (from one or both meshes)
Matching to points in the other mesh
Weighting the correspondences
Rejecting certain (outlier) point pairs
Assigning an error metric to the current transform
Minimizing the error metric w.r.t. transformation
If we look at the preceeding algorithm, we can see that it consists of a bunch of different stages, and there are different choices you can make at each stage. It turns out that there’s been a lot of research on ICP and, to a good approximation, you can classify all the different tweaks to ICP that people have proposed into variants on 6 different stages.
S. Rusinkiewicz

27. Point-to-Plane Error Metric
Using point-to-plane distance instead of point-to-point lets flat regions slide along each other [Chen & Medioni 91]
The idea is that instead of minimizing the distance between pairs of points, we’ll minimize the distance between one of the points and the plane passing through the other and perpendicular to its normal. (This, of course, assumes that you’ve estimated normals, which is typically done just by fitting planes to small neighborhoods of points, etc.)
The big advantage of this is that it better captures the notion that sliding two planes along each other doesn’t increase the distance between them. Consider, for example, the situation shown at right. You have two scans that are mostly planar, with only the little bump constraining their left-and-right motion. Using the original point-to-point distance, however, the pairs of points in the flat region will prevent the scans from “sliding along each other” to reach the correct transformation: think of them as little springs that want to minimize their length. Using the point-to-plane distance in this case will lead to significantly faster convergence.
S. Rusinkiewicz

28. Point-to-Plane Error Metric
Error function: where R is a rotation matrix, t is translation vector
Linearize (i.e. assume that sin   , cos   1):
Result: overconstrained linear system
So, how do we actually implement this? The first thing is to write down the error function: we’re taking the transformed version of point pi, finding the vector to qi, then taking just the component of that vector along the normal ni. This is a little nasty, since the matrix R involves some trig functions, etc., and minimizing this error function directly (i.e., solving for the components of R and t) can’t be done in closed form. So, we’re going to make the problem linear by assuming that the angles we need to solve for are small, so we can use the usual first-order approximations to sine and cosine. This lets us rewrite the energy function in a way that’s linear in the unknowns, and solve it using standard linear least squares.
In practice, making the assumption that angles are small is not too bad: the angles will be small in later iterations, while during early iterations the error of this approximation tends to be dominated by the fact that the correspondences aren’t really correct… In short, it works really well in practice.
S. Rusinkiewicz

29. Point-to-Plane Error Metric
Overconstrained linear system
Solve using least squares
So, how do we actually implement this? The first thing is to write down the error function: we’re taking the transformed version of point pi, finding the vector to qi, then taking just the component of that vector along the normal ni. This is a little nasty, since the matrix R involves some trig functions, etc., and minimizing this error function directly (i.e., solving for the components of R and t) can’t be done in closed form. So, we’re going to make the problem linear by assuming that the angles we need to solve for are small, so we can use the usual first-order approximations to sine and cosine. This lets us rewrite the energy function in a way that’s linear in the unknowns, and solve it using standard linear least squares.
In practice, making the assumption that angles are small is not too bad: the angles will be small in later iterations, while during early iterations the error of this approximation tends to be dominated by the fact that the correspondences aren’t really correct… In short, it works really well in practice.
S. Rusinkiewicz

30. Closest Compatible Point
Closest point often a bad approximation to corresponding point
Can improve matching effectiveness by restricting match to compatible points
Compatibility of colors [Godin et al. 94]
Compatibility of normals [Pulli 99]
Other possibilities: curvatures, higher-order derivatives, and other local features
The idea is that instead of just blindly choosing the closest point as your correspondence, you instead try to find better ones. One way of phrasing this is as selecting the closest point that is compatible with the source point, by some metric(s). This can be done based on color, normals, or any of the features covered earlier.
Note that there’s a different way of incorporating more information into the matching process, which is to directly do the matching in a higher-dimensional space, where 3 of the dimensions are just Euclidean coordinates, while the remaining dimensions correspond to your feature space. This certainly can be made to work, but you sometimes encounter difficulties relating to how the different dimensions are scaled. That is, there’s no natural way of saying e.g. how a distance of 3mm in space should be weighted relative to a distance of 10 in a color coordinate. In practice, you need to use something like a Mahalanobis distance (i.e., give the dimensions a weight computed using variances (and covariances) of the data itself).
S. Rusinkiewicz

31. Global Registration Goal
Given: n scans around an object
Goal: align them all
First attempt: ICP each scan to one other
Now let’s turn to global registration. Of course, the problem we’re trying to solve is aligning more than just 2 scans. As hinted earlier, just aligning scans pairwise can lead to disappointing results.
S. Rusinkiewicz

32. Global Registration Goal
Want method for distributing accumulated error among all scans
We really want to optimize the pairwise distances of all pairs of overlapping scans.
S. Rusinkiewicz

33. Correspondence Space Search the whole model for correspondences
Range query for descriptor values
Cluster and pick representatives Q P

34. Evaluating Correspondences
Coordinate root mean squared distance
Requires best aligning transform
Looks at correspondences individually

35. Rigidity Constraint
Pair-wise distances between features and correspondences should be the same Q P

36. Rigidity Constraint
Pair-wise distances between features and correspondences should be the same Q P

37. Rigidity Constraint
Pair-wise distances between features and correspondences should be the same Q P

38. Alignment Results
Input: 2 scans
Our alignment
Refined by ICP

39. Alignment Results
Our alignment
Refined by ICP
Input: 10 scans

40. Non-rigid Alignment Algorithm Input Algorithm Goal
[Allen et al. 2003]
Algorithm Input
Set of human range images
All in similar poses
Set of colored feature markers
Algorithm Goal
Develop correspondence from template to target
Compute affine transform for each vertex
Minimize error function
T. Funkhouser

41. Optimization Variables
Algorithm viewed as optimization problem
Given an initial template surface with vertices vi
Corresponding affine transformation matrices Ti
Current state is Ti vi for all i (see diagram)
Find values of Ti to minimize objective function
Attempts to find a “good fit” (blue) of template (cyan) to target (red)
T. Funkhouser

42. Objective Function Objective Function has three weighted terms
Data error
Smoothness error
Marker error
Will use different weights in each phase of process
Multistep / Multi-resolution fitting process
T. Funkhouser

43. Objective Function – Data Error
Data Error term requires current match to be close to target
Uses distance from each transformed vertex to the target surface
Weighted by confidence measure (from scanning)
Hole regions have weight = 0
Sums total error
Distance function
Uses transformed template vertex
Takes minimum distance to “compatible” vertices in target
Compatible defined as those with normal w/in 90 deg.
T. Funkhouser

44. Objective Function – Smoothness Error
Measures smoothness of deformation applied to template
Problem is under-constrained using data error
E_s measures change in T_I between adjacent vertices
Encourages similarly-shaped features to be mapped to each other
Uses Frobenius norm (vector L2 norm)
T. Funkhouser

45. Objective Function – Marker Error
Data and Smoothness Error can hit local minima
Example: left arm transformed to right arm
Solution: Use pre-labeled markers on the test subjects
Viewed as white dots in the range image
Correspondences set up beforehand (as in Consistent Mesh Parameterization)
74 markers per subject (not all are used, however)
Measure distance from template marker to target marker
K_I are the indices of the markers in template, m_I are target markers
T. Funkhouser

46. Algorithm Procedure Minimize error function using L-BFGS-B algorithm
Quasi-Newton method with limited memory usage
T. Funkhouser

47. Motivation – Correct Hole Filling
Also, in some places, such as between the legs, holes should be added
T. Funkhouser

48. More Information http://www.cs.princeton.edu/~bjbrown/iccv05_course/
(Surface Matching and Registration)