1. 3D reconstruction Class 16

2. 3D photography course schedule Introduction Aug 24, 26(no course) Aug.31,Sep.2(no course) Sep. 7, 9(no course) Sep. 14, 16Projective GeometryCamera Model and Calibration (assignment 1) Feb. 21, 23Camera Calib. and SVMFeature matching (assignment 2) Feb. 28, 30Feature trackingEpipolar geometry (assignment 3) Oct. 5, 7Computing FTriangulation and MVG Oct. 12, 14(university day)(fall break) Oct. 19, 21StereoActive ranging Oct. 26, 28Structure from motionSfM and Self-calibration Nov. 2, 4Shape-from-silhouettesSpace carving Nov. 9, 113D modelingAppearance Modeling Nov.12 papers (2-3pm SN115) Nov. 16, 18(VMV’04) Nov. 23, 25papers & discussion(Thanksgiving) Nov.30,Dec.2papers & discussionpapers and discussion Dec.3 papers (2-3pm SN115) Dec. 7?Project presentations

3. Papers Li Exact Voxel Occupancy with Graph Cuts Sudipta Stereo without epipolar lines Chris A graph cut based adaptive structured light approach for real-time range acquisition Nathan Space-time faces Brian Depth-from-focus … Chad Interactive Modeling from Dense Color and Sparse Depth Seon Joo Outdoor calibration of active cameras Jason spectral partitioning Sriram Christine http://www.unc.edu/courses/2004fall/comp/290b/089/papers/

4. Ideas for a project? ChrisWide-area display reconstruction Nathan? BrianDepth-from-focus/defocus LiVisual-hulls with occlusions ChadLaser scanner for 3D environments Seon JooCollaborative 3D tracking JasonSfM for long sequences Sudipta Combining exact silhouettes and photoconsistency SriramPanoramic cameras self-calibration Christine desktop lamp scanner

5. 3D modeling Aligning range images Pairwise Globally Surface reconstruction Single range image Merged (some slides from S. Rusinkiewicz, J. Ponce,…)

6. Aligning 3D Data If correct correspondences are known, it is possible to find correct relative rotation/translation

7. q = a +  q is a quaternion, R a 2 R is its real part, and R  2 R 3 is its imaginary part. Sum of quaternions: Multiplication by a scalar: Quaternion product: Conjugate: ) Operations on quaternions: Norm: Note: Intermezzo: quaternions

8. Let R denote the rotation of angle  about the unit vector u. Define Then for any vector , Reciprocally, if q = a + ( b, c, d ) T is a unit quaternion, the corresponding rotation matrix is: Intermezzo: quaternions and rotations

9. Problem: Find the rotation matrix R and the vector t that minimize i=1 E =  | x i ’ – R x i – t | 2. n At a minimum: 0 =  E/  t = –2  ( x i ’ – R x i – t ). n i=1 Or.. t = x’ – R x. If y i = x i –x and y i ’ = x i ’ –x’, the error is (at a minimum): i=1 E =  | y i ’ – R y i | 2 n i=1 =  | y i ’ q – q y i | 2 n i=1 =  | y i ’ – q y i q| 2 |q| 2 n Or E = Linear least squares !! Estimate rigid transformation

10. Aligning 3D Data How to find corresponding points? Previous systems based on user input, feature matching, surface signatures, etc.

11. Aligning 3D Data Alternative: assume closest points correspond to each other, compute the best transform…

12. Aligning 3D Data … and iterate to find alignment Iterated Closest Points (ICP) [Besl & McKay 92] Converges if starting position “close enough“

13. ICP Variants Classic ICP algorithm not real-time To improve speed: examine stages of ICP and evaluate proposed variants [Rusinkiewicz & Levoy, 3DIM 2001] 1.Selecting source points (from one or both meshes) 2.Matching to points in the other mesh 3.Weighting the correspondences 4.Rejecting certain (outlier) point pairs 5.Assigning an error metric to the current transform 6.Minimizing the error metric

14. ICP Variant – Point-to-Plane Error Metric Using point-to-plane distance instead of point-to- point lets flat regions slide along each other more easily [Chen & Medioni 91]

15. Finding Corresponding Points Finding closest point is most expensive stage of ICP Brute force search – O(n) Spatial data structure (e.g., k-d tree) – O(log n) Voxel grid – O(1), but large constant, slow preprocessing

16. Finding Corresponding Points For range images, simply project point [Blais 95] Constant-time, fast Does not require precomputing a spatial data structure

17. High-Speed ICP Algorithm ICP algorithm with projection-based correspondences, point-to-plane matching can align meshes in a few tens of ms. (cf. over 1 sec. with closest-point) [Rusinkiewicz & Levoy, 3DIM 2001]

18. 3D Global Registration

19. 3D Global registration The problem: Given: n scans around an object Goal: align them all First attempt: ICP each scan to one other

20. Want method for distributing accumulated error among all scans 3D Global registration

21. Approach #1: Avoid the Problem In some cases have 1 scan that covers large part of surface (e.g., cylindrical scan) Align all other scans to this “anchor” Disadvantage: not always practical to obtain anchor scan

22. Approach #2: The Greedy Solution Align each new scan to all previous scans Disadvantages: Order dependent Doesn’t spread out error

23. Approach #3: Brute-Force Solution While not converged: For each scan: For each point: – For every other scan » Find closest point Minimize error w.r.t. transforms of all scans Disadvantage: Solve (np)  (np) matrix equation, where n is number of scans and p is number of points per scan

24. Approach #3a: Slightly Less Brute- Force While not converged: For each scan: For each point: – For every other scan » Find closest point Minimize error w.r.t. transform of this scan Faster than previous method (matrices are p  p)

25. Graph Methods Many globalreg algorithms create a graph of pairwise alignments between scans Scan 1 Scan 5 Scan 4 Scan 3 Scan 2 Scan 6

26. Pulli’s Algorithm Perform pairwise ICPs, record sample (e.g. 200) of corresponding points For each scan, starting w. most connected Align scan to existing set While (change in error) > threshold Align each scan to others All alignments during globalreg phase use precomputed corresponding points

27. Sharp et al. Algorithm Perform pairwise ICPs, record only optimal rotation/translation for each Decompose alignment graph into cycles While (change in error) > tolerance For each cycle: Spread out error equally among all scans in the cycle For each scan belonging to more than 1 cycle: Assign average transform to scan

28. Lu and Milios Algorithm Perform pairwise ICPs, record optimal rotation/translation and covariance for each Least squares simultaneous minimization of all errors (covariance-weighted) Requires linearization of rotations Worse than the ICP case, since don’t converge to (incremental rotation) = 0

29. Open Questions in Global Registration Best way to do correctly-weighted globalreg without linearizing rotations? How to prevent bias (if many scans in one area, few scans in another)? Robust outlier detection For graph methods, pairwise ICP often automated One bad ICP can throw off the entire model

30. Bad ICP in Globalreg One bad ICP can throw off the entire model Correct Globalreg Globalreg Including Bad ICP

31. Huber’s Automatic Modeling System Start with unaligned cans Generate candidate pairwise alignments using spin images Incrementally add scans, checking consistency ICP error in overlapping regions Free space consistency Occupied space consistency

32. Spin Images Johnson and Hebert “Signature” that captures local shape Similar shapes  similar spin images

33. Computing Spin Images Start with a point on a 3D model Find (averaged) surface normal at that point Define coordinate system centered at this point, oriented according to surface normal and two (arbitrary) tangents Express other points (within some distance) in terms of the new coordinates

34. Computing Spin Images Compute histogram of locations of other points, in new coordinate system, ignoring rotation around normal:

35. Computing Spin Images

36. Spin Image Parameters Size of neighborhood Determines whether local or global shape is captured Big neighborhood: more discriminatory power Small neighborhood: resistance to clutter Size of bins in histogram: Big bins: less sensitive to noise Small bins: captures more detail, less storage

37. Using Spin Images for Alignment Not robust

38. Verification – Overlap Distance Views 1 and 2: 48%, 2.1 mm Views 1 and 9: 15%, 3.1 mm

39. Verification – Visibility Consistency consistent surfaces S1S1S1S1 S2S2S2S2 C1C1C1C1 C2C2C2C2 surfaces are close free space violation S1S1S1S1 S2S2S2S2 C1C1C1C1 C2C2C2C2 S 2 blocks S 1 occupied space violation S1S1S1S1 S2S2S2S2 C1C1C1C1 C2C2C2C2 S 1 is not observed

40. Automatically Modeling the Angel top view connectivity after local verification horizontal slice 1 4 9 5 6 8 11 12 14 15 13 10 2 3 7 1 4 9 5 6 8 11 121415 13 10 2 3 7 initial model (bad matches in red)

41. Inferring New Pose Relations 1 4 9 5 6 8 11 12 14 15 13 10 2 3 7 30% overlap 1% overlap 1 9 5 6 8 11 121415 13 10 2 3 4 7

42. horizontal slice The Final Angel Model top view

43. Problems With Reconstruction from Point Clouds

44. Surface Reconstruction from Range Images Often an easier problem than reconstruction from arbitrary point clouds Implicit information about adjacency, connectivity Roughly uniform spacing

45. Surface Reconstruction From Range Images First, construct surface from each range image Then, merge resulting surfaces Obtain average surface in overlapping regions Control point density

46. Range Image Tesselation Given a range image, connect up the neighbors

47. 3D surface model Depth image Triangle mesh Texture image Textured 3D Wireframe model

48. Range Image Tesselation Caveat #1: can’t be too aggressive Introduce distance threshold for tesselation

49. Caveat #2: Which way to triangulate? Possible heuristics: Shorter diagonal Dihedral angle closer to 180  Maximize smallest angle in both triangles Always the same way (best triangle strips) Range Image Tesselation

50. Scan Merging Using Zippering Turk & Levoy, 1994 Erode geometry in overlapping areas Stitch scans together along seam Re-introduce all data Weighted average

51. Zippering

52. Point Weighting Higher weights to points facing the camera Favor higher sampling rates

53. Point Weighting Lower weights (tapering to 0) near boundaries Smooth blends between views

54. Volumetric Reconstruction Implicit function defined volumetrically Usually stored sampled on a 3D grid Can be compressed (e.g., using RLE) Another possibility: hierarchical data structures Can extract isosurface (i.e., subset of space where implicit function = some constant)

55. Volumetric Reconstruction Overview Generate signed distance function (or something close to it) for each scan Compute average (possibly weighted) Extract isosurface

56. Volumetric integration (Curless and Levoy, Siggraph´96) sensor range surfaces volume distance depth weight (~accuracy) signed distance to surface surface1 surface2 combined estimate use voxel space new surface as zero-crossing (find using marching cubes) least-squares estimate (zero derivative=minimum)

57. Volumetric Reconstruction Benefits Always generates a manifold surface Can control sampling density Averaging of signed distance functions corresponds to averaging the surfaces

58. Volumetric Reconstruction Drawbacks Represent a 3D entity rather than 2D Running time Storage Resampling step – bandlimits the function Generates consistent topology, but not always the topology you wanted Problems with very thin surfaces

59. From volume to mesh: Marching Cubes First 2D, Marching Squares “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.

60. From volume to mesh: Marching Cubes “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.

61. From volume to mesh: Marching Cubes Improvement “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.

62. Multiple depth imagesVolumetric integration Volumetric 3D integration

63. Next class: appearance modeling