1. Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University of Singapore * Presented by Binh-Son Hua

2. Problem Statement Range images Color images from untracked camera... 3D model Colored 3D model Automatically register color images to 3D model 2

3. Motivations Applications of active range sensing Manufacturing, cultural heritage modeling, etc. Photometric properties needed for visually-realistic models Only some range scanners can capture color Color may not have required resolution E.g. for close-up or zoomed-in views of paintings View-dependent reflection requires many color images from different directions Therefore, better to capture color separately However, impractical to manually register color images to 3D geometry 3

4. Previous Work Feature-based approaches Match corresponding features in both color images and 3D model Can be fully automated Restricted to certain types of objects [Stamos & Allen, ICCV 2001], [Liu & Stamos, CVPR 2005] Statistics-based approaches Used only if reflected intensities of range sensing light were recorded with range data Sensing light often not in visible light spectrum Compute statistical dependence between color images and sensing light intensities Mutual information, chi-square, cross-correlation Camera calibrated & tracked, or co-locate with scanner [Williams et al, 2004], [Hantak & Lastra, 3DPVT 2006] 4

5. Our Approach Color images... Detailed scanned 3D modelColored 3D model Color mapping Registration Multiview geometry reconstruction Sparse 3D model 5

6. Steps 1. Data acquisition 2. Multiview geometry reconstruction 3. Approximate registration of sparse model to detailed model 4. Registration refinement 5. Color mapping 6

7. 1. Data Acquisition Range data Laser range scanner Color images Uncalibrated and untracked digital camera Project special light pattern on large textureless surfaces Improve image feature detection and MVG reconstruction 7

8. 2. MVG Reconstruction Detect and match features in color images Use SIFT Compute MVG Structure-from-motion Incrementally add a new image and apply sparse bundle adjustment (SBA) Result is a sparse 3D model 3D point cloud Camera parameters 8

9. 2. MVG Reconstruction Example sparse 3D model 9

10. 3. Approximate Registration To align sparse model with detailed model Unknown relative scale and pose Register one image in MVG to 3D model User input 6 point correspondences Estimated transformation propagated to other views and 3D points in MVG Sparse model only approximately aligned to detailed model Error in user inputs Error in MVG Geometric distortion in detailed model 10

11. 4. Registration Refinement Need non-rigid alignment of MVG with detailed model To overcome geometric distortion in range images Registration refinement Automatically detect planes in detailed model Identify 3D points in MVG near the planes Refine MVG to minimize distance between 3D points and planes Easily incorporated into sparse bundle adjustment Better than using ICP algorithm Two models are treated as rigid shapes Cannot refine MVG 11

12. 4. Registration Refinement Example result 12 Before registration refinement After registration refinement

13. 5. Color Mapping Colors from different views can be used for view- dependent rendering View-dependent texture mapping Surface light field We simply want to assign a single color to each surface point, but Simple averaging blurs out details Different exposures Occlusions Depth boundaries Vignetting and view-dependent reflection 13

14. 5. Color Mapping Use weighted blending Use lower weights near image and depth boundaries Preserve fine details 14 With details preservation Without details preservation

15. 5. Color Mapping Smooth color and intensity transitions With weighted blending Without weighted blending 15

16. Result Office scene 30 color images (7 with projected pattern) 16

17. Conclusion Achieve accuracies within 3–5 pixels everywhere on each image Not reliant on detection of any specific type of features in both color images and geometric model Project light pattern to improve robustness of MVG Better registration accuracy in face of geometric distortion Effective color mapping method 17

18. Acknowledgements The Photo Tourism team For sharing part of their code on MVG Prashast Khandelwal For contribution to preliminary work Singapore Ministry of Education For the funding 18