Semi-automatic Range to Range Registration: A Feature-based Method Chao Chen & Ioannis Stamos Computer Science Department Graduate Center, Hunter College The City University of New York

2 2D Pictures Motivation Goal: highly accurate photo-realistic description of 3D world Applications: urban planning, historical preservation, virtual reality Computational efficient Range scans 3D model Photo-realistic 3D Model Highly accurate Minimum human interaction Registration Texture Mapping

3 Contribution Related methods on range image registration Iterative Closest Point algorithm [Besl & McKay, Chen & Medioni, Rusinkiewicz] Require close initial registration Spin images [Johnson & Hebert] More suitable for curved surfaces; needs accurate normal Previous feature-based algorithm [Stamos & Leordeanu] Exhaustively searching line pairs; high complexity Symmetric structures require manual registration Our semi-automatic 3D registration system No rough pre-registration required Automated registration procedure: Utilize global information to compute transformation ICP algorithm to optimize registration Context-sensitive user interface to: Display registration result at each step Conveniently adjust translation and rotation

4 Outline Automated registration procedures Previous exhaustive search approach Improved automated registration Global stitching process to register all images User interface Experimental results Conclusions and future work

5 Exhaustive Search Approach Range image segmentation Segmented Planar Area Exterior and Interior borders shown Interior border Range sensing direction Each Segmented Planar Area is shown with different color for clarity Intersection line

6 Exhaustive Search Approach Left range image coordinate system Right range image coordinate system Line in left range image Line in right range image Plane normal in left image Plane normal in right image x left y left z left y right x right z right One pair of correctly matched lines provides rotation

7 Exhaustive Search Approach Two pairs of correctly matched lines provide exact translation Rotated l eft range image coordinate system Right range image coordinate system x left y left z left y right x right z right

8 Exhaustive Search Approach Find the two pairs of corresponding lines that maximizes the total number of line matches Consider two corresponding line pairs Compute transformation Grade of computed transform: total number of line matches Keep the transform with the highest grade At the end refine best transform using all matched lines White lines (left scan) Blue lines (right scan) Red/Green lines (matches)

9 Exhaustive Search Approach No initial registration needed High computational complexity Symmetry problem unsolved Improvements Extract object-based coordinate system Context-sensitive user interface

10 Framework of New Solution Lines and planes from segmentation Image1 line clustering Rotation estimation Translation estimation Image2 line clustering Transform refinement by ICP Display registered pair Rotation adjustment Translation adjustment Wrong registration due to symmetry Next pair of scans Correct registration Global stitching Automated RegistrationUser Interactions

11 Line Clustering Line clustering Line directions Plane normals Building’s local coordinate system

12 Rotation Estimation Rotation estimation R = [x 2 y 2 z 2 ] T * [x 1 y 1 z 1 ] 24 possible R’s? y1y1 x1x1 z1z1 y2y2 x2x2 z2z2

13 Rotation Estimation Heuristic: eliminate candidates based on observations Scanner moves on the ground plane: y axis not change much Overlapping images from close by viewpoints: smallest rotation candidate is chosen x 1 * R = x 2 y 1 * R = y 2 z 1 * R = z 2 (0,1,0) * R = R 11 : projection of y1 on y2 R 11 > cos(45°) = to 5 R’s Sort by R 00 +R 11 +R 22 Return the R with the largest diagonal sum Building -z 1 y1y1 x1x1 -z 2 x2x2 y2y2 Scanner position 1Scanner position 2

14 y1y1 x1x1 z1z1 Translation estimation Left and right axes parallel accordingly after rotation Pick robust line pairs to estimate translation y1y1 x1x1 z1z1 Translation Estimation R y2y2 x2x2 z2z2

15 Translation Estimation One pair of matched lines provides an estimated translation Two pairs with similar estimated translations provide translation candidate y1y1 x1x1 z1z1 y2y2 x2x2 z2z2 Line pair 1 Line pair 2 T

16 Translation Estimation Two types of translation candidates T = (d 1 + d 2 ) / 2 d1d1 d2d2 T  linear system y1y1 x1x1 z1z1 y2y2 x2x2 z2z2

17 Translation Estimation Find the translation that maximizes the total number of line matches Cluster all estimated T’s, pick 10 most frequently appeared T’s For each T: Find all matches, solve linear system to update R&T Count matched line pairs again Choose the R&T with the most number of matched line pairs Refine transformation with ICP

18 Need manual adjustment Display draggers Adjust R/T, optimize save Rotation wrong by 90° Display other rotations Choose a new R, compute T correct Registration System Flowchart start Read in all image pairs, form the transformation graph Read in one image pair Last pair? Automated registration Display to user save Find pivot image, compute path Compute transform from each image to pivot image Global optimization Global stitching Display and saveexit Y N 123 save correct Display other rotations Choose a new R, compute T Rotation wrong by 90° Display draggers Adjust R/T, optimize save Need manual adjustment

19 User Interface Display window: Points and lines of registered two scans Correct Rotation & Translation Rotation Symmetry by 90deg. Need Manual Adjustment on R/T

20 Need manual adjustment Display draggers Adjust R/T, optimize save 3 correct save Rotation wrong by 90° User Interface start Read in all image pairs, form the transformation graph Read in one image pair Last pair? Automated registration Display to user Display other rotations Choose a new R, compute T Find pivot image, compute path Compute transform from each image to pivot image Global optimization Global stitching Display and saveexit Y N 12 Display other rotations Choose a new R, compute T Rotation wrong by 90°

21 Rotation wrong by 90 degrees: choose from other candidate rotations computed previously User Interface

22 correct Rotation wrong by 90° saveDisplay other rotations Choose a new R, compute T 12 Need manual adjustment Display draggers Adjust R/T, optimize save User Interface start Read in all image pairs, form the transformation graph Read in one image pair Last pair? Automated registration Display to user Find pivot image, compute path Compute transform from each image to pivot image Global optimization Global stitching Display and saveexit Y N 3 Display draggers Adjust R/T, optimize save Need manual adjustment

23 Adjusting rotation and translation based on the building’s coordinate system White draggers for translation Blue spheres for rotation User Interface

24 User Interface Display window: Points and lines of registered two scans Correct Rotation & Translation Rotation Symmetry by 90deg. Need Manual Adjustment on R/T Exhaustive Search Approach ICP Optimization

25 correct Rotation wrong by 90° Need manual adjustment Display draggers Adjust R/T, optimize save Global Stitching start Read in all image pairs, form the transformation graph Read in one image pair Last pair? Automated registration Display to user saveDisplay other rotations Choose a new R, compute T Find pivot image, compute path Compute transform from each image to pivot image Global optimization Global stitching Display and saveexit Y N 123

26 Global Stitching Global stitching for all images Pivot image – the image with the most number of neighbors Transform composition – along the strongest path from each image to the pivot image Further improvement to consider Global optimization to minimize registration error

27 Thomas Hunter building, Hunter College 14 scans, 15 pairs (13 automated, 2 manually adjusted) 10~20 seconds per pair; a few minutes for entire registration Experimental Results

28 Registration Results Shepard Hall, City College 20 scans, 24 pairs (9 automated, 8 R symmetry, 7 adjust R/T) 20~90 seconds per pair; 1 hour for entire registration v ideo

29 Registration Results Great Hall (interior of Shepard Hall) 21 scans, 44 pairs (12 automated, 18 R symmetry, 13 adjust R/T ) 20~90 seconds per pair; 1.5 hour for entire registration

30 Registration Results Great Hall interior scene

31 Registration Results Great Hall interior scene

32 Registration Results Great Hall interior scene

33 Algorithm Performance Lines in two scans Time for automated registration Number of matching lines Average error of matching planes

34 Algorithm Performance Thomas Hunter building (rectangular) Shepard Hall (intricate) Great Hall (much symmetry) Number of scans Number of pairs Automated / Rotation only / Manual 13 / 2 / 09 / 8 / 712 / 18 / 13 Approximate number of lines per scan Average time to register a pair 10-20s20-90s Approximate time for entire procedure a few minutes1 hour1.5 hour Average distance between matching planes Before ICP Optimization 21.77mm51.72mm17.59mm After ICP Optimization 1.77mm3.23mm7.26mm

35 Performance Comparison Twice as fast as previous exhaustive searching approach Suitable for structures with many lines, new approach more suitable for cubic-like shapes; not as general as previous approach Accuracy Similar to previous method before applying ICP algorithm Greatly improved after applying ICP algorithm

36 Semi-automatic registration system Automated 3D registration routines Context-sensitive user interface Fast computation, accurate registration Future work Global optimization Extract higher ordered curvatures from range data for faster and more accurate feature-based registration Conclusions

Spin Image Representation Histogram of surface points about a rotation around surface normal at the sample Point at varying radii from the sample point A method of measuring shape and curvature with local support

38 Acknowledgement NSF CAREER IIS NSF MRI/RUI EIA Conference committee and all audiences Contact us: Ioannis Stamos, Cecilia Chao Chen,