Range Image Segmentation for Modeling and Object Detection in Urban Scenes Cecilia Chen & Ioannis Stamos Computer Science Department Graduate Center, Hunter College The City University of New York

Motivations Extract geometrically meaningful segments Model large-scale urban scenes Provide high-level scene understanding

Step 1 – Range Segmentation Range Scan Planar/Smooth Regions Non-smooth Regions +

Step 2 – Segment Merging

Planar/Smooth Regions Range Segmentation Range Scans Planar/Smooth Regions Non-smooth Regions +

Range Segmentation Details All segments

Range Segmentation Normal computation Planar/Smooth region extraction Identify all locally-planar points Compute surface normal for all points Planar/Smooth region extraction Region growing based on local plane continuity Plane fitting to differentiate two region types Non-smooth region extraction Region growing based on spatial continuity

Range Segmentation Normal computation Planar points (a)(b)(c) Local plane fitting Fitting error < θfit Planar edge points (d)(e) Find planar neighbors Select a best local plane from those of neighbors Non-planar points (f) P (d) N2 F S1 S2 d2 N1 P (a) S F d1 (e) F S2 S1 P (b) S F P (c) S F (f) F S

Range Segmentation Planar/Smooth region growing N (a neighbor of P) belongs to P’s region if Distance (N, P’s local plane) < θpoint2plane Intensity differs by <θintensity Angle (P’s local plane, N’s local plane) < θangle Non-smooth region growing N (a neighbor of P) belongs to P’s region if Distance (N, P) < θpoint2point Intensity differs by <θintensity Angle (P’s local plane, N’s local plane) < θ’angle

Range Segmentation Cooper Union results Execution time < 2 minutes Quality is shown in model generation results

Range Segmentation Planar/smooth and non-smooth regions

Range Segmentation Previous approaches Our algorithm Only planar and smooth regions Inaccurate region boundaries Non-smooth regions are discard or approximated with smaller planar segments Our algorithm Planar, smooth, non-smooth regions Accurate region boundaries Non-smooth regions are grown based on spatial closeness, preserving original surface shapes

Segment Merging and Modeling …… Segmented Scans Segment Merging Range Registration Merged Segments Surface mesh per segment

Merging Segments Same region Identify overlapping regions Merge overlapping regions

Merging Segments Algorithm design Advantages Only consider segments at area of overlap Utilize grid structure and generate z-buffers Use each range image as pivot image Transform and project other range images Identify and merge overlapping segment sets Advantages Time efficient, due to grid utilization Memory efficient, as only overlapping scans need to be in memory Ability to be parallelized

Merging Segments (2 scans) Merging two range scans I1 and I2 I1 has n1 segments; I2 has n2 segments. I2 I1 (Segment label in transformed I2, depth, surface normal) (Segment label in I1, depth, surface normal) Z21 Z11

Merging Segments (2 scans) Two scans merged

Merging Segments (all scans) Merging range scans I1, I2, …, In Each scan (Ii) as pivot; other scans tranformed Ik Zki Ij Ii (Segment label in transformed Ij, depth, surface normal) (Segment label in Ii, depth, surface normal) Zji Zii

Experimental Results Cooper Union building, NYC 8 scans merged in 4 minutes Average surface fitting error of planar segments is 3mm 1760 planar region 382 non-planar regions

Experimental Results Details of merged segments Merged points Surface meshes

Experimental Results Grand Central Station, NYC 15 range scans merged in 10 minutes Average surface fitting error of planar segments is 4mm 1393 planar regions 787 non-planar regions

Experimental Results Details of merged segments Merged points Surface meshes

Experimental Results Surface meshes per segment

Conclusions and Future Work Achievements Extracting geometrically meaningful segments Providing Accurate region boundaries Merging segments for modeling Future Work Analysis of segment spatial relationship for high level scene understanding Feature extraction from each segment for object recognition

Thanks ! istamos@hunter.cuny.edu http://www.cs.hunter.cuny.edu/~ioannis/Vision.htm