Real-Time 3D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Ilya Korsunsky Princeton University Stanford University Hunter College

3D Scanning

Possible Research Goals Low noiseLow noise Guaranteed high accuracyGuaranteed high accuracy High speedHigh speed Low costLow cost Automatic operationAutomatic operation No holesNo holes

3D Model Acquisition Pipeline 3D Scanner

3D Model Acquisition Pipeline 3D Scanner View Planning

3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment View Planning

3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMerging View Planning

3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning

3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay

3D Model Acquisition Difficulties Much (often most) time spent on “last 20%”Much (often most) time spent on “last 20%” – Merging, Displaying, View Planning Pipeline not optimized for hole-fillingPipeline not optimized for hole-filling Not sufficient just to speed up scanner – must design pipeline for fast feedbackNot sufficient just to speed up scanner – must design pipeline for fast feedback

Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Human

Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMerging Done?Done? View Planning DisplayDisplay Challenge: Real Time

Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part I: Structured-Light Triangulation

Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part II: Fast ICP

Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part III: Voxel Grid

Triangulation Project laser stripe onto objectProject laser stripe onto object Object Laser CameraCamera

CameraCamera Triangulation Depth from ray-plane triangulationDepth from ray-plane triangulation Laser (x,y) Object

Triangulation Faster acquisition: project multiple stripesFaster acquisition: project multiple stripes Correspondence problem: which stripe is which?Correspondence problem: which stripe is which?

Continuum of Triangulation Methods Slow, robust Fast, fragile Multi-stripeMulti-frame Single-frame Single-stripe

Time-Coded Light Patterns Assign each stripe a unique illumination code over time [Posdamer 82]Assign each stripe a unique illumination code over time [Posdamer 82] Space Time

Illumination history = (WB),(BW),(WB) CodeCode Codes for Moving Scenes Assign time codes to stripe boundariesAssign time codes to stripe boundaries Perform frame-to-frame tracking of corresponding boundariesPerform frame-to-frame tracking of corresponding boundaries – Propagate illumination history [Hall-Holt & Rusinkiewicz, ICCV 2001] [Hall-Holt & Rusinkiewicz, ICCV 2001]

Designing a Code Want many “features” to track: lots of black/white edges at each frameWant many “features” to track: lots of black/white edges at each frame Try to minimize ghosts – WW or BB “boundaries” that can’t be seen directlyTry to minimize ghosts – WW or BB “boundaries” that can’t be seen directly

Designing a Code [Hall-Holt & Rusinkiewicz, ICCV 2001]

Implementation Pipeline:Pipeline: DLP projector illuminates 60 Hz.DLP projector illuminates 60 Hz. Synchronized NTSC camera captures videoSynchronized NTSC camera captures video Pipeline returns range 60 Hz.Pipeline returns range 60 Hz. ProjectCodeProjectCodeCaptureImagesCaptureImagesFindBoundariesFindBoundariesMatchBoundariesMatchBoundariesDecodeDecodeComputeRangeComputeRange

Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part II: Fast ICP

Aligning 3D Data This range scanner can be used for any moving objectsThis range scanner can be used for any moving objects For rigid objects, range images can be aligned to each other as object movesFor rigid objects, range images can be aligned to each other as object moves

ICP in the Real-Time Pipeline Potential problem with ICP: local minimaPotential problem with ICP: local minima – In this pipeline, scans close together – Very likely to converge to correct (global) minimum Basic ICP algorithm too slow (~ seconds)Basic ICP algorithm too slow (~ seconds) – Point-to-plane minimization – Projection-based matching – With these tweaks, running time ~ milliseconds [Rusinkiewicz & Levoy, 3DIM 2001]

Anchor Scans If object moves too quickly or is out of range, ICP will failIf object moves too quickly or is out of range, ICP will fail Global driftGlobal drift Instead of aligning to last scan, keep set of scans to realign toInstead of aligning to last scan, keep set of scans to realign to Large: have enough dataLarge: have enough data Distinct: don’t store too manyDistinct: don’t store too many Allows user to restart scanningAllows user to restart scanning

Recapturing Alignment

Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part III: Voxel Grid

Merging and Rendering Goal: visualize the model well enough to be able to see holesGoal: visualize the model well enough to be able to see holes Cannot display all the scanned data – accumulates linearly with timeCannot display all the scanned data – accumulates linearly with time Standard high-quality merging methods: processing time ~ 1 minute per scanStandard high-quality merging methods: processing time ~ 1 minute per scan Use fast voxel based merging techniqueUse fast voxel based merging technique

Merging and Rendering

+

Point rendering, using accumulated normals for lightingPoint rendering, using accumulated normals for lighting

Problems with Merging Misaligned Scans (error in ICP)Misaligned Scans (error in ICP) – Moderate misalignment Voxels rendered on top of one anotherVoxels rendered on top of one another Noisy but OKNoisy but OK

Problems with Merging Misaligned Scans (error in ICP)Misaligned Scans (error in ICP) – Outliers Give wrong modelGive wrong model Eliminate thin and long triangles and associated pointsEliminate thin and long triangles and associated points Can be aggressive eliminating b/c of large amt dataCan be aggressive eliminating b/c of large amt data Also rejects low quality dataAlso rejects low quality data

Example: Photograph 18 cm.

Result

Postprocessing Real-time displayReal-time display – Quality/speed tradeoff – Goal: let user evaluate coverage, fill holes Offline postprocessing for high-quality modelsOffline postprocessing for high-quality models – Global registration – High-quality merging (e.g., using VRIP [Curless 96])

Example: Photograph 18 cm.

Postprocessed Model

Summary 3D model acquisition pipeline optimized for obtaining complete, hole-free models3D model acquisition pipeline optimized for obtaining complete, hole-free models Use human’s time most efficientlyUse human’s time most efficiently Pieces of pipeline selected for real-time use:Pieces of pipeline selected for real-time use: – Structured-light scanner for moving objects – Fast ICP variant – Simple grid-based merging, point rendering

Limitations Prototype noisier than commercial systemsPrototype noisier than commercial systems – Could be made equivalent with careful engineering – Ultimate limitations on quality: focus, texture Scan-to-scan ICP not perfect  alignment driftScan-to-scan ICP not perfect  alignment drift – Due to noise, miscalibration, degenerate geometry – Reduced, but not eliminated, by “anchor scans” – Possibly combine ICP with separate trackers