Design and Calibration of a Multi-View TOF Sensor Fusion System Young Min Kim, Derek Chan, Christian Theobalt, Sebastian Thrun Stanford University

Young Min Kim TOF-CV Workshop 2 Outline  Motivation  System Architecture  Depth Sensor Characteristics  System Calibration  Results and Conclusion

Young Min Kim TOF-CV Workshop 3 Outline  Motivation  System Architecture  Depth Sensor Characteristics  System Calibration  Results and Conclusion

Young Min Kim TOF-CV Workshop 4 Motivation  Goal: Reconstruct geometry and texture of entire scene from minimum sensor data  State-of-the-art [Waschbuesch et al. 2006, 2007] –Stereo [Laurentini et al. 94, Kanade et al. 97, Matusik et al. 00, Matsuyama et al. 02, Wuermlin 02, Carranza et al. 03, Cheung et al. 03, Zitnick et al. 04, Bajcsy et al. 04, …]

Young Min Kim TOF-CV Workshop 5 Motivation  Limitation to stereo –Correspondence problem –Dependency on texture –Densely spaced cameras  Our idea: Build a system that can combine –TOF sensor –Video cameras

Young Min Kim TOF-CV Workshop 6 Outline  Introduction  System Architecture  Depth Sensor Characteristics  System Calibration  Results and Conclusion

Young Min Kim TOF-CV Workshop 7 Multi-view Sensor Fusion System

Young Min Kim TOF-CV Workshop 8 Multi-view Sensor Fusion System Point Grey Flea x768 pixels 30 Hz High resolution No depth

Young Min Kim TOF-CV Workshop 9 Multi-view Sensor Fusion System Swissranger SR3000 Flash Ladar 3D geometry at 60 Hz Resolution: 176 x 144 No visual interference Noisy low-resolution data Point Grey Flea x768 pixels 30 Hz High resolution No depth

Young Min Kim TOF-CV Workshop 10 Multi-view Sensor Fusion System Swissranger SR3000 Flash Ladar 3D geometry at 60 Hz Resolution: 176 x 144 No visual interference Noisy low-resolution data Point Grey Flea x768 pixels 30 Hz High resolution No depth Main Contribution System Architecture And Calibration

Young Min Kim TOF-CV Workshop 11 System Architecture … 19 MHz 20 MHz 21 MHz FireWire B FireWire A – Synchronization bus

Young Min Kim TOF-CV Workshop 12 System Architecture … 19 MHz 20 MHz 21 MHz FireWire B FireWire A – Synchronization bus –Synchronization  Hardware synch for video cameras  Initiate software for Swissrangers –Modulation frequency  Scales up to 4 Swissrangers

Young Min Kim TOF-CV Workshop 13 Example Data

Young Min Kim TOF-CV Workshop 14 Outline  Introduction  System Architecture  Depth Sensor Characteristics  System Calibration  Results and Conclusion

Young Min Kim TOF-CV Workshop 15 Related Work: TOF-Camera  Random noise characteristics of earlier models  Systematic depth errors –PMD (photomic mixer device) –Lookup table for Swissranger  Detailed analysis on noise characteristics of single TOF sensors  Our work –practical model for systematic bias –calibration for multiple cameras [Anderson et al. 05, Herbert et al. 92, …] [Lindner et al. 06, Lindner et al. 07] [Kahlmann et al. 92] [Rapp 07]

Young Min Kim TOF-CV Workshop 16 Depth Sensor Characteristics  Independent for each pixel (u, v)  Measurement model d m (u, v)=d g (u, v)+d r (u, v)+d s (u, v) Measurement uncertainty MeasuredGround-truth d r (u,v): random noise d s (u,v): systematic bias (u, v) d m (u, v) Center of Projection Image plane

Young Min Kim TOF-CV Workshop 17 Systematic Bias  Two components of systematic bias –Distance misalignment + rigid, directional –Influence of orientation, reflectance, and amplitude

Young Min Kim TOF-CV Workshop 18 Systematic Bias d s (u, v) = f(r) d’ s (u, v) d s (u, v) ≈ d’ s (u, v) for r > 0.3 r=0.3  Influence of orientation, reflectance, and amplitude  ratio of normalized amplitude

Young Min Kim TOF-CV Workshop 19 Outline  Introduction  System Architecture  Depth Sensor Characteristics  System Calibration  Results and Conclusion

Young Min Kim TOF-CV Workshop 20 Calibration  Video Cameras: –Standard calibration toolboxes –Intrinsic parameters + extrinsic parameters from checkerboard  Depth Cameras –No off-the-shelf solution –Camera provides XYZ / intensity –Use procedure based on optical method  practicability

Young Min Kim TOF-CV Workshop 21 Depth Camera Calibration  Use calibration procedure for video cameras  Space 1: optical camera model – viewpoint/projection  Camera XYZ  Space 2: 3D point cloud in sensor coordinates  Space 1 and Space 2 don’t match -> compensate

Young Min Kim TOF-CV Workshop 22 Depth Camera – Compensate Systematic Bias  Compensation: deform Space 2 to match Space 1 Intrinsics … N checkerboard positions spanning view frustum  ground truth point clouds in space 1 and measured point clouds in space 2

Young Min Kim TOF-CV Workshop 23 Depth Camera – Compensate Systematic Bias  Compensation: deform Space 2 to match Space 1  Step 1: Rigid alignment Intrinsics K, A … N checkerboard positions spanning view frustum  ground truth point clouds in space 1 and measured point clouds in space 2 P R rigid,t rigid P1 Space 1 Space 2

Young Min Kim TOF-CV Workshop 24 Depth Camera – Compensate Systematic Bias  Step 2: Warp of ray direction Space 1 Space 2 interpolate Φ(i,j),Ω(i,j) P1 P2

Young Min Kim TOF-CV Workshop 25 Depth Camera – Compensate Systematic Bias  Step 2: Warp of ray direction  Step 3: Constant per-pixel length bias along ray Space 1 Space 2 interpolate Φ(i,j),Ω(i,j) D(i,j) P1 P2 P3

Young Min Kim TOF-CV Workshop 26 Outline  Introduction  System Architecture  Depth Sensor Characteristics  System Calibration  Results and Conclusion

Young Min Kim TOF-CV Workshop 27 Result

Young Min Kim TOF-CV Workshop 28 Result

Young Min Kim TOF-CV Workshop 29 Result Combination of two depth maps Projectively textured from three video cameras

Young Min Kim TOF-CV Workshop 30 Result Combination of three depth maps Before bias correctionAfter bias correction

Young Min Kim TOF-CV Workshop 31 Result  Mean error of 4.94 cm reduced into 1.36cm

Young Min Kim TOF-CV Workshop 32 Conclusion  Design of a multi-view TOF fusion recording system  Detailed analysis of depth measurement inaccuracy  Calibration of depth and video data into a common frame  Starting point for improved dynamic shape and texture reconstruction  Acknowledgement: Max Planck Center for Visual Computing and Communication

Thank you