High-quality Scanning using Time-Of-Flight Depth Superresolution 17nd March 2008 Sebastian Schuon Prepared for: Final Presentation CS223B, Winter 2008

Project_26.ptt 1 Sebastian Schuon  Problem Statement Time-of-Flight (TOF) Cameras Have a Low Resolution ZCam by 3DV Systems (Israel) Measurement principle: time of flight Depth recording: 320x240, 8bit ► Less noisy depth images with higher resolution desired Native resolution Superresolution (4x) Geometry rendering from superresolution

Project_26.ptt 2 Sebastian Schuon  Approach Combine Several Images to Increase Resolution Use multiple, here N=15, recordings from different viewpoints by translating the camera Estimating the high resolution image resembles to an optimization problem Optimization is multi-objective: similarity and smoothness is enforced ►

Project_26.ptt 3 Sebastian Schuon  Results Subtle Details Become Visible and Noise is Reduced Depth map 3D geometry rendering Native resolutionSuperresolution (4x)

Project_26.ptt 4 Sebastian Schuon  Backup

Project_26.ptt 5 Sebastian Schuon  Hidden Slide Distribution of Work Project solely undertaken by Sebastian Schuon (me) Supervision / Collaboration: Christian Theobalt James Davis (UCSC) Additional imagery for paper provided by Hylke Buisman

Project_26.ptt 6 Sebastian Schuon  Superresolution Goals Enhance resolution Reduce noise Recording Resolution (320 x 240) (Contrast enhanced) Super Resolution (4x upsample)

Project_26.ptt 7 Sebastian Schuon  Depth Camera Theory ZCam by 3DV Systems (Israel) RGB and Depth camera in one housing (“RGBD”) RGB: 30fps, Depth: 320x240, 30fps Measurement principle: time of flight Depth image: distance between camera and object (not Z-coordinate) Unprojection to 3D coordinates necessary Specification of tracking window required

Project_26.ptt 8 Sebastian Schuon  Depth Camera Results Recording scene with different layers of depth Black, shiny surfaces tend to be problematic One needs to know where to record (tracking window) Unprojection of depth images leads to 3D representation

Project_26.ptt 9 Sebastian Schuon  Depth Camera Noise Characteristic Outer regions tend to be a lot more noisy Noise can be approximated with Gaussian Hypothesis: Noise increases quadratic with distance Hypothesis : Noise is correlated with color of object recorded Variance Plot Pixel Distribution over Time at Center

Project_26.ptt 10 Sebastian Schuon  Depth Camera Systematic Bias RBG Processing disabled Variance Plot (RGB disabled) RBG Processing enabled Variance Plot (RGB enabled)

Project_26.ptt 11 Sebastian Schuon  Superresolution Theory Based on Shift-And-Add family of algorithms Quite well studied for grayscale and color images, overview in [Farsiu04] We used Bilateral Shift-And-Add [Farsiu03] Formulation as inverse problem Our approach: rotating camera

Project_26.ptt 12 Sebastian Schuon  Superresolution Results Simple Approach Depth Image - Superresolution (Contrast enhanced) Depth Image - Recording Resolution (Contrast enhanced) 3D Rendering - Recording Resolution 3D Rendering - Superresolution

Project_26.ptt 13 Sebastian Schuon  Conclusion Findings Camera Interface and Software still beta / undocumented Interesting effects can happen, that are not expected Superresolution on depth images is feasible Further steps Reimplementation of known algorithms Ideas for improvement: – Depth camera specific noise characteristic – Incorporating confidence map

Project_26.ptt 14 Sebastian Schuon  Depth Camera Systematic Bias

Project_26.ptt 15 Sebastian Schuon  Superresolution Improved Approach