1. High-quality Scanning using Time-Of-Flight Depth Superresolution 17nd March 2008 Sebastian Schuon schuon@cs.stanford.edu Prepared for: Final Presentation CS223B, Winter 2008

2. Project_26.ptt 1 Sebastian Schuon  schuon@cs.stanford.edu Problem Statement Time-of-Flight (TOF) Cameras Have a Low Resolution ZCam by 3DV Systems (Israel) Measurement principle: time of flight Depth recording: 320x240, 8bit ► Less noisy depth images with higher resolution desired Native resolution Superresolution (4x) Geometry rendering from superresolution

3. Project_26.ptt 2 Sebastian Schuon  schuon@cs.stanford.edu Approach Combine Several Images to Increase Resolution Use multiple, here N=15, recordings from different viewpoints by translating the camera Estimating the high resolution image resembles to an optimization problem Optimization is multi-objective: similarity and smoothness is enforced ►

4. Project_26.ptt 3 Sebastian Schuon  schuon@cs.stanford.edu Results Subtle Details Become Visible and Noise is Reduced Depth map 3D geometry rendering Native resolutionSuperresolution (4x)

5. Project_26.ptt 4 Sebastian Schuon  schuon@cs.stanford.edu Backup

6. Project_26.ptt 5 Sebastian Schuon  schuon@cs.stanford.edu Hidden Slide Distribution of Work Project solely undertaken by Sebastian Schuon (me) Supervision / Collaboration: Christian Theobalt James Davis (UCSC) Additional imagery for paper provided by Hylke Buisman

7. Project_26.ptt 6 Sebastian Schuon  schuon@cs.stanford.edu Superresolution Goals Enhance resolution Reduce noise Recording Resolution (320 x 240) (Contrast enhanced) Super Resolution (4x upsample)

8. Project_26.ptt 7 Sebastian Schuon  schuon@cs.stanford.edu Depth Camera Theory ZCam by 3DV Systems (Israel) RGB and Depth camera in one housing (“RGBD”) RGB: 640x320 @ 30fps, Depth: 320x240, 160x120 @ 30fps Measurement principle: time of flight Depth image: distance between camera and object (not Z-coordinate) Unprojection to 3D coordinates necessary Specification of tracking window required

9. Project_26.ptt 8 Sebastian Schuon  schuon@cs.stanford.edu Depth Camera Results Recording scene with different layers of depth Black, shiny surfaces tend to be problematic One needs to know where to record (tracking window) Unprojection of depth images leads to 3D representation

10. Project_26.ptt 9 Sebastian Schuon  schuon@cs.stanford.edu Depth Camera Noise Characteristic Outer regions tend to be a lot more noisy Noise can be approximated with Gaussian Hypothesis: Noise increases quadratic with distance Hypothesis : Noise is correlated with color of object recorded Variance Plot Pixel Distribution over Time at Center

11. Project_26.ptt 10 Sebastian Schuon  schuon@cs.stanford.edu Depth Camera Systematic Bias RBG Processing disabled Variance Plot (RGB disabled) RBG Processing enabled Variance Plot (RGB enabled)

12. Project_26.ptt 11 Sebastian Schuon  schuon@cs.stanford.edu Superresolution Theory Based on Shift-And-Add family of algorithms Quite well studied for grayscale and color images, overview in [Farsiu04] We used Bilateral Shift-And-Add [Farsiu03] Formulation as inverse problem Our approach: rotating camera

13. Project_26.ptt 12 Sebastian Schuon  schuon@cs.stanford.edu Superresolution Results Simple Approach Depth Image - Superresolution (Contrast enhanced) Depth Image - Recording Resolution (Contrast enhanced) 3D Rendering - Recording Resolution 3D Rendering - Superresolution

14. Project_26.ptt 13 Sebastian Schuon  schuon@cs.stanford.edu Conclusion Findings Camera Interface and Software still beta / undocumented Interesting effects can happen, that are not expected Superresolution on depth images is feasible Further steps Reimplementation of known algorithms Ideas for improvement: – Depth camera specific noise characteristic – Incorporating confidence map

15. Project_26.ptt 14 Sebastian Schuon  schuon@cs.stanford.edu Depth Camera Systematic Bias

16. Project_26.ptt 15 Sebastian Schuon  schuon@cs.stanford.edu Superresolution Improved Approach