1. PG 2011 Pacific Graphics 2011 The 19th Pacific Conference on Computer Graphics and Applications (Pacific Graphics 2011) will be held on September 21 to 23, 2011 in Kaohsiung, Taiwan. PG 2011 Pacific Graphics 2011 The 19th Pacific Conference on Computer Graphics and Applications (Pacific Graphics 2011) will be held on September 21 to 23, 2011 in Kaohsiung, Taiwan. PG 2011 Pacific Graphics 2011 The 19th Pacific Conference on Computer Graphics and Applications (Pacific Graphics 2011) will be held on September 21 to 23, 2011 in Kaohsiung, Taiwan. Exposure Fusion for Time-Of-Flight Imaging Uwe Hahne, Marc Alexa

2. Depth imaging [Shotton2011] [Audi] Applications

3. [PMDTec] [Microsoft] [PTGrey] Devices Depth imaging

4. General problem Video #1

5. Our results Video #2

6. Contribution + No calibration + Real-time capable + Error reduction - Only for time-of-flight imaging...

7. [PMDTec]

8. P opt t

9. [PMDTec] P opt t 50 ns (ω = 20 MHz) Phase shift  distance Amplitude  intensity

10. [PMDTec] Integration time P opt t between 50 and 3000 µs

11. Problem Noisy background Noisy foreground Short integration time (50 µs)Long integration time (1250 µs)

12. Sampling S0S0 S1S1 S0S0 S2S2 S3S3 P opt t

13. Sampling S0S0 S1S1 S0S0 S2S2 S3S3 P opt t Short integration time  low SNR

14. Sampling S0S0 S1S1 S0S0 S2S2 S3S3 P opt t Long integration time  saturation

15. What is the optimal integration time?

16. 1. Adaptation (May2006)  Updating the integration time based on data from previous frames.  Only a global solution Existing approaches [Lindner2007] 2. Calibration (Lindner, Schiller, Kolb,...)  Capture samples of known geometry and correct the measurements.  Laborious and device specific Video #3

17. [Lindner2007] 1. Adaptation (May2006)  Updating the integration time based on data from previous frames.  Only a global solution 2. Calibration (Lindner, Schiller, Kolb,...)  Capture samples of known geometry and correct the measurements.  Laborious and device specific Existing approaches

18. Our approach How is this problem solved in photography?  High Dynamic Range (HDR) Imaging

19. HDR imaging Over-exposed Details Under-exposed Details Missing details [Images are courtesy of Jacques Joffre] Radiance map 10101110100101011010101 01010100100101010010010 11000010100101000101001 10101110100101011010101 01010100100101010111010 01010110101010101010010

20. Over-exposed Under-exposed Depth imaging

21. Algorithm 1.Capture a series of depth images with varying integration times. 2.Compute weights using quality measures. 3.Fuse images together as affine combination of weighted depth maps.

22. Depth maps Process overview Laplace pyramidsWeight maps Fused pyramidResulting fusion

23. Depth maps Process overview Laplace pyramidsWeight maps Fused pyramidResulting fusion

24. Quality measures ContrastWell exposednessEntropySurfaceWeight maps

25. Quality measures Well exposedness Emphasizes those pixels where the amplitude is in a “well” range and hence removes over- and underexposure. [Mertens2007]

26. Quality measures Contrast Good contrast in the amplitude image indicates absence of flying pixels. [Mertens2007]

27. Quality measures Entropy The entropy is a measure for the amount of information. [Goshtasby2005]

28. Quality measures Surface [Malpica2009] A measure for surface smoothness which indicates less noise. σ = Gaussian blurred squared depth map μ = Gaussian blurred depth map

29. Quality measures ContrastWell exposednessEntropySurface Weight maps

30. Depth maps Process overview Laplace pyramidsWeight maps Fused pyramidResulting fusion

31. Evaluation Impact of each quality measure -Computation time

32. Evaluation Depth map quality What is the reference?

33. Reference We compared our results to single exposures with the integration time t’. The integration time t’ is the global optimum found by the method from [May2006]. We use N = 4 exposures (i = 0..N-1) with integration times.

34. Evaluation Depth map quality What is the reference? What does quality mean? -Less temporal noise  Stability over time test

35. Capture a static image over time and determine the standard deviation for each pixel. Overall reduction of mean standard deviation by 25%. Stability over time Comparison F > S F < S F ≈ S mm Single exposure (S)Fused result (F)

36. Evaluation Depth map quality What is the reference? What does quality mean? -Less temporal noise  Stability over time test -Better processing results  Run ICP and compute 3D reconstruction error

37. 3D reconstruction

38. Evaluation Depth map quality  What does quality mean?  Less temporal noise  Stability over time test  Better processing results  Run ICP and compute 3D reconstruction error  Impact of each quality measure  Computation times

39. Computation

40. Evaluation Depth map quality  What does quality mean?  Less temporal noise  Stability over time test  Better processing results  Run ICP and compute 3D reconstruction error  Impact of each quality measure  Computation times  Precision  Plane fit in planar regions

41. We captured test scenes with planar regions. Planar regions

42. We ran our fusion algorithm... Planar regions Depth mapsLaplace pyramidsWeight maps Fused pyramidResulting fusion

43. Planar regions Contrast Well expo.EntropySurface Weight maps...and compared the fusion results of all combinations of quality measures.

44. Planar regions We identified planar regions...

45. Planar regions...and fitted planes into the 3D data.

46. Interesting results Planar regions Only contrast does work alone Entropy is time consuming

47. Planar regions

48. 1 cm Planar regions Surface Only contrast does work alone

49. Planar regions Combinations are valuable Surface

50. Planar regions All four quality measures lead to the smallest error Surface

51. Conclusion A new method for time-of-flight imaging: + No calibration + Real-time capable + Reduced error

52. Kaohsiung, Taiwan Pacific Graphics 2011. 52 | Kaohsiung, Taiwan Pacific Graphics 2011. 52 | Kaohsiung, Taiwan Pacific Graphics 2011. 52 | Thank you Acknowledgements -Martin Profittlich (PMDTec) for TOF camera support. -Bernd Bickel and Tim Weyrich for helpful discussions.

53. PG 2011 Pacific Graphics 2011 The 19th Pacific Conference on Computer Graphics and Applications (Pacific Graphics 2011) will be held on September 21 to 23, 2011 in Kaohsiung, Taiwan. PG 2011 Pacific Graphics 2011 The 19th Pacific Conference on Computer Graphics and Applications (Pacific Graphics 2011) will be held on September 21 to 23, 2011 in Kaohsiung, Taiwan. PG 2011 Pacific Graphics 2011 The 19th Pacific Conference on Computer Graphics and Applications (Pacific Graphics 2011) will be held on September 21 to 23, 2011 in Kaohsiung, Taiwan. Exposure Fusion for Time-Of-Flight Imaging Uwe Hahne, Marc Alexa

54. References Shotton, Jamie, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. “Real-Time Human Pose Recognition in Parts from Single Depth Images.” In CVPR, (to appear), 2011. Büttgen, Bernhard, Thierry Oggier, Michael Lehmann, Rolf Kaufmann, and Felix Lustenberger. CCD / CMOS Lock-In Pixel for Range Imaging : Challenges, Limitations and State-of-the-Art. Measurement, 2005. Lindner, Marvin, and Andreas Kolb. “Calibration of the intensity-related distance error of the PMD TOF-camera.” Proceedings of SPIE (2007): 67640W-67640W-8. Goshtasby, A. “Fusion of multi-exposure images” Image and Vision Computing (2005),23 (6),p. 611-618 Mertens, T., Kautz, J., Van Reeth, F. “Exposure Fusion” 15th Pacific Conference on Computer Graphics and Applications (PG'07), p. 382-390 Stefan May, Bjorn Werner, Hartmut Surmann, Kai Pervolz “3D time-of-flight cameras for mobile robotics”, IEEE/RSJ International Conference on Intelligent Robots and Systems 2006, p. 790-795