The 18th Meeting on Image Recognition and Understanding 2015/7/29 Depth Image Enhancement Using Local Tangent Plane Approximations Kiyoshi MatsuoYoshimitsu Aoki Hokuyo Automatic Co., Ltd.Keio University ○

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO Background –Research purpose –Related work Proposed method (overview and highlight) –Local coordinates based depth image enhancement Experimental results Conclusion 2 Contents

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO Depth image enhancement for consumer RGB-D cameras 3 Research purpose Ex. Soft Kinetic DS311 > Designed for human interaction applications * Consumer RGB-D camera + low cost + portable - low accuracy - low resolution Enhancement of both the accuracy and resolution could help many applications. Ex. fine-grained object recognition, precise indoor navigation, and etc…

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO 4 Related work “An application of Markov random fields to range sensing” [ NIPS 2006 Diebel et al. ] “High quality depth map upsampling for 3d-tof cameras” [ ICCV 2011 Park et al. ] “A Joint intensity and depth co-sparse analysis model for depth map surperresolution” [ ICCV 2013 Kiechle et al. ] “Depth map inpainting under a second-order smoothness prior” [ Image Analysis vol Herrera et al. ] etc. 1. Measured depth data is used as optimization data prior. 2. Various pixel based information is used as optimization smoothness prior. (gradient, segmentation, edge saliency, non-local mean, and co-sparseness etc.) 3. Depth enhancement is achieved by an optimization of over the image plane. [ Diebel et al ] Global optimization based enhancement Markov Random Field “Joint bilateral upsampling” [ ACM transactions on Graphics 2007 Kopf et al. ] “Spatial-depth super resolution for range images” [ CVPR 2007 Yang et al. ] “Pixel weighted average strategy for depth sensor data fusion” [ ICIP 2010 Garcia et al. ] “Upsampling range data in dynamic environments” [ CVPR 2010 Dolson et al. ] “Joint geodesic upsampling of depth images” [ CVPR 2013 Liu et al. ] etc. Local filter based enhancement 1. Local measured depth data is summarized by using similarity weights. 2. Similarity weights are defined on the global image plane coordinates. (pixel-distance, color-difference, gradient, and color geodesic distance, etc.) 3. Depth enhancement is achieved by local calculations on the image plane. Joint bilateral upsampling filter [ Kopf et al ]

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO 5 Related work Previous enhancement methods use pixel-coordinates. These methods smooth the surfaces to be parallel to the image plane. Measuring object Image plane No geometric relationship The geometries of surfaces are not take into account, (and are sometimes corrupted by them)

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO 6 Contributions To improve accuracy, we introduce local tangent planes as local coordinates to handle the geometries. Local coordinatesGlobal coordinates * pixel-coordinates of the image* charts defined on each local tangent * depend only on the measuring device (independent of the measuring surfaces) * depend on the local geometries of measuring surfaces Accurate depth image enhancement is achieved. (especially in noisy cases)

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO 7 Proposed method (overview) A pair of depth and color images * Input data 2. Surface reconstruction * Output data A high-resolution accurate depth image Only overview of our method (more details will be shown at our poster.) 1. Estimation of local tangent planes Proposed in ICCVW 2013

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO 8 How to use local coordinates ? Measured surface Height smoothed surface To recover smooth surfaces, we smooth normal components of surfaces on each local coordinates Colored by the height from the tangent plane Smoothed surface These filters on local coordinates smooth the surfaces while preserving the local geometries of local tangent planes.

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO The interior regions of surfaces were reconstructed more accurately. 9 Experimental results MRF [Diebel and Thrun] PWAS [Garcia et al] Previous ours [Matsuo and Aoki] Our method Source RGB-D image Captured by SoftKinetic DS311

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Unique Product & Best Sensing HOKUYO 10 Conclusion Local coordinate based depth image enhancement. –Local tangent planes are used as local coordinates. Normal component smoothing reduces granular noises. –These filters preserve the local geometries of local tangents. The effectiveness is shown by real sensor data.

The 18th Meeting on Image Recognition and Understanding 2015/7/29 Thank you for this opportunity to present to you today. And thank you for your attention ! If you have any questions and comments, visit our poster IS 2-4.