1. ICPR/WDIA-2012
High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image
Jui-Chiu Chiang, Zheng-Feng Liu, and Wen-Nung Lie
National Chung Cheng University
Taiwan
2012/11/11

2. Outline Introduction Our hybrid camera system
The proposed depth processing algorithm
Experimental results
Conclusion
簡介視訊編碼技術的趨勢
2012/11/11

3. Color + Depth in 3DTV/FTV
Depth image, in combination with DIBR (Depth Image-Based Rendering) technique, is important in multi-view 3DTV/FTV applications.
Color image
Depth image
DIBR
view9
view8
view7
view6
view5
view4
view3
view2
view1
2012/11/11

4. Depth acquisition From stereo matching via disparity estimation
Unreliable for textureless regions
Computationally intensive
From active photo-electrical sensing
Lower resolution
Higher acquisition speed and accuracy
Kinect
SR4000
ToF
2012/11/11

5. Hybrid camera systems
High-resolution color cameras + Low-resolution depth camera (mis-alignment between their optical axis)
[12]: Use warped data of depth camera to refine ROI disparity of the dual-eye camera
Proposed: one color image is used to refine the warped (to both sides) and up-sampled depth images
2012/11/11

6. Processing diagram of our system
2012/11/11

7. Generation of preliminary high-resolution depth image
Purpose: to generate high-resolution depth image at the left and right sides
Calibrated camera parameters
2012/11/11

8. Pre-processing of the depth map
33 median filter
Low resolution to high resolution depth warping
Forward warping with a block size of 55 pixels
Post-processing by morphological operation
Morphological opening
2012/11/11

9. Generation of high quality high-resolution depth image
Color-image-guided depth map refinement
2012/11/11

10. Color segmentation by mean-shift algorithm
Binarizing the warped depth image to have foreground/background classification
Color segmentation by mean-shift algorithm
Label each segment as FG or BG
Refine the corresponding depth values according to the segment categorization
Bilateral filtering is performed for the edge parts of the modified depth image
2012/11/11

11. Experiments – system configurations
Color image: 640x480
Depth image: 176x144 (SR4000)
Intel Core 2 Duo Q GHz, and DDR GB
2012/11/11

12. Preliminary high-resolution depth image
After depth warping
Traditional depth warping
After morphological operation
2012/11/11

13. High-quality high-resolution depth image
After correction
Mean-shift
Incorrect pixels
Bilateral filtering
2012/11/11

14. A series of generated depth maps
2012/11/11

15. Virtual view synthesis based on various depth maps
Hole filled
2012/11/11
Bilateral filtering
Bilateral filtering on edges
Bilateral filtering on edges + Gaussian

16. Speed performance Procedure Time (sec) Pre-processing Depth Warping
0.117
Color segmentation
0.742
Post-processing
Bilateral filtering on edges
0.352
Gaussian filtering
0.104
Novel view synthesis by DIBR (including hole filling)
0.144
2012/11/11

17. Conclusion
A hybrid camera system consisting of high-resolution color camera and low-resolution depth camera is proposed
A high-quality high-resolution depth images can be obtained for the left and right eyes – multi-view video plus depth (N=2 channels).
Future work
Combine Kinect device and stereo camera to get better depth image and cheaper cost
2012/11/11