Joint Depth Map and Color Consistency Estimation for Stereo Images with Different Illuminations and Cameras Yong Seok Heo, Kyoung Mu Lee and Sang Uk Lee IEEE Transactions on Pattern Analysis and Machine Intelligence 2012

Overview Introduction Relate Work Algorithm Experimental Results

Introduction (1/3) image color values can be easily affected by radiometric variations including global intensity changes and local intensity changes (caused by varying light, vignetting and non-Lambertian surface) and noise. For stereo matching, most algorithms assume radiometrically calibrated images However, there exist many real and practical situations or challenging applications in which radiometric variations between stereo images are inevitable.

Introduction (2/3) For examples: 3D reconstruction of aerial images [1], general multiview stereo [4], 3D modeling with internet photos (e.g. Photo Tourism [5] and Photosynth [6] ), and PhotoModeler [7], etc.

In general, color consistency and stereo matching are a chicken-and-egg problem. Color consistency can enhance the performance of stereo matching, while accurate disparity maps can improve the color consistency or constancy. In this paper, new iterative framework that infers both accurate disparity maps and colorconsistent images for radiometrically varying stereo images are proposed. Introduction (3/3)

Relate Work Stereo Matching: – Census transform (7 × 7) [41] – Mutual Information (MI) [22] – Adaptive Normalized Cross Correlation (ANCC) [15] Color Consistency: – Color Histogram Equalized (CHE) [39]

Non-parametric Local Transforms for Computing Visual Correspondence [41] R. Zabih and J. Woodfill, in Proc. European Conference on Computer Vision, 1994.

Stereo Processing by Semiglobal Matching and Mutual Information [22] H. Hirschmuller, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328–341, 2008.

Robust Stereo Matching Using Adaptive Normalized Cross-Correlation [15] The whole window information around matching pixels is used by the NCC in order to find the mean and standard deviation. Weight distribution information around matching pixels using the bilateral filter. Y. S. Heo, K. M. Lee, and S. U. Lee, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 33, no. 4, pp. 807–822, 2011.

Log-chromaticity Color Space [15](1/2) [15] Y. S. Heo, K. M. Lee, and S. U. Lee, “Robust stereo matching using adaptive normalized cross-correlation,” IEEE Trans. Pattern Analysis and Machine Intelligencea

Log-chromaticity Color Space (2/2)

SIFT

Joint probability density function (pdf) represents the statistical relationship between the left and right image color values. compute the joint pdf by means of the SIFT descriptors rather than pixel values to encode the spatial gradient information. Joint Probability Density Function (1/2)

Joint Probability Density Function (2/2)

Linear Function Estimation

Disparity Map Estimation (1/9) [34] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222–1239, 2001.

Disparity Map Estimation (2/9)

Disparity Map Estimation (3/9)

[35] X. Hu and P. Mordohai, “Evaluation of stereo confidence indoors and outdoors,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Disparity Map Estimation (4/9)

Disparity Map Estimation (5/9)

[36] L. Hong and G. Chen, “Segment-based stereo matching using graph cuts,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, [37] J. Sun, Y. Li, S. B. Kang, and H.-Y. Shum, “Symmetric stereo matching for occlusion handling,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, [38] D. Comaniciu and P. Meer, “Mean shift: A robust approach toward feature space analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603–619, Disparity Map Estimation (6/9)

Disparity Map Estimation (7/9)

Disparity Map Estimation (8/9)

[34] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222–1239, Disparity map estimation (9/9)

Fast approximate energy minimization via graph cuts [34] The minimum cut problem is to find the cut with smallest cost. There are numerous algorithms for this problem with low-order polynomial complexity. Using two move to refine that: swap and expansion. T. Gevers and H. Stokman, IEEE Trans. Pattern Analysis and Machine Intelligence, Jan

Occlusion Map Estimation [37] [37] J. Sun, Y. Li, S. B. Kang, and H.-Y. Shum, “Symmetric stereo matching for occlusion handling,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2005.

Color Histogram Equalization (CHE)

Stereo Color Histogram Equalization (SCHE)

Boosting the Disparity Map Estimation Using SCHE Images The color consistency of the SCHE images is based on the accurate disparity map estimation. Conversely, the estimation of the disparity map can benefit from the color-consistent SCHE stereo images.

Experimental Results Using various test images with ground truth disparity map such as Aloe, Dolls, Moebius, Art, Laundry, Reindeer, Rocks1, and Cloth4 dataset in [40] that have different radiometric variations. Each data set has three different camera exposures (0~2) and three different configurations of the light source (1~3). The total running time of our method for Aloe images (size : 427 × 370, disparity range : 0-70), for example, is about 4 minutes on a PC with Intel(R) Core(TM) i7-2600K 4.5GHz CPU. “

Experimental for SCHE

Color consistency performance Computed the RMSE [39] images performed individually for the left and right original images. ‘CHE1’ means RMSE for the stereo images after individually performing CHE using original input stereo images. ‘CHE2’ means RMSE for the stereo images after individually performing CHE using stereo images in the log-chromaticity color space. [39] G. Finlayson, S. Hordley, G. Schaefer, and G. Y. Tian, “Illuminant and device invariant colour using histogram equalisation,” Pattern Recognition, vol. 38, no. 2, pp. 179–190, 2005.

Effects of SIFT Weight in MI Computation

Effects of Adaptive Weight

Stereo Matching Performance Comparison Census transform [41] Mutual Information (MI) [22] Normalized Cross Correlation (NCC) and Adaptive Normalized Cross Correlation (ANCC) [15]. To evaluate the effects of exposure changes, we only changed the index of exposure.

Stereo Matching Performance Comparison

Different Configurations of the Light Source Only changed the index of the light configuration while fixing the exposure.

Tests for Scenes With Different Cameras The left images were taken by Canon IXUS 870 IS, and the right images were taken by Sony Cyber-shot DSC-W570. The left images were taken with flash, while the right images were taken without flash. The exposure times of the left and right images were set 1/60 sec and 1/100 sec, respectively.