Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee Department of EECS, ASRI, Seoul National University, 151-742, Seoul, Korea IEEE Conference on Computer Vision and Pattern Recognition, 2009.

Outline Introduction System Overview Mutual Information as a Stereo Correspondence Measure Proposed Algorithm Experiments Conclusion

Introduction

Introduction Radiometric variations (between two input images) Degrade the performance of stereo matching algorithms. Mutual Information : Powerful measure which can find any global relationship of intensities Erroneous as regards local radiometric variations

Introduction Different camera exposures (global) Different light configurations (local) Conventional MI Conventional MI

Objective + To present a new method: Based on mutual information combined with SIFT descriptor Superior to the state-of-the art algorithms (conventional mutual information-based) Mutual Information global radiometric variations (camera exposure) SIFT descriptor local radiometric variations (light configuration) +

System Overview

Proposed Alogorithm

Mutual Information (as a Stereo Correspondence Measure)

Energy Function Energy Function: In MAP-MRF framework f: disparity map

Mutual Information Used as a data cost: Measures the mutual dependence of the two random variables Left / Right Image Disparity Map

Mutual Information Entropy: Joint Entropy: P(i): marginal probability of intensity i P(iL,iR): joint probability of intensity iL and iR Entropy Entropy Joint Entropy 1 1 1

Mutual Information Suppose you are reporting the result of rolling a fair eight-sided die. What is the entropy? →The probability distribution is f (x) = 1/8, for x =1··8 , Therefore entropy is: = 8(1/8)log 8 = 3 bits

Mutual Information H( IL, IR ) H( IL | IR ) H( IR | IL ) MI( IL; IR ) Entropy Entropy Joint Entropy H( IL, IR ) H( IL | IR ) H( IR | IL ) MI( IL; IR ) H( IL ) H( IR )

Mutual Information Entropy: Joint Entropy: P(i): marginal probability of intensity i P(iL,iR): joint probability of intensity iL and iR Entropy Entropy Joint Entropy i1 i2 1 1 1

Pixel-wise Mutual Information Previous: Mutual Information of whole images Difficult to use it as a data cost in an energy minimization framework → Pixel-wise

Pixel-wise Mutual Information = P(.) / P(., .) : marginal / joint probability G(.) / G(., .) : 1D / 2D Gaussian function

Left Image Intensity Right Image Intensity

Conventional MI Do not encode spatial information Cannot handle the local radiometric variations caused by light configuration change Collect correspondences in the joint probability assuming that there is a global transformation The shape of the corresponding joint probability is very sparse. Do not encode spatial information

Conventional MI Different camera exposures (global) Different light configurations (local) Conventional MI Conventional MI

Proposed Algorithm

111

Log-chromaticity Color Space Transform the input color images to log- chromaticity color space [5] To deal with local as well as global radiometric variations Used to establish a linear relationship between color values of input images [5] Y. S. Heo, K.M. Lee, and S. U. Lee. Illumination and camera invariant stereo matching. In Proc. of CVPR, 2008

111

SIFT Descriptor Robust and accurately depicts local gradient information Computed for every pixel in the log-chromaticity color space

Energy Function ( ) Data Cost: Mutual Information: SIFT descriptor distance: constant ( ) Log-chromaticity intensity

111

Joint Probability Using SIFT Descriptor A joint probability is computed at each channel by use of the estimated disparity map from the previous iteration. Wrong disparity can induce an incorrect joint probability. Incorporate the spatial information in the joint probability computation step Adopt the SIFT descriptor

Joint Probability Using SIFT Descriptor K-channel SIFT-weighted joint probability: 　 : Euclidean distance VL,K(P) / VR,K(P) : SIFT descriptors for the pixel P l : SIFT descriptor size T = 1 If True T = 0 If false i1 i2

Joint Probability Using SIFT Descriptor A joint probability is computed at each channel Use estimated disparity map from the previous iteration. is governed by the constraint that corresponding pixels should have similar gradient structures.

111

Energy Function Data Cost: Smooth Cost: MI SIFT

Energy Minimization The total energy is minimized by the Graph-cuts expansion algorithm[3]. [3] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI, 23(11):1222–1239, 2001.

Graph Cut e1=e2<e3=e4 G=(V,E) V={v1,v2,v3,v4} (set of all points)，E={e1,e2,e3,e4} (set of all edges) e1=e2<e3=e4

Energy Minimization

Energy Minimization

Experiments

Experimental Results The std. dev σ of the Gaussian function is 10, τ = 30, l = 4*4*8 = 128 The window size of the SIFT descriptor : 9X9 λ = 0.1, μ = 1.1, VMAX=5 The total running time of our method for most images does not exceed 8 minutes. Aloe image (size : 427 X 370 / disparity range : 0-70) is about 6 minutes on a PC with PENTIUM-4 2.4GHz CPU.

MI vs. SIFT L: illum(1)-exp(1) / R: illum(3)-exp(1) MI SIFT MI + SIFT Error Rate 17.6% 11.97 % 9.27 % MI SIFT MI + SIFT Error Rate 26.45% 17.87 % 11.83 %

L: illum(1)-exp(1) / R: illum(3)-exp(1)

Different Exposure Left Image Right Image Ground Truth Proposed Rank/BT NCC ANCC MI

Different Light Source Configurations Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

Different Exposure Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

Different Light Source Configurations Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

Different Exposure Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

Different Light Source Configurations Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

Exposure Exposure Light Configuration Light Configuration

Exposure Exposure Light Configuration Light Configuration

Conclusion

Conclusion Propose a new stereo matching algorithm based on : mutual information (MI) combined with SIFT descriptor Quite robust and accurate to local as well as global radiometric variations