1. Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space
Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee
Department of EECS, ASRI, Seoul National University, , Seoul, Korea
IEEE Conference on Computer Vision and Pattern Recognition, 2009.

2. Outline Introduction System Overview
Mutual Information as a Stereo Correspondence Measure
Proposed Algorithm
Experiments
Conclusion

3. Introduction

4. Introduction Radiometric variations (between two input images)
Degrade the performance of stereo matching algorithms.
Mutual Information :
Powerful measure which can find any global relationship of intensities
Erroneous as regards local radiometric variations

5. Introduction Different camera exposures (global)
Different light configurations (local)
Conventional MI
Conventional MI

6. Objective + To present a new method:
Based on mutual information combined with SIFT descriptor
Superior to the state-of-the art algorithms (conventional mutual information-based)
Mutual Information
global radiometric variations
(camera exposure)
SIFT descriptor
local radiometric variations
(light configuration) +

7. System Overview

8. Proposed Alogorithm

9. Mutual Information (as a Stereo Correspondence Measure)

10. Energy Function
Energy Function:
In MAP-MRF framework
f: disparity map

11. Mutual Information Used as a data cost:
Measures the mutual dependence of the two random variables
Left / Right
Image
Disparity Map

12. Mutual Information Entropy: Joint Entropy:
P(i): marginal probability of intensity i
P(iL,iR): joint probability of intensity iL and iR
Entropy
Entropy
Joint Entropy 1 1 1

13. Mutual Information
Suppose you are reporting the result of rolling a fair eight-sided die. What is the entropy?
→The probability distribution is f (x) = 1/8, for x =1··8 ,
Therefore entropy is:
= 8(1/8)log 8 = 3 bits

14. Mutual Information H( IL, IR ) H( IL | IR ) H( IR | IL ) MI( IL; IR )
Entropy
Entropy
Joint Entropy
H( IL, IR )
H( IL | IR )
H( IR | IL )
MI( IL; IR )
H( IL )
H( IR )

15. Mutual Information Entropy: Joint Entropy:
P(i): marginal probability of intensity i
P(iL,iR): joint probability of intensity iL and iR
Entropy
Entropy
Joint Entropy i1 i2 1 1 1

16. Pixel-wise Mutual Information
Previous: Mutual Information of whole images
Difficult to use it as a data cost in an energy minimization framework → Pixel-wise

17. Pixel-wise Mutual Information=
P(.) / P(., .) :
marginal / joint probability
G(.) / G(., .) :
1D / 2D Gaussian function

19. Left Image Intensity
Right Image Intensity

20. Conventional MI Do not encode spatial information
Cannot handle the local radiometric variations caused by light configuration change
Collect correspondences in the joint probability assuming that there is a global transformation
The shape of the corresponding joint probability is very sparse.
Do not encode spatial information

21. Conventional MI Different camera exposures (global)
Different light configurations (local)
Conventional MI
Conventional MI

22. Proposed Algorithm

23. 111

24. Log-chromaticity Color Space
Transform the input color images to log- chromaticity color space [5]
To deal with local as well as global radiometric variations
Used to establish a linear relationship between color values of input images
[5] Y. S. Heo, K.M. Lee, and S. U. Lee. Illumination and camera invariant stereo matching. In Proc. of CVPR, 2008

25. 111

26. SIFT Descriptor
Robust and accurately depicts local gradient information
Computed for every pixel in the log-chromaticity color space

27. Energy Function ( ) Data Cost: Mutual Information:
SIFT descriptor distance:
constant
( )
Log-chromaticity
intensity

28. 111

29. Joint Probability Using SIFT Descriptor
A joint probability is computed at each channel
by use of the estimated disparity map from the previous iteration.
Wrong disparity can induce an incorrect joint probability.
Incorporate the spatial information in the joint probability computation step
Adopt the SIFT descriptor

30. Joint Probability Using SIFT Descriptor
K-channel SIFT-weighted joint probability:
　 : Euclidean distance
VL,K(P) / VR,K(P) : SIFT descriptors for the pixel P
l : SIFT descriptor size
T = 1 If True
T = 0 If false i1 i2

31. Joint Probability Using SIFT Descriptor
A joint probability is computed at each channel
Use estimated disparity map from the previous iteration.
is governed by the constraint that corresponding pixels should have similar gradient structures.

32. 111

33. Energy Function
Data Cost:
Smooth Cost: MI
SIFT

34. Energy Minimization
The total energy is minimized by the Graph-cuts expansion algorithm[3].
[3] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI, 23(11):1222–1239, 2001.

35. Graph Cut e1=e2<e3=e4 G=(V,E)
V={v1,v2,v3,v4} (set of all points)，E={e1,e2,e3,e4} (set of all edges)
e1=e2<e3=e4

36. Energy Minimization

37. Energy Minimization

38. Experiments

39. Experimental Results
The std. dev σ of the Gaussian function is 10, τ = 30, l = 4*4*8 = 128
The window size of the SIFT descriptor : 9X9
λ = 0.1, μ = 1.1, VMAX=5
The total running time of our method for most images does not exceed 8 minutes.
Aloe image (size : 427 X 370 / disparity range : 0-70) is about 6 minutes on a PC with PENTIUM GHz CPU.

40. MI vs. SIFT L: illum(1)-exp(1) / R: illum(3)-exp(1) MI SIFT MI + SIFT
Error Rate
17.6% % %
MI SIFT MI + SIFT
Error Rate
26.45% % %

41. L: illum(1)-exp(1) / R: illum(3)-exp(1)

42. Different Exposure Left Image Right Image Ground Truth Proposed
Rank/BT NCC ANCC MI

43. Different Light Source Configurations
Left Image Right Image Ground Truth Proposed
111
Rank/BT NCC ANCC MI

44. Different Exposure Left Image Right Image Ground Truth Proposed 111
Rank/BT NCC ANCC MI

45. Different Light Source Configurations
Left Image Right Image Ground Truth Proposed
111
Rank/BT NCC ANCC MI

46. Different Exposure Left Image Right Image Ground Truth Proposed 111
Rank/BT NCC ANCC MI

47. Different Light Source Configurations
Left Image Right Image Ground Truth Proposed
111
Rank/BT NCC ANCC MI

48. Exposure
Exposure
Light Configuration
Light Configuration

49. Exposure
Exposure
Light Configuration
Light Configuration

50. Conclusion

51. Conclusion Propose a new stereo matching algorithm based on :
mutual information (MI) combined with
SIFT descriptor
Quite robust and accurate to local as well as global radiometric variations