2. A Convex Optimization Approach for Depth Estimation Under Illumination Variation Wided Miled, Student Member, IEEE, Jean-Christophe Pesquet, Senior Member, IEEE, and Michel Parent

3. 2 Abstract Illumination changes cause serious problems in many computer vision applications. A spatially varying multiplicative model is developed to account for brightness changes induced between left and right views.Illumination changes cause serious problems in many computer vision applications. A spatially varying multiplicative model is developed to account for brightness changes induced between left and right views. The recovery of the depth information of a scene from stereo images is an active area of research in computer vision. The need for an accurate and dense depth map arises in many applications such as autonomous navigation, 3-D reconstruction and 3-D television.The recovery of the depth information of a scene from stereo images is an active area of research in computer vision. The need for an accurate and dense depth map arises in many applications such as autonomous navigation, 3-D reconstruction and 3-D television. 2

4. 3 I. INTRODUCTION Feature-based methods:Feature-based methods: Extract salient features from both images, such as edges, segments, or curves. Extract salient features from both images, such as edges, segments, or curves. An interpolation step is required if a dense map is desired, but accurate. An interpolation step is required if a dense map is desired, but accurate. Region-based methods:Region-based methods: It have the advantage of directly generating dense disparity estimates by correlation over local windows, but not accurate. It have the advantage of directly generating dense disparity estimates by correlation over local windows, but not accurate. Many global stereo algorithms have, therefore, been developed based on dynamic programming, graph cuts, or belief propagation. Variational approaches have also been very effective for solving the matching problem globally Many global stereo algorithms have, therefore, been developed based on dynamic programming, graph cuts, or belief propagation. Variational approaches have also been very effective for solving the matching problem globally

5. 4 II.MODEL FOR ILLUMINATION VARIATIONS The intensity of an image pixel: I i (s) = ρ(s) R i (n(s)), for i ∈﹛ l,r ﹜。 I i (s) = ρ(s) R i (n(s)), for i ∈﹛ l,r ﹜。 Assuming that the stereo images have been rectified, so that the geometry of the cameras can be considered as horizontal epipolar, and using the Image Irradiance Equation: I r ( x-u(s), y ) = v(s) I l (s) I r ( x-u(s), y ) = v(s) I l (s) 4

6. 5 II.MODEL FOR ILLUMINATION VARIATIONS The disparity u and illumination v can be computed by minimizing the following cost function based on the sum of squared differences (SSD) metric: following cost function based on the sum of squared differences (SSD) metric: Ĵ ( u, v ) = ∑ s ∈ D [ v(s)I l (s) – I r ( x-u(s), y )] 2, D ⊂ N 2 This expression is nonconvex with respect to the displacement field u. Thus, to avoid a nonconvex minimization, we assume that I r is a differentiable function and we consider a Taylor expansion of the nonlinear term I r ( x-ū, y ) around an initial estimate ū as follows: I r ( x-u, y ) ≈ I r ( x-ū, y ) - ( u-ū ) ∇ I r x ( x-ū, y )

7. 6 II.MODEL FOR ILLUMINATION VARIATIONS To simplify the notations: Ĵ ( u, v ) ≈ ∑ s ∈ D [ L 1 (s)u(s) + L 2 (s)v(s) – r(s) ] 2 where L 1 (s) = ∇ I r x ( x-ū, y ), L 2 (s) = I l (s), r(s) = I r ( x-ū(s), y ) + ū(s)L 1 (s) Our goal is to simultaneously recover u and v. Thus, setting w = ( u, v) T and L = [ L 1, L 2 ], we end up with the following quadratic criterion to be minimized: J D J D ( w ) = ∑ s ∈ D [ L(s)w(s) – r(s) ] 2

8. 7 III. SET THEORETIC ESTIMATION FindFind w ∈ S=∩ i=1 m S i such that such that J(w) = inf J(S) where J: H→]-∞,+∞] is a convex function. J: H→]-∞,+∞] is a convex function. (S i ) 1≤i ≤m are closed convex sets of H. (S i ) 1≤i ≤m are closed convex sets of H. Constraint sets can be modelled as level sets : Constraint sets can be modelled as level sets : ∀ i ∈ { 1,…,m }, S i = { w ∈ H | f i (w) ≤ δ i } where where ∀ i ∈ { 1,…,m }, f i :H →R is continuous convex function ∀ i ∈ { 1,…,m }, f i :H →R is continuous convex function (δ i ) 1≤i ≤m are real-valued parameters. (δ i ) 1≤i ≤m are real-valued parameters.

9. 8 III. SET THEORETIC ESTIMATION A. Global Objective Function ( 1 / 2 ) The initial disparity estimate ū: ū(x,y) = arg min u ∈ U ∑ (i,j) ∈ β [ β x,y (u) I l (x+i,y+j) – I r (x+i-u,y+j) ] 2 where U⊂N is the search disparity set。 βcorresponds to the matching block centered at the pixel (x,y)。 β βx,y(u) is the following least squares estimate of the illumination factor for block β: βx,y(u)=∑ (i,j) ∈ β I l (x+i,y+j)I r (x+i-u,y+j) /∑ (i,j) ∈ β I l (x+I,y+i) 2 The initial illumination field ϋ : ϋ(x,y) = βx,y ( ū(x,y) ) )) )

10. 9 III. SET THEORETIC ESTIMATION A. Global Objective Function ( 2 / 2 )A. Global Objective Function ( 2 / 2 ) J D\O (w) = ∑ s ∈ D\O [ L(s)w(s) – r(s) ] 2 J(w) = ∑ s ∈ D\O [ L(s)w(s) – r(s) ] 2 + α∑ s ∈ D | w(s) - ŵ (s). 2 2 where ϋ) is an initial estimate as described above ŵ = ( ū, ϋ) is an initial estimate as described above denotes the Euclidean norm in R 2 |. 2 denotes the Euclidean norm in R 2 is a positive constant α is a positive constant

11. 10 III. SET THEORETIC ESTIMATION B. Convex Constraints 1) Constraints on the Disparity Image: Total Variation Based Regularization: For a differentiable analog image u defined on a spatial domain Ω TV(u) = ∫ Ω | ∇ u(s) | ds where ∇ u denotes the gradient of u S a 1 = { (u,v) ∈ H | TV(u) ≤ T u } where a : stands for analog constraint sets.

12. 11 III. SET THEORETIC ESTIMATION B. Convex Constraints 1) Constraints on the Disparity Image: Disparity Range Constraint: Sa2 = { (u,v) ∈ H | u min ≤ u ≤ u max }

13. 12 III. SET THEORETIC ESTIMATION B. Convex Constraints 1) Constraints on the Disparity Image: Nagel–Enkelmann Based Regularization: where I denotes the 2 2 identity matrix r is chosen according to gradient norm value range |∇I|<<r:uniform areas, |∇I|>>r:edge

14. 13 III. SET THEORETIC ESTIMATION B. Convex Constraints 2) Constraints on the Illumination Field: Tikhonov Based Regularization: Illumination Range Constraint: Sa5 = { (u,v) ∈ H | v min ≤ v ≤ v max } where v min = 0.8 v max = 1.2

15. 14 IV. EXPERIMENTAL RESULTS N β is the total number of pixels inβ

16. 15 IV. EXPERIMENTAL RESULTS (x0, y0) is (128, 128) α is the standard deviation of the illumination change. illumination change.

17. 16 IV. EXPERIMENTAL RESULTS δ s is fixed to 1

18. 17 IV. EXPERIMENTAL RESULTS

19. 18 IV. EXPERIMENTAL RESULTS

20. 19 IV. EXPERIMENTAL RESULTS

21. 20 IV. EXPERIMENTAL RESULTS

22. 21 IV. EXPERIMENTAL RESULTS

23. 22 IV. EXPERIMENTAL RESULTS

24. 23

25. Thank you for your listening ! The more you learn, the more you know. The more you know, the more you forget. The more you forget, the less you know. 2009.09.01