Road Scene Analysis by Stereovision: a Robust and Quasi-Dense Approach Nicolas Hautière 1, Raphaël Labayrade 2, Mathias Perrollaz 2, Didier Aubert 2 1 LCPC – French Research Institute for Public Works 2 INRETS – French National Institute for Research in Transportation and Safety

2/43 Outline of the Presentation Problematic Problematic The ″v-disparity ″ approach The ″v-disparity ″ approach The quasi-dense matching algorithm The quasi-dense matching algorithm The robust and quasi-dense approach The robust and quasi-dense approach Example Example Conclusion Conclusion

3/43 Problematic Robust detection of both the longitunal and lateral positions of vertical objects by in-vehicle stereovision. Robust detection of both the longitunal and lateral positions of vertical objects by in-vehicle stereovision. Due to real-time constraints, sparse matching techniques are more encountered in the literature, but poorly reconstruct the 3D structure. Due to real-time constraints, sparse matching techniques are more encountered in the literature, but poorly reconstruct the 3D structure. Voting techniques (eg. Hough transform) provide a high rate of robustness: Voting techniques (eg. Hough transform) provide a high rate of robustness:  The v-disparity approach is now widely used  The ″ v-disparity ″ approach is now widely used

4/43 The Road Scene Model The road scene is assumed to be composed of:  A road surface composed of horizontal and oblique planes  Vertical objects considered as vertical planes

5/43 The ″ v-disparity ″ approach [Aubert, 2005] Robust computation of longitudinal position of objects Grabbing of right and left images Computation of a sparse disparity map Computation of ″ v-disparity ″ image Global surfaces extraction Extraction of the longitudinal position ″″ ″v-disparity″ image = v coordinate of a pixel towards its disparity Δ (performing accumulation from the disparity map along scanning lines) [Aubert, 2005] ] D. Aubert and R. Labayrade, “Road obstacles detection by stereovision: the "v-disparity" approach,” Annals of Telecommunications, vol. 60, no. 11–12, 2005.

6/43 The ″v-disparity″ approach Computation of lateral position of objects is problematic ″v-disparity″ approach relies on horizontal gradients ″v-disparity″ approach relies on horizontal gradients Consequently, ″u-disparity″ approach is not robust enough to compute the lateral position of objects, eg: Consequently, ″u-disparity″ approach is not robust enough to compute the lateral position of objects, eg: LIDAR is often used to fill this gap.  LIDAR is often used to fill this gap. u-disparity image ″″ ″u-disparity″ image = u coordinate of a pixel towards its disparity Δ (performing accumulation from the disparity map along scanning lines)

7/43 How to cope with this situation ? Densification of the disparity map is a solution Densification of the disparity map is a solution Problem: dense disparity map schemes are still costly to implement Problem: dense disparity map schemes are still costly to implement An in-between method exists: the quasi-dense matching algorithm [Lhuillier, 2002] but has not been yet tested for in-vehicle stereovision An in-between method exists: the quasi-dense matching algorithm [Lhuillier, 2002] but has not been yet tested for in-vehicle stereovision  Let’s do it ! [Lhuillier, 2002] M. Lhuillier and L. Quan, “Match propagation for image-based modeling and rendering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 8, pp. 1140–1146, 2002.

8/43 The quasi-dense matching algorithm Idea: propagation of the initial seeds in a way similar to a region growing, guided not by a criterion of homogeneity but by a score of correlation Idea: propagation of the initial seeds in a way similar to a region growing, guided not by a criterion of homogeneity but by a score of correlation Initial seeds are the local maxima of ZNCC correlation Initial seeds are the local maxima of ZNCC correlation Disparity propagation if correlation is good enough in close neighborhoods by allowing a small gradient of disparity: Disparity propagation if correlation is good enough in close neighborhoods by allowing a small gradient of disparity: Disparity is propagated only in textured areas, i.e. only if: Disparity is propagated only in textured areas, i.e. only if: a c b B A C Neighborhood of pixel a in I 1 Neighborhood of pixel A in I 2

9/43 Initial seeds (ZNCC>0.9) t=0.05 t=0.01 reconstructed ″″ ″ v-disparity ″ images not reliable !  Disparity is propagated along horizontal edges ″″  However, the method creates some correlated matching errors to which ″ v-disparity ″ approach is sensitive ! The quasi-dense matching algorithm: examples

10/43 Proposed solution: the robust and quasi-dense approach Idea: Idea: 1. Computation of v-disparity image and extraction of global surfaces 1. Computation of ″v-disparity″ image and extraction of global surfaces 2. Propagation of disparity except that 2. Propagation of disparity except that for each match candidate we check if it belongs to one of the planes of the ″v-disparity″ image  We add a global constraint on the quasi-dense matching algorithm

11/43 t=0.05 t=0.01 reconstructed « v-disparity » images OK  By adding a global constraint on the disparity propagation, matching errors are much less numerous !  However, there are till some errors on occluded contours and periodic low textured areas Robust and quasi-dense approach: examples

12/43 Application: extraction of lateral position of objects ″″  ″ u-disparity ″ image computation is now reliable  Fitting a bounding box is possible.

13/43 Results Bad Contrasted Video (Daytime Fog) Standard « u-v disparity » approach  Few false detections, low detection rate Quasi-dense approach (t=0.05)  Good detection rate, lots of false detections Robust and quasi-dense approach (t=0.05)  Good detection rate, few false alarms

14/43 Conclusion We have presented a stereovision method We have presented a stereovision method Based on ″v-disparity″ approach and the quasi-dense matching algorithm Based on ″v-disparity″ approach and the quasi-dense matching algorithm Computing reliable quasi-dense disparity maps Computing reliable quasi-dense disparity maps Detecting robustly both lateral and longitudinal positions of objects Detecting robustly both lateral and longitudinal positions of objects Performing well under adverse conditions Performing well under adverse conditions Perspectives: Perspectives: Quantitative assessment of the method Quantitative assessment of the method Comparison with other schemes Comparison with other schemes