1. Feature-Based Stereo Matching Using Graph Cuts Gorkem Saygili, Laurens van der Maaten, Emile A. Hendriks ASCI Conference 2011

2. Overview Introduction implementation – Color Segmentation and Key Point Extraction – Local Pixel Matching – Plane Parameters Decision – Plane Assignment Experimental

3. Introduction (1/2) Stereo matching is one of the key topics in 3D computer vision. The main goal is to find an estimate of the depth information. stereo algorithms can be divided into two classes: local and global approaches. – local : based on aggregation windows that take intensity di ff erences into account – global : make implicit smoothness assumptions on the image and try to minimize the total energy of disparity map based on data and smoothness costs.

4. Introduction (2/2) Popular global methods use graph cuts [9,12] or belief propagation [8,10-11] to minimize the energy function. Most of the reliable and robust stereo matching algorithms rely on over-segmentation of the image [7-10,15].

5. Disparity Estimation

6. Color Segmentation and SURF Key Point Extraction (1/2) Disparities at key point is hard to estimate. Recent state-of-the-art disparity estimation algorithms do not incorporate the key point disparities into the disparity estimation. Such non-trivial disparities are easy to estimate with the use of matching key points between stereo pairs.

7. Color Segmentation and SURF Key Point Extraction (2/2)

8. Then, we can obtain a rough estimate of the disparity of the segment e ffi ciently with a bounded disparity search: Local Pixel Matching (1/4)

9. Local Pixel Matching (2/4)

10. The contents of the aggregation window should not contain disparity discontinuities, we use the following adaptive box matching approach: – Take a box around the pixel of interest. – Aggregate the matching cost for only the pixels that lies on the same region as the pixel of interest inside the box as represented in Eq.7. Local Pixel Matching (3/4)

11. The matching is done for both left image and right image and the disparity is assigned in conjunction with a winner-takes-all optimization. In order to find non-occluded matches, a cross- check is performed. To alleviate the foreground fattening e ff ect described in [6], the minimum value of the left and right disparities is used as the final initial estimate: Local Pixel Matching (4/4) [6] M. Gerrits and P. Bekaert. Local Stereo Matching with Segmentation-Based Outlier Rejection. 3 rd Canadian Conference on Computer and Robot Vision, June 2006.

12. Determining Plane Parameters There are three main approaches to estimate the disparity plane parameters: – (1) a RANSAC solution – (2) a histogram-based solution – (3) a Least-squares solution Here, we choose the RANSAC algorithm that only considers non-occluded disparities as input.

13. RANdom SAmple Consensus (RANSAC)(1/2) A model is fitted to the hypothetical inliers. All other data are then tested against the fitted model and, if a point fits well to the estimated model, also considered as a hypothetical inlier. The estimated model is reasonably good if sufficiently many points have been classified as hypothetical inliers. The model is reestimated from all hypothetical inliers, because it has only been estimated from the initial set of hypothetical inliers. Finally, the model is evaluated by estimating the error of the inliers relative to the model.

14. RANdom SAmple Consensus (RANSAC)(2/2) RANSAC is very robust to outliers such that the algorithm can even work e ff ectively when there are only 50 percent of inliers.[10] Wang and Zheng have shown that RANSAC provides even better solutions than histogram-based approach, however the result mostly depends on the initial set of the algorithm [7]. The third approach is sensitive to outliers. Hong and Chen used least-squares solution with only non-occluded pixel disparities inside the segment [9]. [7] Z. Wang and Z. Zheng. A Region Based Stereo Matching Algorithm Using Cooperative Optimization. CVPR,2008. [9] L. Hong and G. Chen. Segment-Based Stereo Matching Using Graph-Cuts. Proc. CVPR, vol. 1, pp. 74-81, 2004. [10] Q. Yang, L. Wang, R. Yang, H. Stewenius and D. Nister. Stereo Matching with Color- Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 3, pp. 492-504, 2009.

15. Disparity Plane Assignment Using Graph Cuts (1/3) Finally, we assigns a disparity plane to each image segment by minimizing an energy function. The energy minimization problem is solved using a graph cut approach in which each node corresponds to a segment. Our aim is to find a labelling f that assigns each segment t ∈ T to its plane label p ∈ P by minimizing the following energy function:

16. the cost of assigning plane labels to the segments Disparity Plane Assignment Using Graph Cuts (2/3)

17. – N(t) : the set of neighbors of t. – w, σ : scaling parameters – β, τ : the boundary length and mean colour di ff erence between t and q. smoothness term that penalizes the discontinuities in plane labels of neighboring segments. Disparity Plane Assignment Using Graph Cuts (3/3)

18. Experimental (1/3) We performed experiments on the image datasets provided by [2]. A disparity value is defined to be erroneous if the absolute di ff erence from ground truth is larger than 1. As in common practice in the evaluation of stereo algorithm, we look at results: – (1) non-occluded pixels only (nonocc) – (2) all pixels (all) – (3) pixels in image regions that are close to a disparity discontinuity (disc). [2] D. Scharstein and R. Szelinski. Middleburry Stereo Vision Page. http://vision.edu/stereo/eval.

19. Experimental (2/3)

20. Experimental (3/3)