M.S. Student, Hee-Jong Hong Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE M.S. Student, Hee-Jong Hong Sep 24, 2013

Proposed Method : Improve Cost Aggregation Experimental Results Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Outline Introduction Related Works Proposed Method : Improve Cost Aggregation Experimental Results Conclusion

Introduction Goal：Perform efficient cost aggregation. Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Introduction Goal：Perform efficient cost aggregation. Solution : Joint histogram + reduce redundancy Advantage : Low complexity but keep high-quality.

Related Works Complexity of aggregation：O(NBL) Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Related Works N : all pixels (W*H) B : window size L : disparity level Complexity of aggregation：O(NBL) Reduce complexity approach Scale image : Multi Scale Approach D. Min and K. Sohn, “Cost aggregation and occlusion handling with WLS in stereo matching,” IEEE Trans. on Image Processing, 2008. Bilateral filter : Bilateral Approximation C. Richardt, D. Orr, I. P. Davies, A. Criminisi, and N. A. Dodgson, “Real-time spatiotemporal stereo matching using the dual-cross- bilateral grid,” in European Conf. on Computer Vision, 2010 S. Paris and F. Durand, “A fast approximation of the bilateral filter using a signal processing approach,” International Journal of Computer Vision, 2009. Guided filter : Run in constant time => O(NL) C.Rhemann,A.Hosni,M.Bleyer,C.Rother,andM.Gelautz,“Fast cost-volume filtering for visual correspondence and beyond,” in IEEE Conf. on Computer Vision and Pattern Recognition, 2011

Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Proposed Method

Local Method Algorithm Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Local Method Algorithm Cost initialization : Truncated Absolute Difference => Cost aggregation : Weighted filter Disparity computation : Winner take all [4,8]

Improve Cost Aggregation Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Improve Cost Aggregation New formulation for aggregation Remove normalization Joint histogram representation Compact representation for search range Reduce disparity levels Spatial sampling of matching window Regularly sampled neighboring pixels Pixel-independent sampling

New formulation for aggregation Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 New formulation for aggregation Remove normalization => Joint histogram representation J. Ding, J. Liu, W. Zhou, H. Yu, Y. Wang, and X. Gong, “Realtime stereo vision system using adaptive weight cost aggregation approach,” EURASIP J. Image and Video Processing, vol. 2011, p. 20,

Compact Search Range Cost aggregation O( NBD ) => O( NBDc ) Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Compact Search Range Cost aggregation => MC(q)：a subset of disparity levels whose size is Dc. N : all pixels (W*H) B : window size D : disparity level O( NBD ) O( NBDc )

Compact Search Range Non-occluded region of ‘Teddy’ image Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Compact Search Range Non-occluded region of ‘Teddy’ image Dc = 5 (Best) Final Accuracy = 94.2% Dc = 6 Final Accuracy = 94.1% Dc = 60 Final Accuracy = 93.7%

Spatial Sampling of Matching Window Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Spatial Sampling of Matching Window Reason A large matching window and a well-defined weighting function leads to high complexity. Pixels should aggregate in the same object, NOT in the window. Solution Color segmentation => Time consuming (Heavy) Spatial Sampling => Easy but powerful Regular Sampling => Independent from reference pixel => Reduce Complexity

Spatial Sampling of Matching Window Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Spatial Sampling of Matching Window Cost aggregation => S : sampling ratio O( NBDc ) O( NBDc / S2)

Parameter definition Pre-procseeing N : size of image Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Parameter definition N : size of image B : size of matching window N(p)=W×W MD : disparity levels size=D MC : The subset of disparity size=DC<<D S : Sampling ratio Pre-procseeing

Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Result

Experimental Results Pre-processing Post-processing Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Results Pre-processing 5*5 Box filter Post-processing Cross-checking technique Weighted median filter (WMF) Device：Intel Xeon 2.8-GHz CPU (using a single core only) and a 6-GB RAM Parameter setting ( ) = (1.5, 1.7, 31*31, 0.11, 13.5, 2.0)

Experimental Results (a) (b) (c) (d) Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Results (a) (b) (c) (d)

Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Results Using too large box windows (7×7, 9×9) deteriorates the quality, and incurs more computational overhead. Pre-filtering can be seen as the first cost aggregation step and serves the removal of noise.

Experimental Results The smaller S, the better Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Results The smaller S, the better Fig. 5. Performance evaluation: average percent (%) of bad matching pixels for ‘nonocc’, ‘all’ and ‘disc’ regions according to Dc and S. 2 better than 1

Experimental Results The smaller S, the longer Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Results The smaller S, the longer The bigger Dc, the longer

Experimental Results APBP : Average Percentage of Bad Pixels Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Results APBP : Average Percentage of Bad Pixels

Experimental Results Ground truth Error maps Results Original images Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Results Ground truth Error maps Results Original images

Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Experimental Results

Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Conclusion

Conclusion Contribution Future work Joint Histogram Based Cost Aggregation for Stereo Matching - TPAMI 2013 Conclusion Contribution Re-formulate the problem with the relaxed joint histogram. Reduce the complexity of the joint histogram-based aggregation. Achieved both accuracy and efficiency. Future work More elaborate algorithms for selecting the subset of label hypotheses. Estimate the optimal number Dc adaptively. Extend the method to an optical flow estimation.

Thank you!