1. A Non-local Cost Aggregation Method for Stereo Matching
Qingxiong Yang
City University of Hong Kong
2012 IEEE Conference on
Computer Vision and Pattern Recognition

2. Outilne Introduction Related Works Method Experimental Results
Conclusion

3. Introduction _________________________

4. Introduction Goal : Get fast and accurate disparity map.
Solution : Non-local cost aggregation + MST
Advantage : Better in low textures region
Low complexity

5. Related Works _________________________

6. Related Works Matching cost computation Cost (support) aggregation1
Matching cost computation 2
Cost (support) aggregation 3
Disparity computation and optimization 4
Disparity refinement
[21] D. Scharstein and R. Szeliski.
A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.
International Journal of Computer Vision (IJCV), 47:7–42, 2002.

7. Related Works Local methods Global methods 1=>2=>3
A local support region with winner take all
Implicit smoothness
Fast but inaccurate.
1(=>2)=>3
Energy minimization process (GC,BP,DP,Cooperative)
Per-processing
Explicit smoothness
Accurate but slow

8. Comparison (Rank in Middleburry)
Real time
Non-Real
Method 1 - 2 O
(no Seg.) Cross-based Aggregation>>Scanline Optimization 3
Mean-shift >> BP>>Self-adapting 4
Mean-shift >> Cooperative Optimization 5 6
Mean-shift>>Color-weighted>>BP 7
(no Seg.) Seed Detection>>Scanline Propagation 8
Mean-shift>>BP 9
Mean-shift(Region-based)>>B-spline 10
Up-sample>>Bilateral Filter>>BP 11 12
(No Seg.)>>Convex Relaxation>>regularization 13
Mean-shift>>Image Warping>>BP

9. Method _________________________

10. Method Matching cost computation － Bilateral Filter1
Matching cost computation － Bilateral Filter 2
Cost aggregation － MST 3
Disparity computation and optimization 4
Disparity refinement 　－ Median Filter

11. Bilateral Filter
Every sample is replaced by a weighted average of its neighbors.
These weights reflect two forces
How close are the neighbor and the center sample
How similar are the neighbor and the center sample
Edge-preserving and noise reducing smoothing filter

12. Bilateral Filter q p

13. Bilateral Filter
Center Sample : p
Neighborhood : q

14. Bilateral Filter
Total Distance

15. Bilateral Filter
Original image
Gaussian wieght
Bilateral wieght

16. Minimum Spanning Tree Kruskal's Algorithm
Scan all edges increasing weight order, if an edge is safe, add it to F. 4 1 2 3 5 6 10 A B C D E F G H I J
Orginal Graph
PPT By Jonathan Davis

17. 1 4 2 5 3 10 6 A B D C J E F H G I 4 B C 4 2 1 A 4 E F 1 2 D 3 10 5 G 5 6 3 4 I H 3 2 J

18. Sort Edges
(in reality they are placed in a priority queue - not sorted - but sorting them makes the algorithm easier to visualize) 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D 3 10 5 G 5 6 3 4 I H 3 2 J

19. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

20. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

21. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

22. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

23. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

24. Cycle Don’t Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A

25. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

26. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

27. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

28. Cycle Don’t Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A

29. Add Edge 2 5 4 3 10 1 6 B D J C E F H G I A 4 B C 4 2 1 A 4 E F 1 2 D

30. Minimum Spanning Tree Orginal Graph 4 1 2 3 A B C D E F G H I J 4 1 25 6 10 A B C D E F G H I J

31. Cost Computation
Cd(p) : matching cost for pixel p at disparity level d
: aggregated cost
-- σS and σR : constants used to adjust the similarity.

32. Cost Aggregation on a Tree Structure
Weight between p and q
w(p, q) = | I(p)-I(q)| = image gradient
Distance between p and q
D(p, q) = sum of weights of the connected edges
Similarity between p and q
Aggregated cost ＝＞

33. Bilateral Filter vs Tree Structure

34. Cost Aggregation on a MST
Claim 1. Let Tr denote a subtree of a node s and r denote the root node of Tr, then the supports node s received from this subtree is the summation of the supports node s received from r and S(s, r) times the supports node r received from its subtrees.
Supports r = 𝒗 𝑺(𝒓,𝒗) 𝑪 𝒅 𝒗
Supports s = 𝑺 𝒔,𝒓 𝑪 𝒅 𝒓= 𝒗 𝑺(𝒔,𝒗) 𝑪 𝒅 𝒗 = 𝒗 𝑺(𝒔,𝒓)𝑺(𝒔,𝒗) 𝑪 𝒅 𝒗=𝑺(𝒔,𝒓) 𝒗 𝑺(𝒔,𝒗) 𝑪 𝒅 𝒗 s Tr r

35. Cost Aggregation on a MST
Aggregated cost
=>
, if node v is a leaf node
P(vc) denote parent of nodevc

36. Cost Aggregation on a MST

37. Cost Aggregation on a MST
Aggregated cost
=>

38. Cost Aggregation on a MST
Cost aggregation process
Aggregate the original matching cost Cd from leaf nodes towards root node using Eqn. (6)
Aggregate from root node towards leaf nodes using Eqn. (7)
Complexity
Each level：2 addition/subtraction + 3 multiplication

39. Disparity Refinement D：the left disparity map
Unstable：occlusion, lack of texture, specularity
Median filter overlap

40. Experimental Results _________________________

41. Experimental Results
Device： a MacBook Air laptop computer with a 1.8 GHz Intel Core i7 CPU and 4 GB memory
Parameter：
σ = 0.1 (non-local cost aggregation)
Source : Middlebury
HHI database(book arrival)
Microsofy i2i database(Ilkay)

42. Experimental Results Time：
Proposed average runtime ： 90 milliseconds (1.25× slower)
Unnormalized box filter average runtime ： 72 milliseconds.
Local guided image filter average runtime ： 960 milliseconds
[7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz.
Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.
[24] P. Viola and M. Jones. Robust real-time face detection.
International Journal of Computer Vision, volume 57, pages 137–154, 2003.

43. [7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz.
Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.

44. Experimental Results

45. Experimental Results

46. Experimental Results

47. Conclusion _________________________

48. Conclusion Contributions
Outperform all local cost aggregation methods both in speed and accuracy.
Present a near real-time stereo system with accurate disparity results.
Future works
Apply to parallel algorithms
Refine matching cost estimation