1. Binocular Stereo

2. Topics Principle basic equation epipolar line features and strategies for matching Case study Block matching Relaxation DP stereo

3. Basic principles

4. Binocular stereo single image is ambiguous A another image taken from a different direction gives the unique 3D point a’ a”

5. Epipolar line Epipolar plane Epipolar line constraints Corresponding points lie on the Epipolar lines Epipolar line constratints Base line One image point Possible line of sight

6. Epipoles C1C1 C2C2 e1e1 e2e2 intersections of baseline with image planes projection of the optical center in another image the vanishing points of camera motion direction

7. Examples of epipolar lines

10. Rectification rectification

11. Terminology A physical point focal length right image point z left image point base line length right image plane left image plane World coordinate system left image center right image center

12. Pinhole Camera

13. Perspective Projection (X, Y, Z) Image plane X Y -Z u v (u, v) f : focal length View point (Optical center)

14. Basic binocular stereo equation z=-2df/(x”-x’) x”-x’: disparity 2d : base line length x” x’ -z f d d z d + xd - x

15. Features for matching a. brightness b. edges c. edge intervals d. interest points 10 11 12 11 15 16

16. a. relaxation b. coarse to fine c. dynamic programming local optimam Strategies for matching global optimam 10 10 10 10 5 10 10 10 10 10 5 10 10 10 10

17. Classification of stereo methods Features for matching brightness value point edge region Strategies for matching brute-force coarse-to-fine relaxation dynamic programming Constraints for matching epipolar lines disparity limit continuity uniqueness

18. Case study

19. Block-Matching Stereo 1. method b c bcbc 2. problem a. trade-off of window size and resolution b. dull peak b c

20. Cost Function (b) SSD (sum. of squared difference) (a) SAD (sum. of absolute difference) (c) Correlation Near Object left d Background Near Object right Background

21. Moravec Stereo(`79) navigation Moravec “Visual mapping by a robot rover” Proc 6th IJCAI,pp.598-600 (1979)

22. Moravec’s cart Slide stereo Motion stereo

23. Slider stereo (9 eyes stereo) u 9 C 2 = 36 stereo pairs!!! u each stereo has an uncertainty measure u uncertainty = 1 / base-line u each stereo has a confidence measure long base line large uncertainty

26. Coarse to fine expand matching

27. σ estimated distance σ:uncertainty measure area:confidence measure 9 C 2 = 36 curves Interest point

28. 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. Uniqueness Purpose: navigation (Stanford) Moravec Stereo(`81) interest point

29. Near Object Recent Progress left disparity Background How to estimate the disparities? disparity Near Object right Background Minimize some cost function along the epipolar line H. Hirschnuller, "Improvements in Real-Time Correlation-Based Stereo Vision", IEEE Workshop on Stereo and Multi-Baseline Vision, 2001

30. Fatting Effect on Object Boundary Near Object left Background Near Object right Background Background Correspondence Foreground Correspondence u No single window fits at the discontinuity ⇒ Fatting effect of the object

31. Near Object Background Accurate Estimation on Object Boundary u Shiftable Window left Near Object right Background disparity Min{c1,c2,c3,c4} Min{{c1,c2,c3,c4}-c’} c0 c1c2 c3c4 d c0 c1 c3 are used.

32. Consistency Checking u Check if two independent disparity estimation coincide –Left ⇒ Right search –Right ⇒ Left search u Inconsistent disparities are considered as a false match Epipolar line left right Epipolar line 1 st search 2 nd search Check if they coincide

33. Result u SAD with 11x11 window u Shiftable window + consistency checking Disparity mapLeft image

34. Cooperative stereo: Marr-Poggio Stereo(`76) Simulating human visual system (random dot stereo gram) Marr,Poggio “Cooperative computation of stereo disparity” Science 194,283-287

35. Input : random dot stereo left image random dot shift the catch pat right image we can see the height different between the central and peripheral area

36. Constraints –Epipolar line constraint –Uniqueness constraint »each point in a image has only one depth value O.K. No. –Continuity constraint »each point is almost sure to have a depth value near the values of neighbors O.K. No.

37. Uniqueness constraint prohibits two or more matching points on one horizontal or vertical lines continuity constraint attracts more matching on a diagonal line ABCABC D E F A B C ABCABC (E-A) (E-B) (E-C) prohibit attract (D-A) (E-B) (F-C) Same depth

38. nn+1 Relaxation 10 10 10 10 5 10 10 10 10 10 5 10 10 10 10

40. 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness Purpose: simulate the human visual system (MIT) Marr-Poggio Stereo (`76)

41. Recent progress: Graph-cut u Solve graph partition problem in globally optimal way 1.Formulate the problem in energy minimization framework 2.Design a graph such that the sum of cut edges equals to the total energy 3.Find a “Cut” that minimizes the energy i j Vij Cut

42. Example of Graph-cut u Image segmentation Graph={N,e} Ni: Graph node e ij : Edge connecting nodes C: Cut Image={pixel} Vij: Similarity between neighboring pixels Foreground/Background boundary Ni Nj Vij Graph partition Segmentation

43. Solution to a Graph-cut Problem u Min-Cut/Max-Flow algorithm 1.Given source (s) and sink nodes (t) 2.Define capacity on each edge 3.Find the maximum flow from s ⇒ t, satisfying capacity constraints, and cut the bottleneck Yuri Boykov, Vladimir Kolmogorov, "An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision", PAMI, 2004 i j SourceSink Min-Cut = Max-Flow Bottleneck Flow

44. Multi-label Problem u Find the labeling f that minimizes the energy measures the extent to which f is not piece wise smooth measures the disagreement between f and the observed data Measures how well label fp fits pixel p given the observed data Smoothness penalty between adjacent (N) pixels Yuri Boykov and Olga Veksler and Ramin Zabih, “Fast Approximate Energy Minimization via Graph Cuts,” ICCV, 2001

45. u Iterative graph-cut approach u 2 types of move algorithm are proposed –αβ-swap –α-expansion Minimize E under cond. is preserved Minimize E under cond. can be changed to α Multi-label Solution via Graph-cut α β γ α β γ Graph-cut

46. αβ-swap Algorithm 1.Start with an arbitrary labeling f 2.Success:= 0 3.For each pair of labels a.Find f’=arg min E(f’) among f’ within one αβ-swap of f b.If E(f’) < E(f) then f’:=f and success:=1 4.If success=1 goto 2 5.Return f α β γ

47. αβ-swap Graph Structure pqrs α β edgeweight for should be a semi metric

48. αβ-swap Cut u 3 possible cases pq α β pq α β pq α β Cut ααβββα

49. α-expansion Algorithm 1.Start with an arbitrary labeling f 2.Success:= 0 3.For each label a.Find f’=arg min E(f’) among f’ within one α-expansion of f b.If E(f’) < E(f) then f’:=f and success:=1 4.If success=1 goto 2 5.Return f α β γ

50. α-expansion Graph Structure pqrs α edgeweight for should be a metric Auxiliary nodes are added at the boundary of sets P where a b

51. α-expansion Cut u 3 possible cases pq α Cut a pq α a pq α a Because V(a,b) is a metric V(a,b) < V(a,c)+V(c,b) Never happens! ααα

52. Pixel-based Stereo Matching via Graph-cut leftright Image 12L minimizing the cost function by iterative graph cut (α-expansion) Label: Labeling f means disparity assignment p

53. Graph-cut with Occlusions u Occluded pixels are handled explicitly in the graph u Find a subset of A u Find a configuration f such that V. Kolmogorov and R. Zabih, “Computing visual correspondence with occlusions via graph cuts,” ICCV, 2001 leftright p q if the pixels (p,q) correspond otherwise p is an occluded pixel

54. Energy Function to Minimize Occlusion penalty Number of pixels paired with p T(a) is 1 if a=true, otherwise 0

55. Minimization Flow pqrs wxyz left right disparity 0 disparity 1 disparity α (=2) Current assignment Possible assignment after α-expansion α Choose α Partition via Graph-cut (α-expansion) Construct a Graph Assign weight to each edge

56. Result Left ImageGround Truth Graph-cut with Occ. Graph-cut without Occ. L1 correlation

57. DP stereo Ohta-Kanade Stereo(`85) Map making Ohta,Kanade “Stereo by intra- and inter-scanline search using dynamic programming”,IEEE Trans.,Vol. PAMI-7,No.2,pp.139-14

58. now matching become 1D to 1D yet, N line * M L * M R (512 * 100 * 100 * 10 m sec = 15 hours) L1 L2 L3 L4 L5 L6 R1 R2 R3 R4 R5 R6 L R disparity

59. Path Search u Matching problem can be considered as a path search problem u define a cost at each candidate of path segment based some ad-hoc function 10 100 100

60. Dynamic programming We can formalize the path finding problem as the following iterative formula optimum cost to K cost between M and K 3 0 21 Optimum costs are known

61. stereo pair edges

62. pathdisparity depth

63. stereo pair edges depth

64. 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness aerial image analysis (CMU) Ohta-Kanade Stereo(`85) Brightness of interval

65. Recent progress: 4-move, 4-plane DP u Occluded pixels are handled explicitly in 4-move, 4- plane representation u Disparity map is calculated under DP Matching (global energy minimization) A. Criminisi; J. Shotton; A. Blake; C. Rother; P.H.S. Torr, “Efficient Dense-Stereo and Novel-view Synthesis for Gaze Manipulation in One-to-one Teleconferencing,” MSR-TR-2003-59, 2003

66. Occluded path and visible path cannot be distinguished in this representation occluded move Conventional 3-move DP Right Left occluded move Matched move Left scan-line Right scan-line True matching path Approximated matching path Problem Visible only from right Visible from both left and right

67. 4-move DP Occluded path and visible path are handled separately Occluded move (r) Matched move(l) Left scanline Right scanline True matching path Approximated matching path Occluded move(l) Matched move (r)

68. Design of Move Transition Lm Lo Ro Rm Normalized Sum of Squared Difference Move Transition Matching Cost Lm Left Matched Move Right Occluded Move Ro Rm Lo Left Occluded Move Right Matched Move

69. 4-move, 4-plane DP u Each node should hold 4 accumulation costs separately for each move ⇒ 4-plane model

70. Inter-scanline Consistency u Propagate information across scan-lines ⇒ Gaussian filter is applied on Matching cost array Without Gaussian filterWith Gaussian filter Left Occluded Pixels Right Occluded Pixels

71. Result Left Occluded Pixels Right Occluded Pixels Input Images 3-move DP 4-move, 4-plane DP

72. Comparison in Severe Situation Graph-cut with occlusions 4-move, 4-plane DPBlock Matching leftright Input Result Occluded Pixels Texture-less region

73. Summary 1.Two images from two different positions give depth information 2. Epipolar line and plane 3. Basic equation Z=-2df/(x”-x’) x”-x’: disparity 2d : base line length 4. case study Block-matching-based stereo Cooperative stereo DP based stereo

74. Recent directions 1. Block Matching (Local optimization) –Fast for real-time applications (parallel processing, SIMD) –Not accurate in texture-less region 2. Graph-cut variants (Global optimization) uglobally consistent disparity map is obtained u texture-less region is interpolated nicely 3. 4. 4-move, 4-plane, DP Matching (Global optimization) uglobally consistent disparity map is obtained utexture-less region is interpolated nicely uInter scan-line inconsistency is reduced, but yet to be seen 4. Belief Propagation