2. Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens Stereo

3. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary Stereo n Epipolar Geometry n Fundamental/Essential Matrix p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

4. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence: Where to search? Search window?

5. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

6. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Rectification n Problem: Epipolar lines not parallel to scan lines p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

7. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Rectification n Problem: Epipolar lines not parallel to scan lines p l p r P OlOl OrOr PlPl PrPr Epipolar Plane Epipolar Lines Epipoles at ininity Rectified Images

8. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Epipolar Rectified Stereo Images Epipolar line

9. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Epipolar Rectified Images Source: A. Fusiello, Verona, 2000]

10. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example Rectification

11. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Final Step: Image Normalization n Even when the cameras are identical models, there can be differences in gain and sensitivity. n The cameras do not see exactly the same surfaces, so their overall light levels can differ. n For these reasons and more, it is a good idea to normalize the pixels in each window:

12. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

13. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Phantom points

14. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence via Correlation Rectified images LeftRight scanline SSD error disparity (Same as max-correlation / max-cosine for normalized image patch)

15. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Images as Vectors LeftRight Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length.

16. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Image Metrics (Normalized) Sum of Squared Differences Normalized Correlation

17. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Using Correlation LeftDisparity Map Images courtesy of Point Grey Research

18. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 LEFT IMAGE corner line structure Correspondence By Features

19. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence By Features RIGHT IMAGE corner line structure n Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

20. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Correspondences …… Left scan lineRight scan line

21. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

22. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Search Over Correspondences Three cases: –Sequential – cost of match –Occluded – cost of no match –Disoccluded – cost of no match Left scanline Right scanline Occluded Pixels Disoccluded Pixels

23. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

24. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 DP Algorithm: V[0,0] = 0 V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] } d[i,k] = argmin { … } Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

25. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming DP Algorithm: V[0,0] = 0 V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] } d[i,k] = argmin { … }

26. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Find Stereo Alignment D=[X,Y] repeat until D=[1,1] add D to alignment D = d[D] end Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

27. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Commercial-Grade Stereo n Tyzx, a leading stereo camera manufacturer n (here strapped on our DARPA Grand Challenge vehicle) Disparity map

28. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Dense Stereo Matching: Examples n View extrapolation results input depth image novel view [Matthies,Szeliski,Kanade’88]

29. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Dense Stereo Matching n Some other view extrapolation results inputdepth imagenovel view

30. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Dense Stereo Matching n Compute certainty map from correlations input depth map certainty map

31. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 DP for Correspondence n Does this always work? n When would it fail? –Failure Example 1 –Failure Example 2 –Failure Example 3 –Failure Example 4

32. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Problem 1 n Ambiguities

33. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Problem 2 n Multiple occluding objects Figure from Forsyth & Ponce

34. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Problem 3 n Correspondence fail for smooth surfaces (edge = occlusion boundary, poorly localized) n There is currently no good solution to this correspondence problem

35. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Problem 4 n Regions without texture n Highly Specular surfaces n Translucent objects

36. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Some Stereo Results Side view Top view

37. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 More Stereo Results Side view

38. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 A True Challenge! http://www.well.com/user/jimg/stereo/stereo_list.html

39. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

40. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 How can We Improve Stereo? Space-time stereo scanner uses unstructured light to aid in correspondence Result: Dense 3D mesh (noisy)

41. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Active Stereo: Adding Texture to Scene By James Davis, Honda Research, Now UCSC

42. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 rectified Active Stereo (Structured Light)

43. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structured Light: 3-D Result 3D Snapshot By James Davis, Honda Research

44. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter http://www.3dvsystems.com

45. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter http://www.3dvsystems.com

46. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter http://www.3dvsystems.com

47. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter http://www.3dvsystems.com

48. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter http://www.3dvsystems.com

49. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Scanning Laser Range Finders

50. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Scanning Laser Results

51. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Scanning Laser Results

52. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

53. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Assign pixel to different “layers” (objects, sprites)

54. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Track each layer from frame to frame, compute plane eqn. and composite mosaic n Re-compute pixel assignment by comparing original images to sprites

55. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Re-synthesize original or novel images from collection of sprites

56. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Advantages: –can represent occluded regions –can represent transparent and border (mixed) pixels (sprites have alpha value per pixel) –works on texture-less interior regions n Limitations: –fails for high depth-complexity scenes

57. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Fitting Planar Surfaces (with EM) ** ****

58. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Expectation Maximization n 3D Model: Planar surface in 3D Distance point-surface surface normal a y x z displacement b

59. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Mixture Measurement Model Case 1: Measurement z i caused by plane  j §Case 2: Measurement z i caused by something else

60. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Measurement Model with Correspondences correspondence variables C : }

61. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Expected Log-Likelihood Function …after some simple math mapping with known data association probabilistic data association

62. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The EM Algorithm n E-step: given plane params, compute n M-step: given expectations, compute

63. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Choosing the “Right” Number of Planes: AIC J=2J=3J=5J=0J=1J=4 increased data likelihoodincreased prior probability

64. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Determining Number of Surfaces J =1 First model component * * J =1 E-Step * * J =3 Add model components J =3 E-Step J =3 M-step J =1 Prune model J =3 Add model components J =3 E/M Steps * J =2 Prune model

65. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Resulting sprite collection

66. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Estimated depth map

67. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example (here with laser range finder)

68. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example (here with laser range finder) n Another Example

69. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

70. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Motivation and Goals James Diebel

71. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Motivation and Goals James Diebel

72. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Network of Constraints (Markov Random Field) James Diebel

73. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 MRF Approach to Smoothing n Potential function: contains a sensor-model term and a surface prior n The edge potential is important! n Minimize  by conjugate gradient –Optimize systems with tens of thousands of parameters in just a couple seconds –Time to converge is O(N), between 0.7 sec (25,000 nodes in the MRF) and 25 sec (900,000 nodes) Diebel/Thrun, 2006

74. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Possible Edge Potential Functions

75. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Smoothing James Diebel

76. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Smoothing James Diebel

77. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Smoothing James Diebel

78. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Smoothing James Diebel

79. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Movies… Movies in Windows Media Player

80. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields