2. Stanford CS223B Computer Vision, Winter 2006 Lecture 6 Stereo II Professor Sebastian Thrun CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado Stereo

3. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields

4. Sebastian Thrun Stanford University CS223B Computer Vision A Last Word on Preprocessing….

5. Sebastian Thrun Stanford University CS223B Computer Vision Epipolar Rectified Images Epipolar line

6. Sebastian Thrun Stanford University CS223B Computer Vision Epipolar Rectified Images Source: A. Fusiello, Verona, 2000]

7. Sebastian Thrun Stanford University CS223B Computer Vision Image Normalization n Even when the cameras are identical models, there can be differences in gain and sensitivity. n The cameras do not see exactly the same surfaces, so their overall light levels can differ. n For these reasons and more, it is a good idea to normalize the pixels in each window:

8. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields

9. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Phantom points

10. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence via Correlation Rectified images LeftRight scanline SSD error disparity (Same as max-correlation / max-cosine for normalized image patch)

11. Sebastian Thrun Stanford University CS223B Computer Vision Images as Vectors LeftRight Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length.

12. Sebastian Thrun Stanford University CS223B Computer Vision Image Metrics (Normalized) Sum of Squared Differences Normalized Correlation

13. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Using Correlation LeftDisparity Map Images courtesy of Point Grey Research

14. Sebastian Thrun Stanford University CS223B Computer Vision LEFT IMAGE corner line structure Correspondence By Features

15. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence By Features RIGHT IMAGE corner line structure n Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

16. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Correspondences …… Left scanlineRight scanline

17. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

18. Sebastian Thrun Stanford University CS223B Computer Vision Search Over Correspondences Three cases: –Sequential – cost of match –Occluded – cost of no match –Disoccluded – cost of no match Left scanline Right scanline Occluded Pixels Disoccluded Pixels

19. Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

20. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Matching with Dynamic Programming Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Start End

21. Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

22. Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

23. Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching: Examples n View extrapolation results input depth image novel view [Matthies,Szeliski,Kanade’88]

24. Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching n Some other view extrapolation results inputdepth imagenovel view

25. Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching n Compute certainty map from correlations input depth map certainty map

26. Sebastian Thrun Stanford University CS223B Computer Vision DP for Correspondence n Does this always work? n When would it fail? –Failure Example 1 –Failure Example 2 –Failure Example 3

27. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Problem 1 n It is fundamentally ambiguous, even with stereo constraints Ordering constraint……and its failure Figure from Forsyth & Ponce

28. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Problem 2 n Correspondence fail for smooth surfaces n There is currently no good solution to the correspondence problem

29. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Problem 3 n Regions without texture n Highly Specular surfaces n Translucent objects

30. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields

31. Sebastian Thrun Stanford University CS223B Computer Vision How can We Improve Stereo? Space-time stereo scanner uses unstructured light to aid in correspondence Result: Dense 3D mesh (noisy)

32. Sebastian Thrun Stanford University CS223B Computer Vision Prof Marc Levoy @ Stanford By James Davis, Honda Research, Now UCSC

33. Sebastian Thrun Stanford University CS223B Computer Vision rectified Active Stereo (Structured Light)

34. Sebastian Thrun Stanford University CS223B Computer Vision Structured Light: 3-D Result 3D Model3D Snapshot By James Davis, Honda Research

35. Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter http://www.3dvsystems.com

36. Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter http://www.3dvsystems.com

37. Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter http://www.3dvsystems.com

38. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Layered Stereo n Smoothing With Markov Random Fields

39. Sebastian Thrun Stanford University CS223B Computer Vision Disclaimer n The Following Material Shall Not Be Required For the Midterm Exam

40. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Assign pixel to different “layers” (objects, sprites)

41. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Track each layer from frame to frame, compute plane eqn. and composite mosaic n Re-compute pixel assignment by comparing original images to sprites

42. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Re-synthesize original or novel images from collection of sprites

43. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Advantages: –can represent occluded regions –can represent transparent and border (mixed) pixels (sprites have alpha value per pixel) –works on texture-less interior regions n Limitations: –fails for high depth-complexity scenes

44. Sebastian Thrun Stanford University CS223B Computer Vision Fitting Planar Surfaces (with EM) ** ****

45. Sebastian Thrun Stanford University CS223B Computer Vision Expectation Maximization n 3D Model: Planar surface in 3D Distance point-surface surface normal  y x z displacement 

46. Sebastian Thrun Stanford University CS223B Computer Vision Mixture Measurement Model Case 1: Measurement z i caused by plane  j  Case 2: Measurement z i caused by something else

47. Sebastian Thrun Stanford University CS223B Computer Vision Measurement Model with Correspondences correspondence variables C : }

48. Sebastian Thrun Stanford University CS223B Computer Vision Expected Log-Likelihood Function …after some simple math mapping with known data association probabilistic data association

49. Sebastian Thrun Stanford University CS223B Computer Vision The EM Algorithm n E-step: given plane params, compute n M-step: given expectations, compute

50. Sebastian Thrun Stanford University CS223B Computer Vision Choosing the “Right” Number of Planes: AIC J=2J=3J=5J=0J=1J=4 increased data likelihoodincreased prior probability

51. Sebastian Thrun Stanford University CS223B Computer Vision Determining Number of Surfaces J =1 First model component * * J =1 E-Step * * J =3 Add model components J =3 E-Step J =3 M-step J =1 Prune model J =3 Add model components J =3 E/M Steps * J =2 Prune model

52. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Resulting sprite collection

53. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Estimated depth map

54. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields

55. Sebastian Thrun Stanford University CS223B Computer Vision Motivation and Goals James Diebel

56. Sebastian Thrun Stanford University CS223B Computer Vision Motivation and Goals James Diebel

57. Sebastian Thrun Stanford University CS223B Computer Vision Network of Constraints (Markov Random Field) James Diebel

58. Sebastian Thrun Stanford University CS223B Computer Vision MRF Approach to Smoothing n Potential function: contains a sensor-model term and a surface prior n The edge potential is important! n Minimize  by conjugate gradient –Optimize systems with tens of thousands of parameters in just a couple seconds –Time to converge is O(N), between 0.7 sec (25,000 nodes in the MRF) and 25 sec (900,000 nodes) Diebel/Thrun, 2006

59. Sebastian Thrun Stanford University CS223B Computer Vision Possible Edge Potential Functions

60. Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

61. Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

62. Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

63. Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

64. Sebastian Thrun Stanford University CS223B Computer Vision Movies… Movies in Windows Media Player

65. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields