2. Stanford CS223B Computer Vision, Winter 2005 Lecture 6: Stereo 2 Sebastian Thrun, Stanford Rick Szeliski, Microsoft Hendrik Dahlkamp and Dan Morris, Stanford Stereo

3. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Dense and Layered Stereo n Smoothing With Markov Random Fields n Active Range Imaging Technology

4. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Phantom points

5. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence via Correlation Rectified images LeftRight scanline SSD error disparity (Same as max-correlation / max-cosine for normalized image patch)

6. Sebastian Thrun Stanford University CS223B Computer Vision Image Normalization n Even when the cameras are identical models, there can be differences in gain and sensitivity. n The cameras do not see exactly the same surfaces, so their overall light levels can differ. n For these reasons and more, it is a good idea to normalize the pixels in each window:

7. Sebastian Thrun Stanford University CS223B Computer Vision Images as Vectors LeftRight Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length.

8. Sebastian Thrun Stanford University CS223B Computer Vision Image Metrics (Normalized) Sum of Squared Differences Normalized Correlation

9. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Using Correlation LeftDisparity Map Images courtesy of Point Grey Research

10. Sebastian Thrun Stanford University CS223B Computer Vision LEFT IMAGE corner line structure Correspondence By Features

11. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence By Features RIGHT IMAGE corner line structure n Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

12. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Correspondences …… Left scanlineRight scanline

13. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

14. Sebastian Thrun Stanford University CS223B Computer Vision Search Over Correspondences Three cases: –Sequential – cost of match –Occluded – cost of no match –Disoccluded – cost of no match Left scanline Right scanline Occluded Pixels Disoccluded Pixels

15. Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

16. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Matching with Dynamic Programming Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Start End

17. Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

18. Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

19. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence n It is fundamentally ambiguous, even with stereo constraints Ordering constraint……and its failure Figure from Forsyth & Ponce

20. Sebastian Thrun Stanford University CS223B Computer Vision A Last Word on Correspondences n Correspondence fail for smooth surfaces n There is currently no good solution to the correspondence problem

21. Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching n View extrapolation results input depth image novel view [Matthies,Szeliski,Kanade’88]

22. Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching n Some other view extrapolation results inputdepth imagenovel view

23. Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching n Compute certainty map from correlations input depth map certainty map

24. Sebastian Thrun Stanford University CS223B Computer Vision Main Problem n Regions without texture!!!!

25. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Layered Stereo n Smoothing With Markov Random Fields n Active Range Imaging Technology

26. Sebastian Thrun Stanford University CS223B Computer Vision Disclaimer n The Following Material Shall Not Be Required For the Midterm Exam

27. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Assign pixel to different “layers” (objects, sprites)

28. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Track each layer from frame to frame, compute plane eqn. and composite mosaic n Re-compute pixel assignment by comparing original images to sprites

29. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Re-synthesize original or novel images from collection of sprites

30. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Advantages: –can represent occluded regions –can represent transparent and border (mixed) pixels (sprites have alpha value per pixel) –works on texture-less interior regions n Limitations: –fails for high depth-complexity scenes

31. Sebastian Thrun Stanford University CS223B Computer Vision Fitting Planar Surfaces (with EM) ** ****

32. Sebastian Thrun Stanford University CS223B Computer Vision Expectation Maximization n 3D Model: Planar surface in 3D Distance point-surface surface normal  y x z displacement 

33. Sebastian Thrun Stanford University CS223B Computer Vision Mixture Measurement Model Case 1: Measurement z i caused by plane  j  Case 2: Measurement z i caused by something else

34. Sebastian Thrun Stanford University CS223B Computer Vision Measurement Model with Correspondences correspondence variables C : }

35. Sebastian Thrun Stanford University CS223B Computer Vision Expected Log-Likelihood Function …after some simple math mapping with known data association probabilistic data association

36. Sebastian Thrun Stanford University CS223B Computer Vision Choosing the “Right” Number of Planes: AIC J=2J=3J=5J=0J=1J=4 increased data likelihoodincreased prior probability

37. Sebastian Thrun Stanford University CS223B Computer Vision Determining Number of Surfaces J =1 First model component * * J =1 E-Step * * J =3 Add model components J =3 E-Step J =3 M-step J =1 Prune model J =3 Add model components J =3 E/M Steps * J =2 Prune model

38. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Resulting sprite collection

39. Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Estimated depth map

40. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Dense and Layered Stereo n Smoothing With Markov Random Fields n Active Range Imaging Technology

41. Sebastian Thrun Stanford University CS223B Computer Vision Motivation and Goals James Diebel

42. Sebastian Thrun Stanford University CS223B Computer Vision Motivation and Goals James Diebel

43. Sebastian Thrun Stanford University CS223B Computer Vision Network of Constraints (Markov Random Field) James Diebel

44. Sebastian Thrun Stanford University CS223B Computer Vision MRF Approach to Smoothing n Potential function: contains a sensor-model term and a surface prior n The edge potential is important! n Minimize  by conjugate gradient –Optimize systems with tens of thousands of parameters in just a couple seconds –Time to converge is O(N), between 0.7 sec (25,000 nodes in the MRF) and 25 sec (900,000 nodes)

45. Sebastian Thrun Stanford University CS223B Computer Vision Possible Edge Potential Functions

46. Sebastian Thrun Stanford University CS223B Computer Vision Movies… Movies in Windows Media Player

47. Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

48. Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

49. Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

50. Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

51. Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Dense and Layered Stereo n Smoothing With Markov Random Fields n Active Range Imaging Technology

52. Sebastian Thrun Stanford University CS223B Computer Vision How can We Improve Stereo? Space-time stereo scanner uses unstructured light to aid in correspondence Result: Dense 3D mesh (noisy)

53. Sebastian Thrun Stanford University CS223B Computer Vision Prof Marc Levoy @ Stanford By James Davis, Honda Research

54. Sebastian Thrun Stanford University CS223B Computer Vision rectified Active Stereo (Structured Light)

55. Sebastian Thrun Stanford University CS223B Computer Vision Structured Light: 3-D Result 3D Model3D Snapshot By James Davis, Honda Research

56. Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter http://www.3dvsystems.com

57. Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter http://www.3dvsystems.com

58. Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter http://www.3dvsystems.com