1. Last Time Pinhole camera model, projection
A taste of projective geometry
Two view geometry:
Homography
Epipolar geometry, the essential matrix
Camera calibration, the fundamental matrix

2. Epipolar Lines epipolar plane epipolar lines epipolar lines O O’
Baseline

3. Stereo Vision Objective: 3D reconstruction
Input: 2 (or more) images taken with calibrated cameras
Output: 3D structure of scene
Steps:
Rectification
Matching
Depth estimation

4. Rectification
We will assume images have been rectified so that epipolar lines correspond to scan lines
Image planes of cameras are parallel.
Focal points are at same height.
Focal lengths same.
Then, epipolar lines fall along the horizontal scan lines of the images
Any stereo pair can be rectified by rotating and scaling the two image planes (=homography) so that they become parallel to baseline

5. Rectification Image Reprojection
reproject image planes onto common plane parallel to baseline
Notice, only focal point of camera really matters
(Seitz)

6. Cyclopean Coordinates
Origin at midpoint between camera centers
Axes parallel to those of the two (rectified) cameras

7. Disparity The difference is called “disparity”
d is inversely related to Z: greater sensitivity to nearby points
d is directly related to b: sensitivity to small baseline

8. Main Step: Correspondence Search
What to match?
Objects?
More identifiable, but difficult to compute
Pixels?
Easier to handle, but maybe ambiguous
Edges?
Collections of pixels (regions)?

9. Matching objects vs. Pixels
Left
Right
scanline

10. Random Dot Stereogram
Using random dot pairs Julesz showed that recognition is not needed for stereo

11. Random Dot in Motion

12. Finding Matches Under what conditions pixels can be matched?
Ignoring specularities, we can assume that matching pixels have the same brightness (constant brightness assumption)
Still, changes in gain and sensitivity may change the values of pixels
Common solution:
Use larger windows
Normalized correlation
Pros and cons:
Small window: accurate match is more likely
Large window: fewer candidates
We need a method to eliminate false matches

13. Window Size W = 3 W = 20 Small window: accurate match is more likely
Large window: less false positives
W = 3
W = 20

14. Constraining the Search
Restrict search to epipolar lines (1D search)
Use larger elements (larger windows, edges, regions)
Problem: large elements may be distorted
Enforce smoothness
Problem: discontinuities at object boundaries
Enforce ordering
Problem: not always true

15. 1D Search

16. 1D Search
More efficient
Fewer false matches
SSD error
disparity

17. Ordering

18. Ordering

19. Correspondence as Optimization
Most stereo algorithms attempt to minimize a functional that usually consists of two terms:
where
- penalizes for quality of a match (unary)
- penalizes non smooth (or even non fronto-parallel) reconstructions (binary)
Many different optimization approaches were proposed

20. Comparison of Stereo Algorithms
Scene
Ground truth
D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp

21. Scharstein and Szeliski

22. Results with window correlation
Window-based matching
(best window size)
Ground truth

23. Graph Cuts
Graph cuts
Ground truth

24. Stereo Algorithms We’ll briefly review several algorithms:
Dynamic programming
Minimal cut/Max flow
Space carving
Graph cut optimization

25. ?

26. 1D Methods: Dynamic Programming
Discretize the 3-D space
Find the correct curve at every slice
(A slice = epipolar plane)

27. Dynamic programming Find correspondences of each epipolar
line separately

28. Dynamic programming

29. Dynamic programming How do we find the best curve?
Assign weight of all edges
insertion
match
deletion

30. Dynamic programming How do we find the best curve?
Assign weight of all edges
Find shortest path
Dijkstra
insertion
match
deletion

31. Results

32. Dynamic programming Advantages Simple, efficient Globally optimal
Disadvantages
Each slice computed independently (smoothness is not enforced between slices)
Problems due to discretization (tilted planes)

33. Min Cut/Max Flow

34. Min Cut/Max Flow

35. Min Cut/Max Flow

36. Min Cut/Max Flow

37. Min Cut/Max Flow
Objective: find the optimal cut using all the slices simultaneously.

38. Min Cut/Max Flow Construct a graph:
Every voxel (3-D point in space) is a node
Every node is connected to its 6 neighbors

39. Min Cut/Max Flow Weights on the edges:
Data cost: change in pixel value
data
data

40. Min Cut/Max Flow Weights on the edges:
Data cost: change in pixel value
Smoothness cost: change in depth
smooth
smooth
smooth
smooth

41. Min Cut/Max Flow Weights on the edges:
Data cost: change in pixel value
Smoothness cost: change in depth

42. Min Cut/Max Flow
Source
Add source and sink
Find min cut … ∞ ∞
Sink

43. Min Cut/Max Flow
Data penalty
Smoothness penalty

44. Results
Input Min cut Dynamic programming

45. Min Cut/Max Flow Advantages All slices are optimized simultaneously
Efficient
Disadvantages
Extension to multi-camera is difficult
Discretization

46. Space Carving Multi-view stereo
Every point in space corresponds to a match in the images
Compute data term for each match

47. Space Carving Multi-view stereo
Every point in space corresponds to a match in the images
Compute data term for each match (“photo-consistency”)
0.2
0.3
0.9
0.8
0.4
0.5

48. Space Carving Dynamic data term (taking occlusion into account)
Order of sweep is important

49. Space Carving

50. Space Carving
Done for all slices simultaneously

51. Space Carving
Done for all slices simultaneously

52. Space Carving
Done for all slices simultaneously

53. Space Carving Computes a bound on the object, the visual hull
More camera views: better result

54. Space Carving: Results

55. Space Carving: Results

56. Space Carving Advantages True multi-views stereo Handles occlusion
Disadvantages
Limited to visual hull
Lacks smoothness term
Noise may introduce holes, allowing for noise may thicken shape
Discretization

57. Graph Cut Optimization
Stereo is a minimization problem
Possible solution: local search (gradient descent)
Problem: inefficient, local minima
Instead, search larger areas at every iteration

58. Graph Cut Optimization… 2 1 k
Construct a graph to represent the problem:
Nodes:
Pixels (in first image)
k discrete depth values
Edges:
From every pixel node to a
depth node (data term)
Neighboring nodes (smoothness)
Assign weights corresponding to pixel intensities to get a global cost function
depths
pixels

59. Graph Cut Optimization
Objective: Multiway cut
Edges:
Every pixel remains connected to one depth node
Edges between neighboring nodes only if they are connected to same depth node
Nodes are assigned the depth that they are connected to
Multiway cut is NP-complete, solve iteratively … 1 2 3 k … …
depths
pixels

60. Graph Cut Optimization
α-β swap
Nodes labeled α or β, (i.e., connected to or )
can change their labeling to α or β
Edges between neighbors are updated according to the new labeling
Other edges are not changed
Finding best swap = min cut! … 1 2 3 k α β … …
pixels
depths

61. Graph Cut Optimization
Example: 1-2 swap … … 1 2 k 1 2 3 3 k … … … …

62. Graph Cut Optimization
Example: 1-2 swap … 1 … 2 3 k 1 2 3 k … …
Connect the nodes labeled 1 or 2 to both labels

63. Graph Cut Optimization
Example: 1-2 swap … … 1 1 3 k 3 k
Mark 1 as source and 2 as sink
Find minimal cut 2 2

64. Graph Cut Optimization
Example: 1-2 swap … 1 3 k … 1 2 3 k
Erase edges that were on the cut
Result: a new labeling of the 1,2 nodes 2

65. Graph Cut Optimization
Start with an arbitrary labeling
For every pair {α, β} є {1,…,k}
Find the best α-β swap (minimizing the function)
Update the graph (add and erase edges)
Quit when no pair improves the cost function
Induce pixel labels

66. Graph Cuts: Results

67. Graph Cut Optimization1 … 2 3 k
Advantages
State of the art results
Efficient
Bound on approximation quality
Same technique can be applied to other problems (e.g., image restoration)
Disadvantages
Discretization
Occlusion
Still room for improvement

68. Summary Stereo vision: shape reconstruction from two or more images
Steps:
Rectification
Correspondence search
Depth estimation
Algorithms:
Dynamic programming
Min cut/max flow
Space carving
Graph cuts