1. The plan for today Camera matrix
Part A) Notation, preprocessing, and basic concepts.
Part B) 4 Stereo Algorithms
Slides are courtesy of Prof. Ronen Basri

2. Stereo Vision Objective: 3D reconstruction
Input: 2 (or more) images taken with calibrated cameras
Output: 3D structure of scene
Steps:
Rectification
Matching
Depth estimation

3. Rectification Image Reprojection
reproject image planes onto common plane parallel to baseline
Notice, only focal point of camera really matters
(Seitz)

4. Rectification
Any stereo pair can be rectified by rotating and scaling the two image planes (=homography)
We will assume images have been rectified so
Image planes of cameras are parallel.
Focal points are at same height.
Focal lengths same.
Then, epipolar lines fall along the horizontal scan lines of the images

5. Cyclopean Coordinates
Origin at midpoint between camera centers
Axes parallel to those of the two (rectified) cameras

6. Disparity The difference is called “disparity”
d is inversely related to Z: greater sensitivity to nearby points
d is directly related to b: sensitivity to small baseline

7. Main Step: Correspondence Search
What to match?
Objects?
More identifiable, but difficult to compute
Pixels?
Easier to handle, but maybe ambiguous
Edges?
Collections of pixels (regions)?

8. Random Dot Stereogram
Using random dot pairs Julesz showed that recognition is not needed for stereo

9. Random Dot in Motion

10. Finding Matches

11. 1D Search
More efficient
Fewer false matches
SSD error
disparity

12. Finding Matches Under what conditions pixels can be matched?
Ignoring specularities, we can assume that matching pixels have the same brightness (constant brightness assumption)
Still, changes in gain and sensitivity may change the values of pixels

13. Possible solutions Use larger windows Constraint the search
Other metric e.g., Normalized correlation

14. Window Size W = 3 W = 20 Small window: accurate match is more likely
Large window: less false positives
W = 3
W = 20

15. Constraining the Search
Restrict search to epipolar lines (1D search)
Enforce ordering
Problem: not always true
Enforce smoothness
Problem: discontinuities at object boundaries

16. Matching objects vs. Pixels
Left
Right
scanline

17. Ordering

18. Ordering

19. Summary basic ideas
Restrict search to epipolar lines (1D search)
Use larger elements (larger windows, edges, regions)
Problem: large elements may be distorted
Enforce ordering
Problem: not always true
Other similarity measures (e.g., Normalized correlation)
Enforce smoothness
Problem: discontinuities at object boundaries

20. Comparison of Stereo Algorithms
Scene
Ground truth
D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp

21. Scharstein and Szeliski

22. Results with window correlation
Window-based matching
(best window size)
Ground truth

23. Graph Cuts
Graph cuts
Ground truth

24. Correspondence as Optimization
Most stereo algorithms attempt to minimize a functional that usually consists of two terms:
where
- penalizes for quality of a match (unary)
- penalizes non smooth (or even non fronto-parallel) reconstructions (binary)
Many different optimization approaches were proposed

25. Part B) 4 Stereo Algorithms
Dynamic programming
Minimal cut/Max flow
Space carving
Graph cut optimization

26. ?

27. 1D Methods: Dynamic Programming
Discretize the 3-D space
Find the correct curve at every slice
(A slice = epipolar plane)

28. Dynamic programming Find correspondences of each epipolar
line separately

29. Dynamic programming

30. Dynamic programming How do we find the best curve?
Assign weight of all edges
insertion
match
deletion

31. Dynamic programming How do we find the best curve?
Assign weight of all edges
Find shortest path
Dijkstra
insertion
match
deletion

32. Results

33. Dynamic programming Advantages Simple, efficient Globally optimal
Disadvantages
Each slice computed independently (smoothness is not enforced between slices)
Problems due to discretization (tilted planes)

34. Stereo Algorithms Brief review of 4 algorithms: Dynamic programming
Minimal cut/Max flow
Space carving
Graph cut optimization

35. Min Cut/Max Flow

36. Min Cut/Max Flow

37. Min Cut/Max Flow

38. Min Cut/Max Flow

39. Min Cut/Max Flow Main idea: Lets solve all DP problem together.
Objective: find the optimal cut using all the slices simultaneously.

40. Min Cut/Max Flow

41. Min Cut/Max Flow Construct a graph:
Every voxel (3-D point in space) is a node
Every node is connected to its 6 neighbors

42. Min Cut/Max Flow Weights on the edges:
Data cost: change in pixel value
Neighbor
In next slice/row
data
Neighbor
Neighbor
data
Neighbor
In next slice/row

43. Min Cut/Max Flow Weights on the edges:
Data cost: change in pixel value
Smoothness cost: change in depth
smooth
smooth
smooth
smooth

44. Min Cut/Max Flow Weights on the edges:
Data cost: change in pixel value
Smoothness cost: change in depth
data

45. Min Cut/Max Flow
Source
Add source and sink
Find min cut … ∞ ∞
Sink

46. Min Cut/Max Flow
Data penalty
Smoothness penalty

47. Results
Input Min cut Dynamic programming

48. Min Cut/Max Flow Advantages All slices are optimized simultaneously
Efficient
Disadvantages
Extension to multi-camera is difficult
Discretization

49. Space Carving Multi-view stereo
Every point in space corresponds to a match in the images
Compute data term for each match

50. Space Carving Multi-view stereo
Every point in space corresponds to a match in the images
Compute data term for each match (“photo-consistency”)
0.2
0.3
0.9
0.8
0.4
0.5

51. Space Carving Dynamic data term (taking occlusion into account)
Order of sweep is important

52. Space Carving

53. Space Carving
Done for all slices simultaneously

54. Space Carving
Done for all slices simultaneously

55. Space Carving
Done for all slices simultaneously

56. Space Carving Computes a bound on the object, the visual hull
More camera views: better result

57. Space Carving: Results

58. Space Carving: Results

59. Space Carving Advantages True multi-views stereo Handles occlusion
Disadvantages
Limited to visual hull
Lacks smoothness term
Noise may introduce holes, allowing for noise may thicken shape
Discretization

60. Graph Cut Optimization
Stereo is a minimization problem
Possible solution: local search (gradient descent)
Problem: inefficient, local minima
Instead, search larger areas at every iteration

61. Graph Cut Optimization… 2 1 k
Construct a graph to represent the problem:
Nodes:
Pixels (in first image)
k discrete depth values
Edges:
From every pixel node to a
depth node (data term)
Neighboring nodes (smoothness)
Assign weights corresponding to pixel intensities to get a global cost function
depths
pixels

62. Graph Cut Optimization
Objective: Multiway cut
Edges:
Every pixel remains connected to one depth node
Edges between neighboring nodes only if they are connected to same depth node
Nodes are assigned the depth that they are connected to
Multiway cut is NP-complete, solve iteratively … 1 2 3 k … …
depths
pixels

63. Graph Cut Optimization
α-β swap
Nodes labeled α or β, (i.e., connected to or )
can change their labeling to α or β
Edges between neighbors are updated according to the new labeling
Other edges are not changed
Finding best swap = min cut! … 1 2 3 k α β … …
pixels
depths

64. Graph Cut Optimization
Example: 1-2 swap … … 1 2 k 1 2 3 3 k … … … …

65. Graph Cut Optimization
Example: 1-2 swap … 1 … 2 3 k 1 2 3 k … …
Connect the nodes labeled 1 or 2 to both labels

66. Graph Cut Optimization
Example: 1-2 swap … … 1 1 3 k 3 k
Mark 1 as source and 2 as sink
Find minimal cut 2 2

67. Graph Cut Optimization
Example: 1-2 swap … 1 3 k … 1 2 3 k
Erase edges that were on the cut
Result: a new labeling of the 1,2 nodes 2

68. Graph Cut Optimization
Start with an arbitrary labeling
For every pair {α, β} є {1,…,k}
Find the best α-β swap (minimizing the function)
Update the graph (add and erase edges)
Quit when no pair improves the cost function
Induce pixel labels

69. Graph Cuts: Results

70. Graph Cut Optimization1 … 2 3 k
Advantages
State of the art results
Efficient
Bound on approximation quality
Same technique can be applied to other problems (e.g., image restoration)
Disadvantages
Discretization
Occlusion
Still room for improvement

71. Summary Stereo vision: shape reconstruction from two or more images
Steps:
Rectification
Correspondence search
Depth estimation
Algorithms:
Dynamic programming
Min cut/max flow
Space carving
Graph cuts