1. Michael Bleyer LVA Stereo Vision
Global Methods 1
Michael Bleyer
LVA Stereo Vision

2. What happened last time?
Local methods
Pros and cons
Adaptive windows
Slanted surfaces
Occlusion handling in local stereo

3. Outline Stereo as an Energy Minimization Problem
Dynamic programming (DP)
Basic Algorithm
DP Algorithms
Scanline Optimization
Tree DP
Semi global matching
Simple Tree Method

4. Michael Bleyer LVA Stereo Vision
Stereo as an Energy Minimization Problem
Michael Bleyer
LVA Stereo Vision

5. Stereo as an Energy Minimization Problem
Define an energy/cost function to measure the quality of a disparity map:
High energy means that the disparity map is bad.
Low energy means it is good.
Energy function is typically in the form of:
where
D is the disparity map of the left image
Edata measures photo consistency
Esmooth measures smoothness
Global methods express smoothness assumption in an explicit form (as a smoothness term).
Let us take a closer look at Edata and Esmooth.

6. The Data Term
Measures the color dissimilarity for each pixel p of the left image I: where
dp is the disparity of p in the disparity map D
m() is a function computing the color dissimilarity between pixels of left and right images.

7. Nodes correspond to pixels of the left image
The Smoothness Term
The smoothness assumption states that neighboring pixels should be assigned to the same (or similar) disparities:
Nodes correspond to pixels of the left image

8. Edges represent interactions between pixels.
The Smoothness Term
The smoothness assumption states that neighboring pixels should be assigned to the same (or similar) disparities:
Edges represent interactions between pixels.
In our case, interactions occur between a pixel and its 4 spatial neighbours.
We state that the pixel should have the same disparity with its neighbours

9. The Smoothness Term Let us write this idea as a term: where
N is the set of all spatial neighbouring pixels in the left image.
s() is a smoothness function that imposes a penalty if two disparities are different from each other.
Let us use the following smoothness function:
P is a user-defined penalty that balances data and smoothness terms
0 if dp = dq
P otherwise

10. This particular smoothness function is called the Potts model
The Smoothness Term
Let us write this idea as a term:
where
N is the set of all spatial neighbouring pixels in the left image.
s() is a smoothness function that imposes a penalty if two disparities are different from each other.
Let us use the following smoothness function:
P is a user-defined penalty that balances data and smoothness terms
This particular smoothness function is called the Potts model
0 if dp = dq
P otherwise

11. Balancing Data and Smoothness Terms
P = 0
P = 5
P = 10
P = 30
P = 50
P = 5000
Disparity maps generated by energy optimization via alpha-expansion algorithm (no global optimum)

12. The Smoothness Term
Our smoothness term defines the following set of (smoothness) interactions:
This is called the 4-connected grid.

13. Optimizing the 4-connected grid
We are looking for a disparity map D that has the minimum energy E(D) among all possible disparity maps.
This is a very difficult problem:
np-complete problem in the general case => It is not possible to compute the optimal disparity map in reasonable time (most likely).
Why is it difficult:
A pixel has influence on all other pixels in the image:
Changing the disparity of the pixel in the top-left corner might change the disparity of the pixel in the bottom left corner.

14. Optimizing the 4-connected grid
We are looking for a disparity map D that has the minimum energy E(D) among all possible disparity map.
In general, this is a very difficult problem:
np-complete problem in general => It is not possible to compute the optimal disparity map in reasonable time.
Why is it difficult:
Every pixel is connected to every other pixel.
Changing the disparity of the pixel in the top-left corner might change the disparity of the pixel in the bottom left corner.
We will spend two sessions on optimization algorithms.
We will learn about the following algorithms:
Dynamic Programming (This session)
Belief Propagation (Next session)
Graph-Cuts (Next session)

15. Application to Other CV Problems
Our energy function measures the quality of an assignment of pixels to labels.
In our case, labels correspond to disparities.
However, labels can have a different meaning => We can use energy minimization approaches to solve a lot of other computer vision problems.

16. Optical Flow (Very Similar to Stereo)
Input:
2 consecutive frames of a video
Desired output:
Map of 2D vectors representing the movement of each pixel
Labels:
All allowed 2D displacement vectors
Data Term:
Color dissimilarity between corresponding pixels
Smoothness term:
Penalty if neighbouring pixels have different 2D vectors

17. Image Denoising Input: Desired output: Labels: Data Term:
Noisy image
Desired output:
Noise-free image
Labels:
255 intensity values
Data Term:
Dissimilarity between pixel’s intensity and assigned intensity
Smoothness term:
Penalty if neighbouring pixels are assigned to different intensities.

18. Inpainting Input: Desired output: Labels: Data Term: Smoothness term:
Image with partially missing information (red rectangle)
Desired output:
Complete image
Labels:
255 intensity values
Data Term:
0 for each label assignment
Smoothness term:
Penalty if neighbouring pixels are assigned to different intensities.

19. Interactive Image Segmentation
Input:
Color Image
Foreground and background scribbles provided by user.
Desired output:
Binary map
Label 0: Pixel belongs to background
Label 1: Pixel belongs to foreground
Data Term:
Dissimilarity between a pixel’s color and the color models of fore-/background.
Smoothness term:
Penalty on 0/1 label transitions

20. Interactive Image Segmentation
Input:
Color Image
Foreground and background scribbles provided by user.
Desired output:
Binary map
Label 0: Pixel belongs to background
Label 1: Pixel belongs to foreground
Data Term:
Dissimilarity between a pixel’s color and the color models of fore-/background.
Smoothness term:
Penalty on 0/1 label transitions
There are many more computer vision problems that can be modelled by our energy function.

21. Generality of energy functions
Apart from smoothness, we can model other assumptions in the energy function.
Some examples for stereo:
Energy gives infinite costs if uniqueness assumption is violated.
Energy is lower if disparity borders coincide with intensity edges.
In general, if you have a computer vision problem:
Think about what a perfect solution should look like.
Try to express the properties of this perfect solution as an energy function.
Apply one of many existing optimization algorithm to find the solution that minimizes your energy function.
~30% of vision papers work like this.

22. Limitations of Energy Minimization
I have implemented an energy minimization approach, but it gives poor results. Why?
There are 2 reasons:
Energy modeling:
Your energy represents a poor model of your problem.
Ideally, the correct solution should have lower energy than all other possible solutions.
Energy minimization:
Your optimization algorithm delivers a solution that is far off from the exact minimum of your energy.
Problem:
You usually do not know which of the two reasons is the problem in your approach.
However, there are strong indications that energy modeling is the major problem at the current state-of-the-art.

23. Limitations of Energy Minimization
I have implemented an energy minimization approach, but it gives poor results. Why?
There are 2 reasons:
Energy modeling:
Your energy represents a poor model of your problem:
Ideally, the correct solution should have lower energy than all other possible solutions.
If this is the disparity map that has lower energy than all other possible disparity maps, you have done a good job in the energy modelling step.

24. Limitations of Energy Minimization
I have implemented an energy minimization approach, but it gives poor results. Why?
There are 2 reasons:
Energy modeling:
Your energy represents a poor model of your problem.
Ideally, the correct solution should have lower energy than all other possible solutions.
Energy minimization:
Your optimization algorithm delivers a solution that is far off from the exact minimum of your energy.
Problem:
You usually do not know which of the two reasons is the problem in your approach.
However, there are strong indications that energy modeling is the major problem at the current state-of-the-art.

25. Limitations of Energy Minimization
I have implemented an energy minimization approach, but it gives poor results. Why?
There are 2 reasons:
Energy modeling:
Your energy represents a poor model of your problem.
Ideally, the correct solution should have lower energy than all other possible solutions.
Energy minimization:
Your optimization algorithm delivers a solution that is far off from the exact minimum of your energy.
Problem:
You usually do not know which of the two reasons is the problem in your approach.
However, there are strong indications that energy modeling is the major problem at the current state-of-the-art.
Result of applying two different optimization algorithms on the same energy function
ICM
Graph-Cuts

26. Limitations of Energy Minimization
I have implemented an energy minimization approach, but it gives poor results. Why?
There are 2 reasons:
Energy modeling:
Your energy represents a poor model of your problem.
Ideally, the correct solution should have lower energy than all other possible solutions.
Energy minimization:
Your optimization algorithm delivers a solution that is far off from the exact minimum of your energy.
Problem:
You usually do not know which of the two reasons is the problem in your approach.
However, there are strong indications that energy modeling is the major problem at the current state-of-the-art.

27. Limitations of Energy Minimization
I have implemented an energy minimization approach, but it gives poor results. Why?
There are 2 reasons:
Energy modeling:
Your energy represents a poor model of your problem.
Ideally, the correct solution should have lower energy than all other possible solutions.
Energy minimization:
Your optimization algorithm delivers a solution that is far off from the exact minimum of your energy.
Problem:
You usually do not know which of the two reasons is the problem in your approach.
However, there are strong indications that energy modeling is the major problem at the current state-of-the-art.
We will spend this and next sessions on the energy optimization problem.
We will then focus on the modelling component.

28. Michael Bleyer LVA Stereo Vision
Dynamic Programming
Michael Bleyer
LVA Stereo Vision

29. Special Case of our Energy Function
Let us come back to our energy function
where Esmooth is implemented by the Potts model.
Optimization of E is np-complete:
There is most likely no algorithm that can give you the exact minimum in reasonable time.
However, there is a special case:
If the smoothness interactions form a tree in the grid graph, the optimal solution can be efficiently computed using dynamic programming (DP).

30. Example of a tree
A tree is a graph that does not contain cycles.

31. Dynamic Programming - Algorithm
Function L(r) computes the exact energy optimum
r represents the root of the tree (can be chosen arbitrarily)
D is the set of all allowed disparities
m(p,d) are the costs for matching pixel p at disparity d.
s(d,d’) gives a penalty if the disparities d and d’ have different values (smoothness function)
Cp is the set of all siblings of p (Those pixels that have p as a direct predecessor on the path to the root node).

32. Dynamic Programming - Algorithm
If the graph was not a tree, this recursion would run forever.
Function L(r) computes the exact energy optimum
r represents the root of the tree (can be chosen arbitrarily)
D is the set of all allowed disparities
m(p,d) are the costs for matching pixel p at disparity d.
s(d,d’) gives a penalty if the disparities d and d’ have different values (smoothness function)
Cp is the set of all siblings of p (Those pixels that have p as a direct predecessor on the path to the root node).

33. Dynamic Programming – An Example
We will use the Potts model to implement s() with P = 10. Matching costs are given by
m(p,d) r s t u d1 5 20 10 d2 15 30 25 r s t u

34. Dynamic Programming – An Example
We will use the Potts model to implement s() with P = 10. Matching costs are given by
The energy of the optimal disparity assignment is 55.
What is the disparity assignment that has led to this optimal energy?
m(p,d) r s t u d1 5 20 10 d2 15 30 25 r s t u

35. Dynamic Programming – An Example
We can find the disparity sequence that has led to the optimum by back-tracking.
We look which disparity was chosen at each pixel and follow this path.
dr = 1 r
ds = 1 s
dt = 1
du = 1 t u

36. Dynamic Programming – An Example
We can find the disparity sequence that has led to the optimum by back-tracking.
We look which disparity was chosen at each pixel and follow this path.
Setting all pixels to disparity 1 represents the optimal disparity assignment in our example.
dr = 1 r
ds = 1 s
dt = 1
du = 1 t u

37. DP on the 4-connected Grid
Problem:
The 4-connected grid is definitely not a tree!

38. DP on the 4-connected Grid
Idea: We can remove edges (smoothness interactions) so that the 4-connected grid becomes a tree.
The following approaches only differ in the way how they erase edges.
Problem:
The 4-connected grid is definitely not a tree!

39. Scanline DP All vertical edges are deleted from the 4-connected grid.
That is what the majority of DP-based approaches do.
Oftentimes, these approaches implement the ordering assumption (I will skip this, because this is not state-of-the-art anymore.)

40. Scanline DP All vertical edges are deleted from the 4-connected grid.
That is what the majority of DP-based approaches do.
Oftentimes, these approaches implement the ordering assumption (I will skip this, because this is not state-of-the-art anymore.)

41. Scanline DP All vertical edges are deleted from the 4-connected grid.
That is what the majority of DP-based approaches do.
Oftentimes, these approaches implement the ordering assumption (I will skip this, because this is not state-of-the-art anymore.)

42. What will be the problem of this approach?
Scanline DP
What will be the problem of this approach?
All vertical edges are deleted from the 4-connected grid.
That is what the majority of DP-based approaches do.
Oftentimes, these approaches implement the ordering assumption (I will skip this, because this is not state-of-the-art anymore.)

43. The Scanline Streaking Problem
Deleting the vertical smoothness edges leads to horizontal streaks in the disparity maps.
The problem is that smoothness between neighbouring scanlines is not enforced.

44. Tree DP by [Veksler, CVPR2005]
We can obtain a tree structure in a smarter way.
Observation:
Disparity discontinuities are typically aligned with intensity edges, hence:
Two neighbouring pixels of similar intensities are very likely to lie on the same disparity.
Two neighbouring pixels of different intensities are less likely to lie on the same disparity.
=> The smoothness edges between neighbouring pixels of very different intensities are the least important ones. => We should remove those.

45. Tree DP by [Veksler, CVPR2005]
Algorithm for obtaining the tree structure:
For each smoothness edge between two pixels p and q:
Compute a weight w(p,q) by
where I(p) denotes pixel p’s intensity.
Build the minimum spanning tree (MST) using the computed weights:
The MST is the tree connecting all pixels whose sum of weights is minimum among all such trees.
The MST can be computed in linear time using standard graph algorithms.

46. Tree DP by [Veksler, CVPR2005]
Pixels of high intensity difference (no smoothness edge)
Pixels of low intensity difference (smoothness edge)

47. Tree DP by [Veksler, CVPR2005]
Horizontal streaks are effectively reduced.
However, vertical streaks or now present as well.

48. Semi-Global Matching by [Hirschmueller, CVPR2005]
Disclaimer:
The original paper is written from a completely different perspective (no tree DP).
Construct an individual tree at each pixel p.
Tree contains the vertical, horizontal and diagonal lines on which p resides (star shape).

49. Semi-Global Matching by [Hirschmueller, CVPR2005]
No streaks, but isolated pixels

50. Semi-Global Matching by [Hirschmueller, CVPR2005]
Texture
If the tree for pixel p does not capture texture, the algorithm will fail.

51. Simple Tree by [Bleyer, VISAPP2008]
Motivation:
Overcome the problem of [Hirschmueller, CVPR2005] in untextured regions
Idea:
Also generate an individual tree at each pixel p.
This tree contains all pixels of the reference view (=> The problem of missing texture is avoided)
2 tree structures:
Horizontal Tree
Vertical Tree

52. Simple Tree by [Bleyer, VISAPP2008]
Texture cannot be missed by these trees.

53. Simple Tree by [Bleyer, VISAPP2008]
No streaks, no problems in untextured regions.

54. Dynamic Programming - Pros and Cons
DP algorithms are very fast (comparable to local methods)
Good tradeoff between speed and accuracy
Cons:
Can only be applied on tree structures.
Erasing smoothness edges leads to performance degradations.
Optimization algorithms that operate on the full 4-connected grid perform better (next session)

55. Summary Principle of global methods:
Energy modeling
Energy minimization
Energy minimization for CV problems different from stereo
Dynamic Programming:
Scanline optimization
Tree DP
Semi-global matching
Simple Tree

56. References
M. Bleyer, M. Gelautz, Simple but Effective Tree Structures for Dynamic Programming-based Stereo Matching, VISAPP 2008.
H. Hirschmueller, Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information, CVPR 2005.
O. Veksler, Stereo Correspondence by Dynamic Programming on a Tree, CVPR 2005.