1. CS 4487/6587 Algorithms for Image Analysis
Correspondence

2. CS 4487/6587 Algorithms for Image Analysis Correspondence
Matching and Correspondence problems
Stereo, motion, non-rigid registration,…
From 1-dimensional to N-dimensional optimization on graphs
Graph cuts vs. DP and Shortest Paths
Extra Reading: Cormen et.al. “Introduction to Algorithms”, Section 26

3. Template matching
image
Face template
image
In matching we estimate “position” of a rigid template in the image
“Position” includes global location parameters of a rigid template:
- translation, rotation, scale,…

4. Flexible template matching
In flexible template matching we estimate “position” of each rigid component of a template

5. Term “correspondence” is normally used for “matching” of low-level features
Matching between two images on a pixel level
Each pixel has to find a corresponding one
Examples: non-rigid registration, stereo
More generally, matching on feature level
E.g. each model feature has to establish one “corresponding” feature in the observed image

6. More examples of correspondence: non-rigid image registration
Often need to find exact correspondences between image pixels
- All such correspondences specify non-rigid transformation (registration) between two images
q = (912,632)
q = T(p;a)
p = (825,856)

7. More examples of correspondence: required in image morphing
How can we specify the warp?
- First specify corresponding points
- Then a morph can be computed (possibly later)

8. More examples of correspondence: Motion (tracking)
beating heart
We have to establish correspondence between specific points
on the object boundary from frame to frame
- correspondences are visualized by

9. More examples of correspondence: Stereo
known
camera
viewpoints
Two views of the same scene from slightly different point of view

10. More examples of correspondence: Stereo
Right image
Left image
Both images are very similar (like images that you see with your two eyes)
all pixels in the left image are present in the right image (except for “few” occlusions)
Normally, images are horizontally aligned so that objects (pixels) in one image simply
shift horizontally in the other image

11. More examples of correspondence: Stereo
Right image
Left image
(After calibration) all correspondences are along the same horizontal scan lines

12. Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
Correspondences are described by horizontal shifts (in x direction only)
Such shifts are also referred to as “disparities”
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

13. Stereo Right image Left image Disparity map (Depth map)
If x-shifts (disparities) are known for all pixels in the left (or right) image then
we can visualize them as a disparity map
Normally disparity relates to object depth (the larger the shift the closer the object )

14. Stereo Correspondence problem
Human vision can solve it
(even for “random dot” stereograms)
Can computer vision solve it?
Maybe
see Middlebury Stereo Database
for the state-of-the art results

15. Stereo Window based Scan-line based approach Muti-scan-line approach
Matching rigid windows around each pixel
Each window is matched independently
Scan-line based approach
Finding coherent correspondences for each scan-line
Scan-lines are independent
DP, shortest paths
Muti-scan-line approach
Finding coherent correspondences for all pixels
Graph cuts

16. Stereo Correspondence problem Window based approach
Right image
Left image C 1 C 2 C 3
Typical window cost function
Location with the best cost wins
f(x,y)
x,y

17. Fixed Windows
Efficient implementation with “sliding column”, Faugeras et. al.’1993 for window cost function
Running time is independent of window size
f(x,y)
x,y - +

18. Problems with Fixed Windows
small window
large window
better at boundaries
noisy in low texture areas
better in low texture areas
blurred boundaries

19. Variable window size algorithms
Correspondences are still found independently at each pixel
All Window-based solution can be though of as “local” solutions
Fast
How to introduce spatial coherence
Introduce Energy function
Searching for “global” solutions

20. Stereo Correspondence problem Scan-line approach
Scan-line stereo
Try to match coherently pixels in each scan line
DP or shortest paths can be used (1D optimization)
Note: scan lines are still matched independently
Streaking artifacts

21. “Shortest paths” for Scan-line stereo
Left image
e.g. Ohta&Kanade’85, Cox at.al.’96
Right image
Right
occlusion s q
Left occlusion t p
correspondence
Actual implementation in OK’85 and C’96 uses DP algorithm

22. DP for scan-line stereo
Viterbi algorithm can be used to optimize the following energy of disparities of pixels p on a fixed scan-line
Left image
Right image
Viterbi can handle this on non-loopy graphs (e.g., scan-lines) =
spatial coherence
photo consistency

23. Coherent stereo on 2D grid?
Scan-line stereo generates streaking artifacts
Can’t use DP (or Dijkstra) to find spatially coherent disparities/correspondances on a 2D grid 
Now we will learn about graph cuts algorithm 

24. Graph cuts for spatially coherent stereo on 2D grids
We will globally minimize the same energy of disparities for pixels p on a grid G = =
photo consistency
spatial coherence
weights of neighborhood edges, n-links, may reflect (right) image intensity gradients (static cues)
Consider a 2D grid G corresponding to (right) image pixels

25. Multi-scan-line stereo with s-t graph cuts (Roy&Cox’98)

26. Multi-scan-line stereo with s-t graph cuts (Roy&Cox’98)
labels
Disparity labels
L(p) p x y

27. Ishikawa&Geiger 98 What energy do we minimize this way?
Concentrate on one pair of neighboring pixels
Disparity labels
- 0
- 1
- 2
- 3
- 4
- 5
cost of horizontal edges
cost of vertical edges

28. Ishikawa&Geiger 98 What energy do we minimize this way?
Concentrate on one pair of neighboring pixels
Disparity labels
- 0
- 1
- 2
- 3
- 4
- 5
cost of horizontal edges
cost of vertical edges

29. Ishikawa&Geiger 98 What energy do we minimize this way?
The combined energy over the entire grid G is s t
cut
(photo consistency)
cost of vertical edges
cost of horizontal edges
(spatial consistency)

30. Scan-line stereo vs. Multi-scan-line stereo
labels
“cut”
labels
L(p) p
L(p) y x x p
Dynamic Programming
(single scan line optimization)
s-t Graph Cuts
(multi-scan-line optimization)

31. Graph Cuts Basics (see Cormen’s book) Simple 2D example
Goal: divide the graph into two parts separating red and blue nodes
s-t graph cut
“source” S T
“sink”
A graph with two terminals S and T
Cut cost is a sum of severed edge weights
Minimum cost s-t cut can be found in polynomial time

32. Graph cut is an old standard problem with tons of applications outside vision
From Harris & Ross [1955]

33. Graph cut is an old standard problem with tons of applications outside vision
From Harris & Ross [1955]

34. s/t min cut algorithms are widely studied (combinatorial optimization)
Augmenting paths [Ford & Fulkerson, 1962]
Push-relabel [Goldberg-Tarjan, 1986]

35. “Augmenting Paths” Find a path from S to T along non-saturated edges
“source”
A graph with two terminals S T
“sink”
Increase flow along this path until some edge saturates

36. “Augmenting Paths” Find a path from S to T along non-saturated edges
“source”
A graph with two terminals S T
“sink”
Increase flow along this path until some edge saturates
Find next path…
Increase flow…

37. “Augmenting Paths” Find a path from S to T along non-saturated edges
 MIN CUT
“source”
A graph with two terminals S T
“sink”
Increase flow along this path until some edge saturates
Iterate until … all paths from S to T have at least one saturated edge
MAX FLOW

38. Some results from Roy&Cox
minimum
cost s/t cut
multi scan line stereo
(graph cuts)
single scan-line stereo
(DP)

39. Some results from Roy&Cox
minimum
cost s/t cut
multi scan line stereo
(graph cuts)
single scan-line stereo
(DP)

40. Simple Examples: Stereo with only 2 depth layers
i2i
projects from
Microsoft Research
CVPR 2005
Background
Substitution

41. Simple Examples: Stereo with only 2 depth layers
i2i
projects from
Microsoft Research
CVPR 2005
Other
Applications

42. HW Assignment 3 Stereo with only 2 depth layers
Equivalent to Foreground/Background binary image segmentation
Take one stereo pair
first frame from from i2i video sequence (see “image samples”)
You can use MyImage library to read the images.
Use graph cuts to detect foreground and background layers in the right image.
Max-flow/min-cut graph library is provided (see “links”)
Foreground (s), e.g. includes large disparities 5,6,7,8,9,10
Background (t), e.g. includes small disparities 0,1,2,3,4 s t
a cut