1. Machine Learning – Lecture 15
Efficient MRF Inference with Graph Cuts
Bastian Leibe
RWTH Aachen
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAAAAAAAAAAAAAAAAA

2. Lecture Evaluation: Numerical Results
Lecturer
Lecture
B. Leibe

3. Lecture Evaluation: Positive Comments
“Very good lecture. Difficult topics presented in a good & comprehensible way. Good slides.”
“Thanks a lot for the lecture – it gives a very good overview.”
“Good slides. Good examples at whiteboard.”
“Very interesting lecture! ”
“This lecture finally enables me to understand probability theory – but of course it is quite hard to follow the lecture without having understood it beforehand.”
 Glad to hear that!
B. Leibe

4. Lecture Evaluation: Detailed Critique
“At some points I would appreciate if the slides would be a bit more self contained for looking up things later.”
“I think the course is way too hard to follow with only basic understanding in stochastics. A lot of formulae are on the slides but many of them are only explained very roughly.”
“Less formulas, more examples.”
“It is sometimes hard to follow the slides when you haven't been to the lecture (e.g., it is not always clear why different fonts are used in formulas – sometimes it means something, sometimes it doesn't.”
“At the beginning of the semester the textbook wasn't available at the library (and it is very expensive).”

5. Lecture Evaluation: Exercises
“Exercises are too long and are often referring to a lecture two weeks ago.”
“The exercises are very helpful for understanding the contents of the lecture, and the slides in the exercise course are very well-structured and comprehensible.”
B. Leibe

6. Lecture Evaluation: Exercises (cont'd)
“The exercise course is lacking prior examples that own solutions can be based on. Often the idea is to implement something of which the basic understanding has not yet been established. I am disappointed.”
“The only thing that in my opinion could be better are the exercises. The solving of the task normally mean that some algorithm has to be implemented. But the implementation does not require the understanding of the material but just writing down the formulas from slides in Matlab. It would be nice to have more theoretical tasks which really require deeper understanding of the problem and its solution.”
B. Leibe

7. Lecture Evaluation: Varia
“Hard to get fresh air inside with noise outside”
 Unfortunately out of our control... 
B. Leibe

8. Course Outline Fundamentals (2 weeks)
Bayes Decision Theory
Probability Density Estimation
Discriminative Approaches (5 weeks)
Linear Discriminant Functions
Statistical Learning Theory & SVMs
Ensemble Methods & Boosting
Decision Trees & Randomized Trees
Generative Models (4 weeks)
Bayesian Networks
Markov Random Fields
Exact Inference
Applications
B. Leibe

9. Recap: Junction Tree Algorithm
Motivation
Exact inference on general graphs.
Works by turning the initial graph into a junction tree and then running a sum-product-like algorithm.
Intractable on graphs with large cliques.
Main steps
If starting from directed graph, first convert it to an undirected graph by moralization.
Introduce additional links by triangulation in order to reduce the size of cycles.
Find cliques of the moralized, triangulated graph.
Construct a new graph from the maximal cliques.
Remove minimal links to break cycles and get a junction tree.
 Apply regular message passing to perform inference.
B. Leibe

10. Recap: Junction Tree Example
Without triangulation step
The final graph will contain cycles that we cannot break without losing the running intersection property!
B. Leibe
Image source: J. Pearl, 1988

11. Recap: Junction Tree Example
When applying the triangulation
Only small cycles remain that are easy to break.
Running intersection property is maintained.
B. Leibe
Image source: J. Pearl, 1988

12. Recap: MRF Structure for Images
Basic structure
Two components
Observation model
How likely is it that node xi has label Li given observation yi?
This relationship is usually learned from training data.
Neighborhood relations
Simplest case: 4-neighborhood
Serve as smoothing terms.
 Discourage neighboring pixels to have different labels.
This can either be learned or be set to fixed “penalties”.
Noisy observations
“True” image content
B. Leibe

13. Recap: How to Set the Potentials?
Unary potentials
E.g. color model, modeled with a Mixture of Gaussians
 Learn color distributions for each label
B. Leibe

14. Recap: How to Set the Potentials?
Pairwise potentials
Potts Model
Simplest discontinuity preserving model.
Discontinuities between any pair of labels are penalized equally.
Useful when labels are unordered or number of labels is small.
Extension: “contrast sensitive Potts model” where
Discourages label changes except in places where there is also a large change in the observations.
B. Leibe

15. Energy Minimization Goal:
Infer the optimal labeling of the MRF.
Many inference algorithms are available, e.g.
Simulated annealing
Iterated conditional modes (ICM)
Belief propagation
Graph cuts
Variational methods
Monte Carlo sampling
Recently, Graph Cuts have become a popular tool
Only suitable for a certain class of energy functions.
But the solution can be obtained very fast for typical vision problems (~1MPixel/sec).
What you saw in the movie.
Too simple.
Last lecture
Today!
For more complex problems
B. Leibe

16. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Graph construction
Extension to non-binary case
Applications
B. Leibe

17. Example: Image Segmentation
E: {0,1}N → R
0 → fg
1 → bg
N = number of pixels
Image (D)
Slide credit: Pushmeet Kohli
B. Leibe

18. Example: Image Segmentation
E: {0,1}N → R
0 → fg
1 → bg
N = number of pixels
Unary Cost (ci)
Dark (negative) Bright (positive)
Slide credit: Pushmeet Kohli
B. Leibe

19. Example: Image Segmentation
E: {0,1}N → R
0 → fg
1 → bg
N = number of pixels
Discontinuity Cost (cij)
Slide credit: Pushmeet Kohli
B. Leibe

20. Example: Image Segmentation
E: {0,1}N → R
0 → fg
1 → bg
N = number of pixels
How to minimize E(x)?
Global Minimum (x*)
Slide credit: Pushmeet Kohli
B. Leibe

21. Graph Cuts – Basic Idea E(x) Construct a graph such that: s t
Any st-cut corresponds to an assignment of x
The cost of the cut is equal to the energy of x : E(x)
st-mincut t s
Solution
E(x)
Slide credit: Pushmeet Kohli
B. Leibe

22. Graph Cuts for Binary Problems
Idea: convert MRF into source-sink graph s t
a cut
n-links
hard
constraint
Minimum cost cut can be computed in polynomial time
(max-flow/min-cut algorithms)
Slide credit: Yuri Boykov
B. Leibe
[Boykov & Jolly, ICCV’01]

23. Simple Example of Energy
unary potentials
pairwise potentials
t-links
n-links s t
a cut
(binary object segmentation)
Slide credit: Yuri Boykov
B. Leibe

24. Adding Regional Properties
a cut
n-links
t-link
t-link
Regional bias example
Suppose are given
“expected” intensities
of object and background
NOTE: hard constrains are not required, in general.
Slide credit: Yuri Boykov
B. Leibe
[Boykov & Jolly, ICCV’01]

25. Adding Regional Properties
a cut
n-links
t-link
t-link
“expected” intensities of
object and background
can be re-estimated
EM-style optimization
Slide credit: Yuri Boykov
B. Leibe
[Boykov & Jolly, ICCV’01]

26. Adding Regional Properties
More generally, unary potentials can be based on any intensity/color models of object and background.
a cut s t
Object and background color distributions
Slide credit: Yuri Boykov
B. Leibe
[Boykov & Jolly, ICCV’01]

27. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Graph construction
Extension to non-binary case
Applications
Source
Sink v1 v2 2 5 9 4 1
B. Leibe

28. How Does it Work? The s-t-Mincut Problem
Source
Sink v1 v2 2 5 9 4 1
Graph (V, E, C)
Vertices V = {v1, v2 ... vn}
Edges E = {(v1, v2) ....}
Costs C = {c(1, 2) ....}
Slide credit: Pushmeet Kohli
B. Leibe

29. The s-t-Mincut Problem
What is an st-cut?
An st-cut (S,T) divides the nodes between source and sink.
Source 2 9
What is the cost of a st-cut? 1
Sum of cost of all edges going from S to T v1 v2 2 5 4
Sink
= 16
Slide credit: Pushmeet Kohli
B. Leibe

30. The s-t-Mincut Problem
What is an st-cut?
An st-cut (S,T) divides the nodes between source and sink.
Source 2 9
What is the cost of a st-cut? 1
Sum of cost of all edges going from S to T v1 v2 2 5 4
What is the st-mincut?
Sink
st-cut with the minimum cost
= 7
Slide credit: Pushmeet Kohli
B. Leibe

31. How to Compute the s-t-Mincut?
Solve the dual maximum flow problem
Compute the maximum flow between Source and Sink
Source
Sink v1 v2 2 5 9 4 1
Constraints
Edges: Flow < Capacity
Nodes: Flow in = Flow out
In every network, the maximum flow equals the cost of the st-mincut
Min-cut/Max-flow Theorem
Slide credit: Pushmeet Kohli
B. Leibe

32. History of Maxflow Algorithms
Augmenting Path and Push-Relabel
n: #nodes
m: #edges
U: maximum edge weight
Algorithms assume non-negative edge weights
Slide credit: Andrew Goldberg
B. Leibe

33. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 0
Augmenting Path Based Algorithms
Source
Sink v1 v2 2 5 9 4 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

34. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 0
Augmenting Path Based Algorithms
Source
Sink v1 v2 9 4 2 1 2
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 5
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

35. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 0 + 2
Augmenting Path Based Algorithms
Source
Sink v1 v2 9 4 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
2-2
5-2
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

36. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 2
Augmenting Path Based Algorithms
Source
Sink v1 v2 9 4 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 3
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

37. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 2
Augmenting Path Based Algorithms
Source
Sink v1 v2 3 9 4 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

38. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 2
Augmenting Path Based Algorithms
Source
Sink v1 v2 3 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 9 4
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

39. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 2 + 4
Augmenting Path Based Algorithms
Source
Sink v1 v2 3 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 5
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

40. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 6
Augmenting Path Based Algorithms
Source
Sink v1 v2 3 5 2 1
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

41. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 6
Augmenting Path Based Algorithms
Source
Sink v1 v2 2
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 5 1 3
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

42. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 6 + 1
Augmenting Path Based Algorithms
Source
Sink v1 v2 2
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found 4
1-1 2
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

43. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 7
Augmenting Path Based Algorithms
Source
Sink v1 v2 2 4
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

44. Augmenting Path Based Algorithms
Maxflow Algorithms
Flow = 7
Augmenting Path Based Algorithms
Source
Sink v1 v2 2 4
Find path from source to sink with positive capacity
Push maximum possible flow through this path
Repeat until no path can be found
Algorithms assume non-negative capacity
Slide credit: Pushmeet Kohli
B. Leibe

45. Applications: Maxflow in Computer Vision
Specialized algorithms for vision problems
Grid graphs
Low connectivity (m ~ O(n))
Dual search tree augmenting path algorithm
[Boykov and Kolmogorov PAMI 2004]
Finds approximate shortest augmenting paths efficiently.
High worst-case time complexity.
Empirically outperforms other algorithms on vision problems.
Efficient code available on the web
Slide credit: Pushmeet Kohli
B. Leibe

46. When Can s-t Graph Cuts Be Applied?
s-t graph cuts can only globally minimize binary energies that are submodular.
Submodularity is the discrete equivalent to convexity.
Implies that every local energy minimum is a global minimum.
 Solution will be globally optimal.
unary potentials
pairwise potentials
t-links
n-links
[Boros & Hummer, 2002, Kolmogorov & Zabih, 2004]
E(L) can be minimized by s-t graph cuts
Submodularity (“convexity”)
B. Leibe

47. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Graph construction
Extension to non-binary case
Applications
Source
Sink v1 v2 2 5 9 4 1
B. Leibe

48. Recap: Simple Image Segmentation Model
MRF Structure
Example: simple energy function
Smoothness term: penalty cij if neighboring labels disagree.
Observation term: penalty ci if label and observation disagree.
Noisy observations
Observation process
“True” image content
“Smoothness constraints”
Observation
Smoothness
("Potts model")
B. Leibe
Image source: C. Bishop, 2006

49. Converting an MRF into an s-t Graph
Conversion:
Energy:
Unary potentials are straightforward to set.
How?
Just insert xi = 1 and xi = 0 into the unary terms above...
Source
Sink xi xj
B. Leibe

50. Converting an MRF into an s-t Graph
Conversion:
Energy:
Unary potentials are straightforward to set. How?
Pairwise potentials are more tricky, since we don’t know xi!
Trick: the pairwise energy only has an influence if xi  xj.
(Only!) in this case, the cut will go through the edge {xi,xj}.
Source
Sink xi xj
B. Leibe

51. Example: Graph Construction
Source (0) a1 a2
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

52. Example: Graph Construction
Source (0) 2 a1 a2
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

53. Example: Graph Construction
Source (0) 2 a1 a2 5
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

54. Example: Graph Construction
Source (0) 2 9 a1 a2 5 4
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

55. Example: Graph Construction
Source (0) 2 9 1 a1 a2 5 4
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

56. Example: Graph Construction
Source (0) 2 9 1 a1 a2 2 5 4
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

57. Example: Graph Construction
Source (0) 2 9 1 a1 a2 2 5 4
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

58. Example: Graph Construction
Source (0) 2 9
Cost of cut = 11 1 a1 a2
a1 = 1 a2 = 1 2
E (1,1) = 11 5 4
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

59. Example: Graph Construction
Source (0) 2 9
Cost of cut = 7 1 a1 a2
a1 = 1 a2 = 0 2
E (1,0) = 7 5 4
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

60. How Does the Code Look Like?
Graph *g;
For all pixels p
/* Add a node to the graph */
nodeID(p) = g->add_node();
/* Set cost of terminal edges */
set_weights(nodeID(p), fgCost(p), bgCost(p));
end
for all adjacent pixels p,q
add_weights(nodeID(p), nodeID(q), cost);
g->compute_maxflow();
label_p = g->is_connected_to_source(nodeID(p));
// is the label of pixel p (0 or 1)
Source (0)
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

61. How Does the Code Look Like?
Graph *g;
For all pixels p
/* Add a node to the graph */
nodeID(p) = g->add_node();
/* Set cost of terminal edges */
set_weights(nodeID(p), fgCost(p), bgCost(p));
end
for all adjacent pixels p,q
add_weights(nodeID(p), nodeID(q), cost);
g->compute_maxflow();
label_p = g->is_connected_to_source(nodeID(p));
// is the label of pixel p (0 or 1)
Source (0)
bgCost(a1)
bgCost(a2) a1 a2
fgCost(a1)
fgCost(a2)
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

62. How Does the Code Look Like?
Graph *g;
For all pixels p
/* Add a node to the graph */
nodeID(p) = g->add_node();
/* Set cost of terminal edges */
set_weights(nodeID(p), fgCost(p), bgCost(p));
end
for all adjacent pixels p,q
add_weights(nodeID(p), nodeID(q), cost);
g->compute_maxflow();
label_p = g->is_connected_to_source(nodeID(p));
// is the label of pixel p (0 or 1)
Source (0)
bgCost(a1)
bgCost(a2)
cost(p,q) a1 a2
fgCost(a1)
fgCost(a2)
Sink (1)
Slide credit: Pushmeet Kohli
B. Leibe

63. How Does the Code Look Like?
Graph *g;
For all pixels p
/* Add a node to the graph */
nodeID(p) = g->add_node();
/* Set cost of terminal edges */
set_weights(nodeID(p), fgCost(p), bgCost(p));
end
for all adjacent pixels p,q
add_weights(nodeID(p), nodeID(q), cost);
g->compute_maxflow();
label_p = g->is_connected_to_source(nodeID(p));
// is the label of pixel p (0 or 1)
Source (0)
bgCost(a1)
bgCost(a2)
cost(p,q) a1 a2
fgCost(a1)
fgCost(a2)
Sink (1)
a1 = bg a2 = fg
Slide credit: Pushmeet Kohli
B. Leibe

64. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Graph construction
Extension to non-binary case
Applications
other labels 
B. Leibe

65. Dealing with Non-Binary Cases
Limitation to binary energies is often a nuisance.
 E.g. binary segmentation only…
We would like to solve also multi-label problems.
The bad news: Problem is NP-hard with 3 or more labels!
There exist some approximation algorithms which extend graph cuts to the multi-label case:
-Expansion
-Swap
They are no longer guaranteed to return the globally optimal result.
But -Expansion has a guaranteed approximation quality (2-approx) and converges in a few iterations.
B. Leibe

66. -Expansion Move Basic idea:
Break multi-way cut computation into a sequence of binary s-t cuts.
other labels 
Slide credit: Yuri Boykov
B. Leibe

67. -Expansion Algorithm
Start with any initial solution
For each label “” in any (e.g. random) order:
Compute optimal -expansion move (s-t graph cuts).
Decline the move if there is no energy decrease.
Stop when no expansion move would decrease energy.
Slide credit: Yuri Boykov
B. Leibe

68. Example: Stereo Vision
Depth map
ground truth
Depth map
Original pair of “stereo” images
Slide credit: Yuri Boykov
B. Leibe

69. -Expansion Moves
In each -expansion a given label “” grabs space from other labels
initial solution
-expansion
-expansion
-expansion
-expansion
-expansion
-expansion
-expansion
For each move, we choose the expansion that gives the largest decrease in the energy:  binary optimization problem
Slide credit: Yuri Boykov
B. Leibe

70. Topics of This Lecture Solving MRFs with Graph Cuts
Graph cuts for image segmentation
s-t mincut algorithm
Extension to non-binary case
Applications
B. Leibe

71. GraphCut Applications: “GrabCut”
Interactive Image Segmentation [Boykov & Jolly, ICCV’01]
Rough region cues sufficient
Segmentation boundary can be extracted from edges
Procedure
User marks foreground and background regions with a brush.
This is used to create an initial segmentation which can then be corrected by additional brush strokes.
Add interactive image segmentation slide
Additional
segmentation
cues
User segmentation cues
Slide credit: Matthieu Bray

72. Global optimum of the energy
GrabCut: Data Model
Obtained from interactive user input
User marks foreground and background regions with a brush
Alternatively, user can specify a bounding box
Foreground color
Background color
Global optimum of the energy
Slide credit: Carsten Rother
B. Leibe

73. Color model (Mixture of Gaussians) Energy after each iteration
Iterated Graph Cuts
Foreground
Background G R
Foreground & Background
Background G R
Color model (Mixture of Gaussians) 1 2 3 4
Result
Energy after each iteration
Slide credit: Carsten Rother
B. Leibe

74. GrabCut: Example Results
This is included in the newest version of MS Office!
B. Leibe
Image source: Carsten Rother

75. Applications: Interactive 3D Segmentation
Slide credit: Yuri Boykov
B. Leibe
[Y. Boykov, V. Kolmogorov, ICCV’03]

76. Bayesian Networks What we’ve learned so far…
We know they are directed graphical models.
Their joint probability factorizes into conditional probabilities,
We know how to convert them into undirected graphs.
We know how to perform inference for them.
Sum/Max-Product BP for exact inference in (poly)tree-shaped BNs.
Loopy BP for approximate inference in arbitrary BNs.
Junction Tree algorithm for converting arbitrary BNs into trees.
But what are they actually good for?
How do we apply them in practice?
And how do we learn their parameters?
B. Leibe
Image source: C. Bishop, 2006

77. References and Further Reading
A gentle introduction to Graph Cuts can be found in the following paper:
Y. Boykov, O. Veksler, Graph Cuts in Vision and Graphics: Theories and Applications. In Handbook of Mathematical Models in Computer Vision, edited by N. Paragios, Y. Chen and O. Faugeras, Springer, 2006.
Try the GraphCut implementation at
B. Leibe