Machine Learning – Lecture 15 Efficient MRF Inference with Graph Cuts 23.01.2014 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAAAAAAAAAAAAAAAAA

Lecture Evaluation: Numerical Results Lecturer Lecture B. Leibe

Lecture Evaluation: Positive Comments “Very good lecture. Difficult topics presented in a good & comprehensible way. Good slides.” “Thanks a lot for the lecture – it gives a very good overview.” “Good slides. Good examples at whiteboard.” “Very interesting lecture! ” “This lecture finally enables me to understand probability theory – but of course it is quite hard to follow the lecture without having understood it beforehand.”  Glad to hear that! B. Leibe

Lecture Evaluation: Detailed Critique “At some points I would appreciate if the slides would be a bit more self contained for looking up things later.” “I think the course is way too hard to follow with only basic understanding in stochastics. A lot of formulae are on the slides but many of them are only explained very roughly.” “Less formulas, more examples.” “It is sometimes hard to follow the slides when you haven't been to the lecture (e.g., it is not always clear why different fonts are used in formulas – sometimes it means something, sometimes it doesn't.” “At the beginning of the semester the textbook wasn't available at the library (and it is very expensive).”

Lecture Evaluation: Exercises “Exercises are too long and are often referring to a lecture two weeks ago.” “The exercises are very helpful for understanding the contents of the lecture, and the slides in the exercise course are very well-structured and comprehensible.” B. Leibe

Lecture Evaluation: Exercises (cont'd) “The exercise course is lacking prior examples that own solutions can be based on. Often the idea is to implement something of which the basic understanding has not yet been established. I am disappointed.” “The only thing that in my opinion could be better are the exercises. The solving of the task normally mean that some algorithm has to be implemented. But the implementation does not require the understanding of the material but just writing down the formulas from slides in Matlab. It would be nice to have more theoretical tasks which really require deeper understanding of the problem and its solution.” B. Leibe

Lecture Evaluation: Varia “Hard to get fresh air inside with noise outside”  Unfortunately out of our control...  B. Leibe

Course Outline Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Discriminative Approaches (5 weeks) Linear Discriminant Functions Statistical Learning Theory & SVMs Ensemble Methods & Boosting Decision Trees & Randomized Trees Generative Models (4 weeks) Bayesian Networks Markov Random Fields Exact Inference Applications B. Leibe

Recap: Junction Tree Algorithm Motivation Exact inference on general graphs. Works by turning the initial graph into a junction tree and then running a sum-product-like algorithm. Intractable on graphs with large cliques. Main steps If starting from directed graph, first convert it to an undirected graph by moralization. Introduce additional links by triangulation in order to reduce the size of cycles. Find cliques of the moralized, triangulated graph. Construct a new graph from the maximal cliques. Remove minimal links to break cycles and get a junction tree.  Apply regular message passing to perform inference. B. Leibe

Recap: Junction Tree Example Without triangulation step The final graph will contain cycles that we cannot break without losing the running intersection property! B. Leibe Image source: J. Pearl, 1988

Recap: Junction Tree Example When applying the triangulation Only small cycles remain that are easy to break. Running intersection property is maintained. B. Leibe Image source: J. Pearl, 1988

Recap: MRF Structure for Images Basic structure Two components Observation model How likely is it that node xi has label Li given observation yi? This relationship is usually learned from training data. Neighborhood relations Simplest case: 4-neighborhood Serve as smoothing terms.  Discourage neighboring pixels to have different labels. This can either be learned or be set to fixed “penalties”. Noisy observations “True” image content B. Leibe

Recap: How to Set the Potentials? Unary potentials E.g. color model, modeled with a Mixture of Gaussians  Learn color distributions for each label B. Leibe

Recap: How to Set the Potentials? Pairwise potentials Potts Model Simplest discontinuity preserving model. Discontinuities between any pair of labels are penalized equally. Useful when labels are unordered or number of labels is small. Extension: “contrast sensitive Potts model” where Discourages label changes except in places where there is also a large change in the observations. B. Leibe

Energy Minimization Goal: Infer the optimal labeling of the MRF. Many inference algorithms are available, e.g. Simulated annealing Iterated conditional modes (ICM) Belief propagation Graph cuts Variational methods Monte Carlo sampling Recently, Graph Cuts have become a popular tool Only suitable for a certain class of energy functions. But the solution can be obtained very fast for typical vision problems (~1MPixel/sec). What you saw in the movie. Too simple. Last lecture Today! For more complex problems B. Leibe

Topics of This Lecture Solving MRFs with Graph Cuts Graph cuts for image segmentation s-t mincut algorithm Graph construction Extension to non-binary case Applications B. Leibe

Example: Image Segmentation E: {0,1}N → R 0 → fg 1 → bg N = number of pixels Image (D) Slide credit: Pushmeet Kohli B. Leibe

Example: Image Segmentation E: {0,1}N → R 0 → fg 1 → bg N = number of pixels Unary Cost (ci) Dark (negative) Bright (positive) Slide credit: Pushmeet Kohli B. Leibe

Example: Image Segmentation E: {0,1}N → R 0 → fg 1 → bg N = number of pixels Discontinuity Cost (cij) Slide credit: Pushmeet Kohli B. Leibe

Example: Image Segmentation E: {0,1}N → R 0 → fg 1 → bg N = number of pixels How to minimize E(x)? Global Minimum (x*) Slide credit: Pushmeet Kohli B. Leibe

Graph Cuts – Basic Idea E(x) Construct a graph such that: s t Any st-cut corresponds to an assignment of x The cost of the cut is equal to the energy of x : E(x) st-mincut t s Solution E(x) Slide credit: Pushmeet Kohli B. Leibe

Graph Cuts for Binary Problems Idea: convert MRF into source-sink graph s t a cut n-links hard constraint Minimum cost cut can be computed in polynomial time (max-flow/min-cut algorithms) Slide credit: Yuri Boykov B. Leibe [Boykov & Jolly, ICCV’01]

Simple Example of Energy unary potentials pairwise potentials t-links n-links s t a cut (binary object segmentation) Slide credit: Yuri Boykov B. Leibe

Adding Regional Properties a cut n-links t-link t-link Regional bias example Suppose are given “expected” intensities of object and background NOTE: hard constrains are not required, in general. Slide credit: Yuri Boykov B. Leibe [Boykov & Jolly, ICCV’01]

Adding Regional Properties a cut n-links t-link t-link “expected” intensities of object and background can be re-estimated EM-style optimization Slide credit: Yuri Boykov B. Leibe [Boykov & Jolly, ICCV’01]

Adding Regional Properties More generally, unary potentials can be based on any intensity/color models of object and background. a cut s t Object and background color distributions Slide credit: Yuri Boykov B. Leibe [Boykov & Jolly, ICCV’01]

Topics of This Lecture Solving MRFs with Graph Cuts Graph cuts for image segmentation s-t mincut algorithm Graph construction Extension to non-binary case Applications Source Sink v1 v2 2 5 9 4 1 B. Leibe

How Does it Work? The s-t-Mincut Problem Source Sink v1 v2 2 5 9 4 1 Graph (V, E, C) Vertices V = {v1, v2 ... vn} Edges E = {(v1, v2) ....} Costs C = {c(1, 2) ....} Slide credit: Pushmeet Kohli B. Leibe

The s-t-Mincut Problem What is an st-cut? An st-cut (S,T) divides the nodes between source and sink. Source 2 9 What is the cost of a st-cut? 1 Sum of cost of all edges going from S to T v1 v2 2 5 4 Sink 5 + 2 + 9 = 16 Slide credit: Pushmeet Kohli B. Leibe

The s-t-Mincut Problem What is an st-cut? An st-cut (S,T) divides the nodes between source and sink. Source 2 9 What is the cost of a st-cut? 1 Sum of cost of all edges going from S to T v1 v2 2 5 4 What is the st-mincut? Sink st-cut with the minimum cost 2 + 1 + 4 = 7 Slide credit: Pushmeet Kohli B. Leibe

How to Compute the s-t-Mincut? Solve the dual maximum flow problem Compute the maximum flow between Source and Sink Source Sink v1 v2 2 5 9 4 1 Constraints Edges: Flow < Capacity Nodes: Flow in = Flow out In every network, the maximum flow equals the cost of the st-mincut Min-cut/Max-flow Theorem Slide credit: Pushmeet Kohli B. Leibe

History of Maxflow Algorithms Augmenting Path and Push-Relabel n: #nodes m: #edges U: maximum edge weight Algorithms assume non-negative edge weights Slide credit: Andrew Goldberg B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 0 Augmenting Path Based Algorithms Source Sink v1 v2 2 5 9 4 1 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 0 Augmenting Path Based Algorithms Source Sink v1 v2 9 4 2 1 2 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found 5 Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 0 + 2 Augmenting Path Based Algorithms Source Sink v1 v2 9 4 2 1 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found 2-2 5-2 Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 2 Augmenting Path Based Algorithms Source Sink v1 v2 9 4 2 1 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found 3 Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 2 Augmenting Path Based Algorithms Source Sink v1 v2 3 9 4 2 1 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 2 Augmenting Path Based Algorithms Source Sink v1 v2 3 2 1 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found 9 4 Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 2 + 4 Augmenting Path Based Algorithms Source Sink v1 v2 3 2 1 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found 5 Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 6 Augmenting Path Based Algorithms Source Sink v1 v2 3 5 2 1 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 6 Augmenting Path Based Algorithms Source Sink v1 v2 2 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found 5 1 3 Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 6 + 1 Augmenting Path Based Algorithms Source Sink v1 v2 2 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found 4 1-1 2 Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 7 Augmenting Path Based Algorithms Source Sink v1 v2 2 4 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Augmenting Path Based Algorithms Maxflow Algorithms Flow = 7 Augmenting Path Based Algorithms Source Sink v1 v2 2 4 Find path from source to sink with positive capacity Push maximum possible flow through this path Repeat until no path can be found Algorithms assume non-negative capacity Slide credit: Pushmeet Kohli B. Leibe

Applications: Maxflow in Computer Vision Specialized algorithms for vision problems Grid graphs Low connectivity (m ~ O(n)) Dual search tree augmenting path algorithm [Boykov and Kolmogorov PAMI 2004] Finds approximate shortest augmenting paths efficiently. High worst-case time complexity. Empirically outperforms other algorithms on vision problems. Efficient code available on the web http://pub.ist.ac.at/~vnk/software.html Slide credit: Pushmeet Kohli B. Leibe

When Can s-t Graph Cuts Be Applied? s-t graph cuts can only globally minimize binary energies that are submodular. Submodularity is the discrete equivalent to convexity. Implies that every local energy minimum is a global minimum.  Solution will be globally optimal. unary potentials pairwise potentials t-links n-links [Boros & Hummer, 2002, Kolmogorov & Zabih, 2004] E(L) can be minimized by s-t graph cuts Submodularity (“convexity”) B. Leibe

Topics of This Lecture Solving MRFs with Graph Cuts Graph cuts for image segmentation s-t mincut algorithm Graph construction Extension to non-binary case Applications Source Sink v1 v2 2 5 9 4 1 B. Leibe

Recap: Simple Image Segmentation Model MRF Structure Example: simple energy function Smoothness term: penalty cij if neighboring labels disagree. Observation term: penalty ci if label and observation disagree. Noisy observations Observation process “True” image content “Smoothness constraints” Observation Smoothness ("Potts model") B. Leibe Image source: C. Bishop, 2006

Converting an MRF into an s-t Graph Conversion: Energy: Unary potentials are straightforward to set. How? Just insert xi = 1 and xi = 0 into the unary terms above... Source Sink xi xj B. Leibe

Converting an MRF into an s-t Graph Conversion: Energy: Unary potentials are straightforward to set. How? Pairwise potentials are more tricky, since we don’t know xi! Trick: the pairwise energy only has an influence if xi  xj. (Only!) in this case, the cut will go through the edge {xi,xj}. Source Sink xi xj B. Leibe

Example: Graph Construction Source (0) a1 a2 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

Example: Graph Construction Source (0) 2 a1 a2 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

Example: Graph Construction Source (0) 2 a1 a2 5 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

Example: Graph Construction Source (0) 2 9 a1 a2 5 4 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

Example: Graph Construction Source (0) 2 9 1 a1 a2 5 4 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

Example: Graph Construction Source (0) 2 9 1 a1 a2 2 5 4 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

Example: Graph Construction Source (0) 2 9 1 a1 a2 2 5 4 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

Example: Graph Construction Source (0) 2 9 Cost of cut = 11 1 a1 a2 a1 = 1 a2 = 1 2 E (1,1) = 11 5 4 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

Example: Graph Construction Source (0) 2 9 Cost of cut = 7 1 a1 a2 a1 = 1 a2 = 0 2 E (1,0) = 7 5 4 Sink (1) Slide credit: Pushmeet Kohli B. Leibe

How Does the Code Look Like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) Source (0) Sink (1) Slide credit: Pushmeet Kohli B. Leibe

How Does the Code Look Like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) Source (0) bgCost(a1) bgCost(a2) a1 a2 fgCost(a1) fgCost(a2) Sink (1) Slide credit: Pushmeet Kohli B. Leibe

How Does the Code Look Like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) Source (0) bgCost(a1) bgCost(a2) cost(p,q) a1 a2 fgCost(a1) fgCost(a2) Sink (1) Slide credit: Pushmeet Kohli B. Leibe

How Does the Code Look Like? Graph *g; For all pixels p /* Add a node to the graph */ nodeID(p) = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeID(p), fgCost(p), bgCost(p)); end for all adjacent pixels p,q add_weights(nodeID(p), nodeID(q), cost); g->compute_maxflow(); label_p = g->is_connected_to_source(nodeID(p)); // is the label of pixel p (0 or 1) Source (0) bgCost(a1) bgCost(a2) cost(p,q) a1 a2 fgCost(a1) fgCost(a2) Sink (1) a1 = bg a2 = fg Slide credit: Pushmeet Kohli B. Leibe

Topics of This Lecture Solving MRFs with Graph Cuts Graph cuts for image segmentation s-t mincut algorithm Graph construction Extension to non-binary case Applications other labels  B. Leibe

Dealing with Non-Binary Cases Limitation to binary energies is often a nuisance.  E.g. binary segmentation only… We would like to solve also multi-label problems. The bad news: Problem is NP-hard with 3 or more labels! There exist some approximation algorithms which extend graph cuts to the multi-label case: -Expansion -Swap They are no longer guaranteed to return the globally optimal result. But -Expansion has a guaranteed approximation quality (2-approx) and converges in a few iterations. B. Leibe

-Expansion Move Basic idea: Break multi-way cut computation into a sequence of binary s-t cuts. other labels  Slide credit: Yuri Boykov B. Leibe

-Expansion Algorithm Start with any initial solution For each label “” in any (e.g. random) order: Compute optimal -expansion move (s-t graph cuts). Decline the move if there is no energy decrease. Stop when no expansion move would decrease energy. Slide credit: Yuri Boykov B. Leibe

Example: Stereo Vision Depth map ground truth Depth map Original pair of “stereo” images Slide credit: Yuri Boykov B. Leibe

-Expansion Moves In each -expansion a given label “” grabs space from other labels initial solution -expansion -expansion -expansion -expansion -expansion -expansion -expansion For each move, we choose the expansion that gives the largest decrease in the energy:  binary optimization problem Slide credit: Yuri Boykov B. Leibe

Topics of This Lecture Solving MRFs with Graph Cuts Graph cuts for image segmentation s-t mincut algorithm Extension to non-binary case Applications B. Leibe

GraphCut Applications: “GrabCut” Interactive Image Segmentation [Boykov & Jolly, ICCV’01] Rough region cues sufficient Segmentation boundary can be extracted from edges Procedure User marks foreground and background regions with a brush. This is used to create an initial segmentation which can then be corrected by additional brush strokes. Add interactive image segmentation slide Additional segmentation cues User segmentation cues Slide credit: Matthieu Bray

Global optimum of the energy GrabCut: Data Model Obtained from interactive user input User marks foreground and background regions with a brush Alternatively, user can specify a bounding box Foreground color Background color Global optimum of the energy Slide credit: Carsten Rother B. Leibe

Color model (Mixture of Gaussians) Energy after each iteration Iterated Graph Cuts Foreground Background G R Foreground & Background Background G R Color model (Mixture of Gaussians) 1 2 3 4 Result Energy after each iteration Slide credit: Carsten Rother B. Leibe

GrabCut: Example Results This is included in the newest version of MS Office! B. Leibe Image source: Carsten Rother

Applications: Interactive 3D Segmentation Slide credit: Yuri Boykov B. Leibe [Y. Boykov, V. Kolmogorov, ICCV’03]

Bayesian Networks What we’ve learned so far… We know they are directed graphical models. Their joint probability factorizes into conditional probabilities, We know how to convert them into undirected graphs. We know how to perform inference for them. Sum/Max-Product BP for exact inference in (poly)tree-shaped BNs. Loopy BP for approximate inference in arbitrary BNs. Junction Tree algorithm for converting arbitrary BNs into trees. But what are they actually good for? How do we apply them in practice? And how do we learn their parameters? B. Leibe Image source: C. Bishop, 2006

References and Further Reading A gentle introduction to Graph Cuts can be found in the following paper: Y. Boykov, O. Veksler, Graph Cuts in Vision and Graphics: Theories and Applications. In Handbook of Mathematical Models in Computer Vision, edited by N. Paragios, Y. Chen and O. Faugeras, Springer, 2006. Try the GraphCut implementation at http://pub.ist.ac.at/~vnk/software.html B. Leibe