Graph Algorithms for Vision Amy Gale November 5, 2002

Energy Minimization  What is “energy” in this context?  What are some “classic” methods for energy minimization?  Huh? Graphs?  What are some graph-based methods for energy minimization?

Dissecting the Energy Function Energy of labeling f = data term+ smoothness term Cost of giving pixel p label f(p) given its current label. Intuition: penalty for label that differs from observed behavior of pixel. Cost of giving pixel p label f(p) given that its neighbor(s) have label(s) f(q). Intuition: penalty for differing from nearby pixels: we expect piecewise constant labels.

Data Term for Stereo  f(p) is disparity for pixel p, should be integer-valued even if actual disparity is not.  Data term should evaluate the difference between I(p) and the best interpolated value “near” I´(p+f(p)). I p I´I´ p+f(p) I´I´ I f ?? 1 2 0

Answers to questions you might not have asked yet  is the “regularization parameter”: it allows us to control the relative importance of smoothness term vs. data term.  N is a set of ordered pairs that we can define according to the neighbors we want to consider important: 4-neighborhood, 8- neighborhood, etc.

Some Specific Energy Models  Potts Model  Ising Model: Potts Model with two possible labels

Energy is Not Desirable  E(f) represents combined data and smoothness conflicts, so we want to minimize it.  Computer science already has some energy minimization algorithms lying around: Metropolis Algorithm. Simulated Annealing.

The Metropolis Algorithm 1.Start with some f. 2.Generate a random change to get f´ (for an image: change a single pixel label). 3.Compute  E = E(f´) – E(f). 4.If  E <0, set f = f´, otherwise set f = f´ with probability e -  E/T. 5.Go to step 2.

Metropolis: the role of T  T is a parameter to the algorithm.  When T is high, effectively a random search.  When T is low, can easily get stuck in local minima.  Not a great tradeoff either way.  Plus, result is sensitive to initial estimated f.

Simulated Annealing  Simply the Metropolis Algorithm with decreasing T.  Can be proved to find the global minimum if T is decreased “slowly enough”.  A worthwhile vision algorithm?

s t ,1 100,1 s t 1, ,99 100,100 s t 100,1 100, ,1 s t 1 100,100 Graphs  G = (V,E).  Relevant algorithm here: min cut (= max flow) between source s and sink t on an undirected graph. s t 1 100

Energy Minimization by Graph Cuts  Given points P want to build G where cuts in G are related to labelings of P. Labeling = cut Pixel = vertex Label = special vertex (s,t) Edge…?

Building the Graph: Ising Model pixel vertex n-link w(p,q) = ab cds t d-link w(p,s) = c 1 (p) w(p,t) = c 0 (p) label vertex cut cost = c 1 (a) + c 1 (c) + c 1 (d) + c 0 (b) + 2

Cost of a Cut in this Graph  Cut partitions graph vertices into S and T.  Cost of cut is cost of edges between S and T.

So Graph Cuts are Good Things (and that’s that?)  Ising Model allows two possible labels, which isn’t enough for any interesting/useful problem.  In general, the Potts model allows N possible labels, so what can we do? Multi-way Cut? This is NP-hard. By reduction…so is minimizing Potts energy at all! Need some new approach.

Approximation Algorithms  Sometimes our best option in the presence of NP-hardness.  Recall we can minimize the 2-label problem quickly, how can we leverage this?                Now choose 2 new labels and repeat…

 Swap Moves DEFINITION: an  swap move is a reallocation of some set of pixels between  and .  What happens in a single  swap move : To pixel labels? To overall energy?  What happens when we run to convergence?

How do we do this with graph cuts?  For an  swap, find min-cut on the following graph: (wlog) s =  -vertex, t =  -vertex V = {s,t}  {p: f(p)  {  }}(f current labeling)  Convention varies, authors in field (Zabih, Kolmogorov et al) say a cut gives label  to pixel p if it SEVERS the edge ( ,p).

Example (with a nasty surprise) Say we have pixels p 1, p 2, p 3 and possible labels  d(  ) = d(  ) = k / 2 and d(  ) = k. cp1p1 p2p2 p3p3  0kk  k0k  220 Initially:  E(f) = k  swap?   p1p1 p2p2 0 0+k/2 k/2 k+k k no change  swap?   p2p2 p3p3 0+k/2 0+0 k/2 k+0 2+k   swap?  p1p1 p3p3 0+k/2 k+k/2 2+k/2 no change  swap?   p2p2 p3p3 0+k/2 0 k/2 k 2+k  swap?   p1p1 p2p2 0 0+k/2 k/2 k+k k no change  is swap move minimum with E(f) = k BUT  has E(f) = 4 !!!  swap?  p1p1 p3p3 0+k/2 k+k/2 2+k/2 

 Swap Algorithm  Start with arbitrary f  Set change = 0  For each label pair  : find lowest-energy  -  swap f´ using graph cut if E(f´) < E(f) set f = f´ and change = 1  If change == 1 go to step 2

 swap is not a c-approximation algorithm for any c  Because the k in the last example could be anything at all…  Is there something similar we can do?      Now choose a new label and repeat…          

 -Expansion Moves DEFINITION: an  expansion move is a relabeling of some set of pixels to   Intuition: let label  compete against the collection  of all other labels.

Setting up the Graph  Two label vertices, wlog let s correspond to  and t to .  A pixel vertex for every pixel in the image.

Setting up the graph, ctd  Need some extra nodes and some constraints.  For  -expansion, d must be a metric: 1. d(  ) = 0   2. d(  d(  3. d(  d(  d(   Now add a gadget between p,q if (p,q)  N and f(q)  f(p)

Setting up the Graph: Example   p1p1 p3p3 p2p2 a 12 p4p4 a 34 c  (p 1 )c  (p 2 )c  (p 3 )c  (p 4 ) f(p 1 ) =  f(p 2 ) = f(p 3 ) =  f(p 4 ) =  d(  )  c  (p 3 )c  (p 2 )c  (p 1 ) d(  )d(  ) 0 d(  )d(  )

 -expansion Algorithm  Start with arbitrary f  Set change = 0  For each label  : find lowest-energy  -expansion f´ using graph cut if E(f´) < E(f) set f = f´ and change = 1  If change == 1 go to step 2

Optimality of  -expansion Let Let f* be the global optimum solution and f´ be the solution found by  -expansion. THEOREM: E(f´)  2cE(f*)

Some Results  -expansion Simulated Annealing (started from solution given by yet another algorithm, otherwise results would be much much worse) Ground Truth