Schedule Introduction Models: small cliques and special potentials Tea break Inference: Relaxation techniques: LP, Lagrangian, Dual Decomposition Models: global potentials and global parameters + discussion

MRF with global potential GrabCut model [Rother et. al. ‘04] F i = -log Pr(z i |θ F )B i = -log Pr(z i |θ B ) Background Foreground G R θ F/B Gaussian Mixture models E(x,θ F,θ B ) = Problem: for unknown x,θ F,θ B the optimization is NP-hard! [Vicente et al. ‘09] Image z Output x ∑ F i (θ F )x i + B i (θ B )(1-x i )+ ∑ |x i -x j | i,j Є N i θ F/B

GrabCut: Iterated Graph Cuts [Rother et al. Siggraph ‘04] Learning of the colour distributions Graph cut to infer segmentation F x min E(x, θ F, θ B ) θF,θBθF,θB B Most systems with global variables work like that e.g. [ObjCut Kumar et. al. ‘05, PoseCut Bray et al. ’06, LayoutCRF Winn et al. ’06] θ F/B

GrabCut: Iterated Graph Cuts Energy after each Iteration Result Guaranteed to converge

Colour Model Background Foreground & Background G R Background Foreground G R Iterated graph cut

Optimizing over θ’s help after convergence [GrabCut ‘04] no iteration [Boykov&Jolly ‘01] Input after convergence [GrabCut ‘04]

Global optimality? GrabCut (local optimum) Global Optimum [Vicente et al. ‘09] Is it a problem of the optimization or the model?

… first attempt to solve it [Lempisky et al. ECCV ‘08] Model a discrete subset: w F = (1,1,0,1,0,0,0,0); w B = (1,0,0,0,0,0,0,1) #solutions: w F *w B = 2 16 Global Optimum: Exhaustive Search: Graph Cuts Branch-and-MinCut: ~ Graph Cuts (depends on image) E(x,θ F,θ B )= ∑ F i (θ F )x i + B i (θ B )(1-x i ) + ∑ w ij |x i -x j | 8 Gaussians whole image G R w F,B

Branch-and-MinCut w F = (1,1,1,1,0,0,0,0) w B = (1,0,1,1,0,1,0,0) w F = (0,0,*,*,*,*,*,*) w B = (0,*,*,*,*,*,*,*) w F = (*,*,*,*,*,*,*,*) w B = (*,*,*,*,*,*,*,*) min E(x,w F,w B ) = min [ ∑ F i (w F )x i + B i (w B )(1-x i ) + ∑ w ij (x i,x j ) ] ≥ min [∑ min F i (w F )x i + min B i (w B )(1-x i ) + ∑ w ij (x i,x j )] x,w F,w B x wBwB wFwF

Results … E=-618 GrabCut E=-624 (speed-up 481) Branch-and-MinCut E=-593 GrabCut E=-584 (speed-up 141) Branch-and-MinCut

Object Recognition & Segmentation Given exemplar shapes: Test: Speed-up ~900; accuracy 98.8% |w| ~ min E(x,w) with: w = Templates x Position w

… second attempt to solve it [Vicente et al. ICCV ‘09] Eliminate global color model θ F,θ B : θF,θBθF,θB E’(x) = min E(x,θ F,θ B )

Eliminate color model E(x,θ F,θ B )= ∑ F i (θ F )x i + B i (θ B )(1-x i ) + ∑ w ij |x i -x j | Image histogram k given x K = 16 3 k θBθB θ F є [0,1] K is a distributions (∑ θ F = 1) (background same) θFθF k background distribution foreground distribution Optimal θ F/B given by empirical histograms: θ F = n F k /n F n F = ∑x i #fgd. pixel n F = ∑x i #fgd. pixel in bin k Optimal θ F/B given by empirical histograms: θ F = n F k /n F n F = ∑x i #fgd. pixel n F = ∑x i #fgd. pixel in bin k k k Image discretized in bins K i Є B k K K

Eliminate color model E’(x)= g(n F ) + ∑ h k (n F ) + ∑ w ij |x i -x j | with n F = ∑x i, n F = ∑x i min θ F,θ B k k E(x,θ F,θ B )= ∑ F i (θ F )x i + B i (θ B )(1-x i ) + ∑ w ij |x i -x j | k nFnF 0n/2n g Prefers “equal area” segmentation Each color either fore- or background hkhk nFnF 0 max k convex concave i Є B k i E(x,θ F,θ B )= ∑ -n F k log θ F k -n B k log θ B k + ∑ w ij |x i -x j | k (θ F = n F k /n F ) K

How to optimize … Dual Decomposition E(x)= g(n F ) + ∑ h k (n F k ) + ∑ w ij |x i -x j | E 1 (x)E 2 (x) min E(x) = min [ E 1 (x) + y T x + E 2 (x) – y T x ] ≥ min [ E 1 (x’) + y T x’ ] + min [E 2 (x) – y T x] =: L(y) Goal: - maximize concave function L(y) using sub-gradient - no guarantees on E (NP-hard) L(y) E(x’) k x’xxx “paramteric maxflow” gives optimal y=λ1 efficiently [Vicente et al. ICCV ‘09] Simple (no MRF) Robust P n Potts

Some results… Global optimum in 61% of cases (GrabCut database) InputGrabCut Global Optimum (DD) Local Optimum (DD)

Insights on the GrabCut model g 0.4 g0.3 g1.5 g hkhk nFnF Each color either fore- or background 0 max k nFnF 0n/2n g Prefers “equal area” segmentation concave convex

Relationship to Soft P n Potts Image Pairwise CRF only TextonBoost [Shotton et al. ‘06] robust P n Potts [Kohli et al ‘08] One super- pixelization another super- pixelization GrabCut: cluster all colors together Just different type of clustering:

Marginal Probability Field (MPF) What is the prior of a MAP-MRF solution: [Woodford et. al. ICCV ‘09] Training image: 60% black, 40% white MRF is a bad prior since ignores shape of the (feature) distribution ! MAP: prior(x) = 0.6 = Others less likely : prior(x) = 0.6 * 0.4 = Introduce a global term, which controls global statistic

Marginal Probability Field (MPF) [Woodford et. al. ICCV ‘09] Optimization done with Dual Decomposition (different ones) max 0 0 MRF True energy

Examples Segmentation: In-painting: Pairwise MRF – Increase Prior strength Ground truth Noisy input Global gradient prior

Schedule Introduction Models: small cliques and special potentials Tea break Inference: Relaxation techniques: LP, Lagrangian, Dual Decomposition Models: global potentials and global parameters + discussion

Open Questions Many exciting future directions – Exploiting latest ideas for applications (object recognition etc.) – Many other higher-order cliques: Topology, Grammars, etc. (this conference). Comparison of inference techniques needed: – Factor graph message passing vs. transformation vs. LP relaxation? Learning higher order Random Fields