Pseudo-Bound Optimization for Binary Energies Presenter: Meng Tang Joint work with Ismail Ben AyedYuri Boykov 1 / 27

Labeling Problems in Computer Vision foreground selection Geometric model fitting StereoSemantic segmentationDenoising inpainting Binary label Multi-label 2 / 27

Energy Minimization for Labeling Problem foreground selection Semantic segmentation S * = arg S min E(S) s p = 1 (FG) or 0 (BG) s p = ‘sky’ or ‘road’ or ‘bike’ etc. 3 / 27

Basic Pairwise Energies  Common in computer vision Unary termPairwise term Example: interactive segmentation S * = arg S min E(S) e.g. Boros & Hammer Boykov & Jolly / 27  Submodular case: fast global solver (Graph Cuts ) Unary termPairwise term

More difficult energies High-order energies  Entropy minimization for image segments  Matching target distribution  Volume constraints  Convex shape prior Pairwise nonsubmodular energies  Curvature regularization  Segmentation with repulsion  Binary image deconvolution  e.t.c. Roof duality [Boros & Hammer. 2002] QPBO-mincut [Kolmogorov, Rother et al. 2007] TRWS, SRMP [Kolmogorov et al. 2006, 2014] Parallel ICM [Leordeanu et al. 2009] ……………. Region Competition[Zhu, Lee & Yuille. 1995] GrabCut [Rother et al. 2004] [Vicente et al. 2009] [Gould et al. 2011, 2012][Kohli et al. 2007, 2009] [Ayed et al. 2010, 2013][Gorelick et al. 2013, 2014] …………….. 5 / 27

Our framework (Pseudo-Bound Opt.) High-order energies  Entropy minimization for image segments  Matching target distribution  Volume constraints  Convex shape prior Pairwise nonsubmodular energies  Curvature regularization  Segmentation with repulsion  Binary image deconvolution  e.t.c. Roof duality [Boros & Hammer. 2002] QPBO-mincut [Kolmogorov, Rother et al. 2007] TRWS, SRMP [Kolmogorov et al. 2006, 2014] Parallel ICM [Leordeanu et al. 2009] ……………. Region Competition[Zhu, Lee & Yuille. 1995] GrabCut [Rother et al. 2004] [Vicente et al. 2009] [Gould et al. 2011, 2012][Kohli et al. 2007, 2009] [Ayed et al. 2010, 2013][Gorelick et al. 2013, 2014] …………….. 6 / 27

Example of high-order energy  With known appearance models θ 0,θ 1.  Appearance models can be optimized Boykov & Jolly GrabCut [Rother et al. 2004] 7 / 27 fixed variables

From model fitting to entropy optimization: entropy of intensities in entropy of intensities in Note: H(P|Q)  H(P) for any two distributions cross-entropyentropy [Delong et al, IJCV 2012] [Tang et al. ICCV 2013] common energy for categorical clustering, e.g. [Li et al. ICML’04] Decision Forest Classification, e.g. [Criminisi & Shotton. 2013] binary optimization mixed optimization high-order energy 8 / 27

Energy example: color entropy low entropyhigh entropy S S S S S S S S S S S S 9 / 27

Pseudo-bound optimization example: minimize our entropy-based energy E(S) 10 / 27

BCD could be seen as bound optimization for entropy 11 / 27 one standard approach: Block-Coordinate Descent (BCD) GrabCut [Rother et al. 2004] ≥ our entropy energy mixed var. energy

one standard approach: Block-Coordinate Descent (BCD) GrabCut [Rother et al. 2004] BCD could be seen as bound optimization for entropy ≥ our entropy energy mixed var. energy E(S)E(S) E ( S|θ 0,θ 1 ) t StSt S t+1 E ( S|θ 0,θ 1 ) t+1 12 / 27 E(St)E(St) E(S t+1 )

Converges to a local minimum E(S)E(S) StSt S t+1 E(St)E(St) E(S t+1 ) Bound optimization, in general A t+1 (S) At(S)At(S) 13 / 27 (Majorize-Minimize, Auxiliary Function, Surrogate Function)

Local minima examples (for GrabCut) E=2.37×10 6 E=2.41×10 6 E=1.39×10 6 E=1.410× / 27

This work: Pseudo-Bound Optimization solution space S StSt S t+1 At(S)At(S) E(S)E(S) E(St)E(St) (a) (b) (c) Make larger move! General framework for energy-minimization 15 / 27 E(S t+1 )

General form of Pseudo-Bounds Ƒ t (S, λ) = + λ R t (S) Pairwise Submodular Unary Bounding relaxation S λ = min s Ƒ t (S, λ) S t+1 = argmin S λ E(S λ ) Parametric Maxflow Update Iterate BoundsParametricPseudo-Cuts A t (S) (pPBC) n-links s t a cut t-link edge capacities depend linearly on λ. Parametric Max-flow Gallo et al Hochbaum et al Kolmogorov et al Bound 16 / 27

Specific pseudo-bounds Example 1 How to choose auxiliary and bound relaxation functions? Pairwise Submodular Unary Ƒ t (S, λ) = + λ R t (S) A t (S) Ƒ t (S,λ) + λ(|S|-|S t |) = E ( S|θ 0,θ 1 ) t 17 / 27

Specific pseudo-bounds Example 2 Pairwise Submodular Unary |S||S| |St||St| f(|S|) |V0||V0| High-order example: Soft volume constriants Ƒ t (S, λ) = + λ R t (S) A t (S) How to choose auxiliary and bound relaxation functions? 18 / 27

Specific pseudo-bounds Example 3 Ƒ t (S, λ) = + λ R t (S) A t (S) Pairwise Submodular Unary Pairwise example: Nonsubmodluar pairwise m pq s p s q, for m pq >0 How to choose auxiliary and bound relaxation functions? 19 / 27 Gorelick et al. in CVPR 2014

EXPERIMENTAL RESULTS 20 / 27

Experiment results (high-order) Interactive segmentation (entropy minimization) 21 / 27

Experiment results (high-order) Interactive segmentation (GrabCut database) 22 / 27 Rother et al Tang et al Vicente et al. 2009

Unsupervised binary segmentation – without prior (bounding box, appearance etc.) Experiment results (high-order) 23 / 27

Matching appearance distribution Input imageGround truthOur method 24 / 27

Experiment results (high-order) Matching color distribution 25 / 27 Ayed et al Gorelick et al. 2013

Experiment results (pairwise) Segmentation with curvature regularization 26 / 27 *Submodularization for Binary Pairwise Energies, Gorelick et al. in CVPR 2014 * * * *

Conclusion  Achieve state-of-the-art for many binary energy minimization problems  Several options of pseudo-bounds  BCD as in GrabCut is a bound optimization  Optimize pseudo-bounds efficiently with parametric maxflow  General optimization framework for high- order and pairwise binary energy minimization  Code available: / 27