Normalized Cut meets MRF Meng Tang1 Dmitrii Marin1 Ismail Ben Ayed2 Yuri Boykov1 1University of Western Ontario, Canada 2École de Technologie Supérieure, Canada Energy Formulation Optimization: Kernel Cut and Spectral Cut Normalized Cut (NC) MRF General optimization approach: bound optimization E(S) At (S) St St+1 At+1 (S) balanced regularization unary kernel or spectral bound clustering constraints to optimize bound At(S) we use move-making graph cuts [Boykov, Veksler, Zabih, 2001] Why MRF for NC semi-supervision is challenging How to incorporate group priors? weak edge alignment How many clusters? Kernel bound for NC (exact) 8 clusters Normalized Cut is concave: NC approach: cannot-link 3 clusters Lemma (concavity) Function e : R|Ω| → R is concave over region Sk > 0 given p.s.d. affinity matrix A := [Apq]. must-link [Chew et al. 2015] [Eriksson et al. 2010] [Yu & Shi 2004] [Arbelaez et al., 2010] 1st order Taylor expansion is an upper bound for NC: at(Sk) St St+1 at+1(Sk) e(Sk) our approach: (sparsity prior) NC + label cost NC+Potts NC+Potts+seeds NC + Pn Potts MRF MRF MRF MRF Why NC for MRF (equivalently kernel k-means [Dhillon t al., 2004]) Spectral bound for NC (approx.) ML term for 𝜃 𝑘 Step 1. low-rank (m) approximation Step 2. low-dim. isomeric Euclidean embedding for SVDs Probabilistic K-means [Kearns et al., UAI’97] bad clustering (overfitting) Model Fitting (e.g. GMM) (GrabCut without smoothness) Normalized Cut good clustering and low-rank m implying isometry MDS [Cox & Cox., 2000] Kernel PCA [Schölkopf et al., 1998] NC K-means unary spectral bound Summary of contribution new unary kernel and spectral bounds for NC can combine NC with any MRF constraints can combine MRF with balanced clustering MRF with features of any dimension (RGBD, RGBM, RGBXYM, deep,…) We show NC ≈ K-means via low-rank (m) approximation In contrast, K-means is a discretization heuristic in NC spectral relaxation [Shi & Malik 2000, Yu & Shi 2003] Spectral bound accuracy (depending on m) m = 3 K-means m = 10 NC Experiments: NC helps MRF Experiments: MRF helps NC poor scalability to high-dim GrabCut Kernel Cut Potts model improves edge alignment RGBD segmentation BSDS500 measure Covering PRI VOI Spectral Clustering 0.34 0.76 2.76 Our Kernel Cut 0.41 0.78 2.44 Our Spectral Cut 0.42 2.34 Spectral Clustering (no Potts) Our Kernel Cut (NC with Potts) Our Spectral Cut (NC with Potts) High dimension feature : RGBXYM (location XY, motion M) Label cost Robust Pn Potts using image tags (e.g. beach, car) to help clustering Image clustering with tags consistency prior Our Kernel Cut with increasing label cost (a)Video frames (b)Optical flow (c)Kernel Cut(RGBXY) (d)Kernel Cut(RGBXYM)