1School of CS&Eng The Hebrew University Spectral Matting A. Levin D. Lischinski and Y. Weiss. A Closed Form Solution to Natural Image Matting. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2006, New York A. Levin, A. Rav-Acha, D. Lischinski. Spectral Matting. Best paper award runner up. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Minneapolis, June 2007 A. Levin1,2, A. Rav-Acha1, D. Lischinski1. Spectral Matting. IEEE Trans. Pattern Analysis and Machine Intelligence, Oct 2008. 1School of CS&Eng The Hebrew University 2CSAIL MIT

Hard segmentation and matting compositing Source image Matte compositing

Previous approaches to segmentation and matting Input Hard output Matte output Unsupervised Spectral segmentation: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Zelnik and Perona 05 Tolliver and Miller 06

Previous approaches to segmentation and matting Input Hard output Matte output Unsupervised Supervised July and Boykov01 Rother et al 04 Li et al 04

Previous approaches to segmentation and matting Input Hard output Matte output ? Unsupervised Supervised Trimap interface: Bayesian Matting (Chuang et al 01) Poisson Matting (Sun et al 04) Random Walk (Grady et al 05) Scribbles interface: Wang&Cohen 05 Levin et al 06 Easy matting (Guan et al 06)

User guided interface Scribbles Trimap Matting result

Generalized compositing equation 2 layers compositing = x +

Generalized compositing equation 2 layers compositing = x + K layers compositing = x + Matting components

Generalized compositing equation K layers compositing = x + Matting components: “Sparse” layers- 0/1 for most image pixels

Unsupervised matting Input Automatically computed matting components

Building foreground object by simple components addition + + =

Spectral segmentation Spectral segmentation: Analyzing smallest eigenvectors of a graph Laplacian L E.g.: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Maila and shi 01 Zelnik and Perona 05 Tolliver and Miller 06

Problem Formulation = x + Assume a and b are constant   = x +     Assume a and b are constant in a small window

Derivation of the cost function

Derivation

The matting Laplacian semidefinite sparse matrix local function of the image:

The matting affinity  

The matting affinity       Input Color Distribution

Matting and spectral segmentation Typical affinity function Matting affinity function

Eigenvectors of input image Smallest eigenvectors

Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Null Binary indicating vectors Laplacian matrix

Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Smallest eigenvectors- class indicators only up to linear transformation Zero eigenvectors Binary indicating vectors Laplacian matrix Smallest eigenvectors Linear transformation

From eigenvectors to matting components linear transformation

From eigenvectors to matting components Sparsity of matting components Minimize

From eigenvectors to matting components Minimize Newton’s method with initialization

From eigenvectors to matting components 1) Initialization: projection of hard segments Smallest eigenvectors K-means Projection into eigs space 2) Non linear optimization for sparse components

Extracted Matting Components

Brief Summary Construct Matting Laplacian Smallest eigenvectors Linear Transformation Matting components

Grouping Components + + =

Grouping Components Unsupervised matting User-guided matting + + =   Complete foreground matte + + = Unsupervised matting User-guided matting

Unsupervised matting Matting cost function Hypothesis:       Hypothesis: Generate indicating vector b  

Unsupervised matting results

User-guided matting Graph cut method Energy function Unary term Pairwise term Constrained components  

Components with the scribble interface Components (our approach) Levin et al cvpr06 Wang&Cohen 05 Random Walk Poisson

Components with the scribble interface Components (our approach) Levin et al cvpr06 Wang&Cohen 05 Random Walk Poisson

Direct component picking interface Building foreground object by simple components addition + + =

Results

Quantitative evaluation

Spectral matting versus obtaining trimaps from a hard segmentation

Limitations Number of eigenvectors Ground truth matte Matte from

Limitations Number of matting components

Conclusion Derived analogy between hard spectral segmentation to image matting Automatically extract matting components from eigenvectors Automate matte extraction process and suggest new modes of user interaction