Image Matting and Its Applications Chen-Yu Tseng Advisor: Sheng-Jyh Wang

Image Matting A process to extract foreground objects from an image, along with an alpha matte (the opacity of the foreground color) Input ImageAlpha MatteExtracted Foreground

Two Approaches of Image Matting Supervised Matting With User’s Guidance Unsupervised Matting Without User’s Guidance Input ImageUser’s Guidance e.g. Trimap: White  Foreground Black  Background Unknown  Gray

Two Schemes of Supervised Matting Propagation-based Scheme Infer Alpha Matte with Propagation through a Graphical Model A Global-based Approach Sampling-based Scheme Infer Alpha Matte with Some Color Samples A Local-based Approach Foreground Pixel Background Pixel Unknown Pixel Foreground Color Set Background Color Set Unknown Pixel

Propagation-based scheme - Matting Laplacian Approach A Graphical Model with Connectivity between Pixels The Connectivity Is Learned from the Image Structure Capability for Dealing with Both Supervised Matting (Inference Problem) Unsupervised Matting (Decomposition Problem) Foreground Pixel Background Pixel Unknown Pixel

Reference of Matting Laplacian Approach First proposed by Levin et al. for supervised matting (closed-form matting) A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp , Feb Extended to unsupervised matting (spectral matting) A. Levin, A. Rav-Acha, D. Lischinski. “Spectral Matting,” IEEE T. PAMI, vol. 30, no. 10, pp , Oct Extended to learning-based matting Y. Zheng and C. Kambhamettu. “Learning based digital matting,” In ICCV, pages 889–896, Extended to multi-layer matting D. Singaraju, R. Vidal. “Estimation of Alpha Mattes for Multiple Image Layers,” IEEE T. PAMI, vol. 33, no. 7, pp , July 2011.

Matting Laplacian Input Image Estimating Pair-wise Affinity Graphical Model Node: Image Pixels Edge: Affinity Supervised Matting Background Foreground Matting Laplacian Matrix: Recording the Connectivity between Pair of Pixels

Introduction of Graph Laplacian A Graph with Five Vertexes Vertex Index

Introduction of Graph Laplacian A Graph with Five Vertexes Vertex Index

Cutting Cost Function with Graph Laplacian Cost Function for Cutting Criterion Low-cost Assignment High-cost Assignment

Construction of Matting Laplacian Color-model-based Approach (Original) Estimating Affinity Based on Relative Color Distance Learning-based Approach (Extended) Learning Affinity Based on Image Structure

Construction of Matting Laplacian Color-model-based Approach Color Distribution Input Image A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp , Feb g r b

Construction of Matting Laplacian Learning-based Approach Learning Affinity among Local Pixels Linear Alpha-color Model for Single Pixel: Extending to a Local Patch q Assuming all Pixels Sharing the Same Linear Coefficient

Construction of Matting Laplacian Learning-based Approach Derived Linear Coefficient Rewritten Linear Model

Construction of Matting Laplacian Local Cost Function Input Image Patch q Local Linear Model

Construction of Matting Laplacian Local  Global Input Image Patch q

Supervised Matting (Closed-form Matting) Foreground Pixel Background Pixel Unknown Pixel Input Image ForegroundBackgroundUnknown Cost Function for Supervised Matting Affinity Cost Data Cost Optimal Solution

Experimental Results Input ImageAlpha MatteSynthesized Result

Unsupervised Matting (Spectral Matting) Solving Alpha Matte without User’s Guidance Procedures Decomposing Image into Several Matting Components Combining Matting Components into Alpha Matte

Spectral Clustering 1.L is symmetric and positive semi-definite. 2.The smallest eigenvalue of L is 0, the corresponding eigenvector is the constant one vector 1. 3.L has n non-negative, real-valued eigenvalues 0= λ 1 ≦ λ 2 ≦... ≦ λ n A Graph Example

Spectral Clustering & Matting Components Zero-Eigenvectors Binary Indicating Vectors Linear Transformation

Overview of Spectral Matting Input Image Smallest Eigenvectors Matting Components K-means Clustering & Linear Transformation Matting Laplacian

Spectral Clustering & K-means Input Image s-smallest Eigenvectors … Pixel i s-dimensional Space K-means Clustering

Generating Matting Components Smallest Eigenvectors Projection into Eigen Space K-means ………

Reconstructing Alpha Matte from Matting Components =++ Input Image Matting Components Selected Matting ComponentsAlpha Matte

Reconstructing Alpha Matte by Grouping Matting Components Matting cost function Alpha Matte Generation Evaluating All Grouping Hypothesis to Derive the Optimal Alpha Matte

Results by Levin et al.

Summary Constructing Matting Laplacian Solving Supervised Matting Problem Solving Unsupervised Matting Problem

Proposed Approaches Efficient Cell-based Framework for Reducing Computations Multi-scale Analysis Extended Applications (Depth Image Reconstruction) Input Image Reconstructed Depth Depth Reconstruction from Single Image Depth Reconstruction in Shape From Focus (SFF) Input ImageReconstructed Depth

Cell-based Framework Image Pixel-wise Data Distribution Cell-wise Data Distribution Conventional Matting Laplacian Cell-based Matting Laplacian Pixel-wise Affinity Cell-wise Affinity

Multi-scale Affinity Learning Image & Computation Patterns Pixel-based Approach Cell-based Approach

Multi-scale Affinity Learning … Finest Level Coarsest Level … Cell-based Graph

Results of Reconstructed Alpha Matte 1 st Rank 2 nd Rank (a) Grouping Results by Levin et al. (b) Grouping Results by Levin et al. with Coarse-to-fine Scheme. (c) Ours Input

Results (a) Input images(b) Levin’s result(c) Our result

Proposed Approaches Efficient Cell-based Framework for Reducing Computations Multi-scale Analysis Extended Applications (Depth Image Reconstruction) Input Image Reconstructed Depth Depth Reconstruction from Single Image Depth Reconstruction in Shape From Focus (SFF) Input ImageReconstructed Depth

Depth Reconstruction in Shape From Focus (SFF) Optical Direction Multi-focus Image Sequence Optical Direction Focus Value W1W1 W2W2 W2W2 W1W1

Low-SNR Problem Spatially Varying Precision Low-texture  Low-SNR Leading Noisy Result Input Image Observation High- precision Low- precision

Proposed Maximum-a-posteriori Estimation Multi-focus Image Sequence Learning-based Graph Local Learning Inference Reconstructed Depth

Proposed Maximum-a-posteriori Estimation PosteriorLikelihoodPrior Local Observation with Spatial-varying Precision Learned from Image

Likelihood Model Input Observation Precision Result High- precision Low- precision PosteriorLikelihoodPrior Local Observation with Spatial-varying Precision

Prior Model PosteriorLikelihoodPrior Learning from Input Image Learning-based Graph Local Learning Multi-focus Image Sequence

Maximum-a-posteriori Estimation for Depth Reconstruction Input ImageObservationReconstructed Depth

Results of Shape from Focus Input Image M. Mahmood, 2012T. Aydin, 2008Ours S. Nayar, 1994

Conclusions Construction of Matting Laplacian Conventional Approach Multi-scale Cell-based Approach Supervised Matting Spectral Matting Depth Reconstruction