Presenter ： Kuang-Jui Hsu Date ： 2011/5/12(Tues.)

Outline Introduction Image segmentation by graph cut The GrabCut segmentation alog. Transparency Results and conclusion

Introduction What is the GrabCuts?? The user drags a rectangle loosely around an object. The object is then extracted automatically.

Previous approaches to interactive matting Magic Wand Starts with a user-specified point or region Compute a region of connected pixels All the selected pixels fall within some adjustable tolerance of the colour statistics of the specified region.

Previous approaches to interactive matting Intelligent Scissors Allow a user to choose a minimum cost contour The minimum cost path from the cursor position back to the last seed If the computed path deviates from the desired one, addition user-specified seed points are needed.

Previous approaches to interactive matting Bayes Matting Models colour distribution probabilistically The user specifies a trimap T = { } Alpha values are computed over the region

Previous approaches to interactive matting Knockout 2 A proprietary plug-in for Photoshop which is driven from a user-defined trimap.

Previous approaches to interactive matting Graph Cut A powerful optimisation technique that can be used in a setting similar to Bayes Matting, including trimaps and probabilistic

Proposed system: GrabCut 1.Obtain a “hard segmentation using iterative graph cut. 2. Use by border matting, for the boundary. 3. Full transparency, other than at the border.

Image segmentation by graph Cut Make two enhancements to the graph cuts mechanism Iterative estimation Incomplete labelling This allows GrabCut to put light load on the user The user is indicating a region of background, and is free of any need to make foreground region Ideally, a matting tool should be able to produce continuous alpha values

Image segmentation Given an initial trimap T The image is an array z of grey values, indexed by the index n The segmentation of the image is expressed as an array of opacity values at each pixel. General, but for hard segmentation, with 0 for background and with 1 for foreground.

Image segmentation The parameter describe image foreground and background grey-level distribution. background foregroundhistograms of grey values The segmentation task is to infer the unknown opacity variables from the given image data z and the model

Segmentation by energy minimisation Energy function: Date term: Smoothness term: the indicator function taking values 0,1 the set of pairs of neighboring The Euclidean distance of neighboring pixels

Segmentation by energy minimisation denotes expectation over an image sample The choice of β ensures that the exponential term switches appropriately between high and low contrast Obtained as 50 By optimizing performance against truth ground over a training set of 15 images

Segmentation by energy minimisation Energy function: Minimisation is done using a standard minimum cut algorithm

The GrabCut segmentation algorithm This section will outline three development: 1.The monochrome image model is replaced for colour by a Gaussian Mixture Model(GMM) in place of histograms 2. Use iterative procedure that alternate between estimation and parameter learning 3. Users are relaxed by allowing incomplete labelling - only specify for the trimap

Colour data modeling Consider the pixels in RGB colour space Impractical to construct adequate colour space histograms Use 2 GMMs, one for the background and the other for the foreground. Taken to be a full-covariance Gaussian mixture with K components( typically k = 5) An addition vector k = is introduced, with, assigning to each pixel

Colour data modeling Energy function: Date term Smoothness term Gaussian probability distributionMixture weighting corfficients

Colour data modeling Gaussian probability distribution Using the multivariate Gaussian probability distribution

Colour data modeling For example, give a GMM component k in the foreground The mean μ(α, k) and Σ (α, k) is computed from the F(k) Smoothness term: Almost unchanged expect for the contrast term by using Euclidean distance in colour space

Segmentation by iterative energy minimization

Practical benefits of iterative minimisatation This is apparent in two ways: The degree of user editing required, after initialisation and optimisation, is reduced. The initial interaction can be simpler, for example by allowing incomplete labelling by the user.

Practical benefits of iterative minimisatation This is apparent in two ways: The degree of user editing required, after initialisation and optimisation, is reduced. The initial interaction can be simpler, for example by allowing incomplete labelling by the user.

User interaction and incomplete trimaps Incomplete trimaps Further user editing

Incomplete trimaps Incomplete labelling becomes more feasible and more convenient for users. Only specify the background, and let the foreground Iterative minimiastion deals with this incompleteness by allowing provisional labels on some pixels (in the foreground ) which can subsequently retracted In this paper, the initial is determined by the user as a strip of pixels around the outside of marketed rectangle

Segmentation by iterative energy minimization alog.

Transparency A matting tool should be able to produce continuous alpha values Hard segmentation can be augment by “border matting” In border matting, the full transparency is allowed in a narrow strip around the hard segmentation Deal with the problem of matting in the presence of blur and mixed along smooth object boundaries The technical issue: Estimating an alpha value for the strip without generating artefacts Recovering the foreground colour, free of colour bleeding form the backound

Border matting Begin with a closed contour C, obtained by fitting a polyline to the boundary The yellow one A new trimap is computed is the set of pixels in a ribbon of width pixels either side of C The goal is to compute the map, and in order to do this robustly, a strong model is assumed for the shape of α- profile with With two important additions : 1.Regularisation to enhance the quality of estimated α-map 2.A dynamic programming (DP) algorithm for estimating α throughout

Border matting Let t = 1, …, T be a parameterization of contour C, periodic with period T, as curve is cloesed. An index t(n) is assigned to each pixel t(n)t(n) The α-profile is taken to be a soft step- function g: Distance from pixel n to contour C Determine the centre and width respectively of the transition from 0 to 1 in α-profile Assume that all pixels with the same index t share values of the parameter

Border matting Parameter values are estimated by minimizing the following energy function using DP over t : Where is a smoothing regularizer: Its role is to encourage α-values to vary smooth as t increase, along curve C If the contour C is closed, minimization cannot be done exactly using single-pass DP, and approximate by using two complete passes of DP, assuming that the first pass gives the optimal profile for t = T / 2 Date term: denotes a Gaussian probability density for z with mean μ and covariance Σ

Border matting Date term: : size pixels centred on on the boundary C at t

Foreground estimation Estimate foreground pixel colour without bleeding in from the background of the source image. Bleeding occur with Bayes matting because of the probabilistic algorithm used which cannon do so precisely Avoid this by stealing pixels from the foreground itself

Foreground estimation First, the bayes matte algorithm is applied to obtain an estimate of foreground colour on a pixel Second, from the neighbourhood as defined above, the pixel colour that is mose similar to is stolen to form the foreground colour Finally, the combined results of border matting, using both regularised alpha computation and foreground stealing

Foreground estimation result

Results and conclusion Regions of low contrast at the foreground to background

Results and conclusion Camouflage, in which the true foreground and background distributions overlap partially in the colour space

Results and conclusion Background material inside the user rectangle happens not to adequately represented in the background

Results and conclusion Demonstrates that the border matting method can cope with moderately difficult alpha mattes

Results and conclusion For difficult alpha mattes, the matting brush is needed

Results and conclusion A target rectangle of size pixels requires 0.9 sec for initial segmentation and 0.12 sec after each brush stroke on a 2.5 GHz CPU with 512 MB RAM