HCI Project : An Iterative Optimization Approach for Unified Image Segmentation and Matting

Abstract Extracting a matte by previous approaches require the input image to be pre-segmented into three regions (trimap). This pre-segmentation based approach fails for images with large portions of semi-transparent foreground. In this paper we combine the segmentation and matting problem together and propose a unified optimization approach based on Belief Propagation.

Introduction The observed image I(z) (z = (x, y)) is modeled as a linear combination of foreground image F(z) and background image B(z) by an alpha map: I(z) = αzF(z) + (1 − αz)B(z) Image Matting estimating an opacity (alpha value) and foreground and background colors for each pixel in the image.

Limitations of a Trimap To generate good mattes, all these approaches require the user to ”carefully” specify the trimap. it is almost impossible to manually create an optimal trimap.

Limitations of a Trimap (cont.) Automatically generated trimaps based on the binary segmentation result is non-optimal, since it always has uniform thickness regardless of local image characteristics.

MRF Construction Each pixel in and are treated as a node in the MRF Minimize the total energy of the following function ：How well the estimated alpha value , and foreground and background color for fit with the actual color ：The smoothness energy which penalizes inconsistent alpha value changes between two neighbors and

Markov Random Field 上層 – 原值 下層 – 估計值

Predefined arguments Discretize the possible alpha value to 25 levels between 0 and 1,denoted as , k=1,…,25 Each level corresponds to a possible state for a node in the MRF The local neighborhood area is defined to have a radius of r=20

in detail Compute the likelihood of each alpha level as The set of valid foreground samples, are then weighted by their uncertainty and distance, by

in detail The smoothness cost is defined as

Belief Propagation Optimization Use loopy belief propagation (BP) to solve problem Finding a labeling with minimum energy corresponds to the MAP estimation problem It works by passing messages along links in the constructed path

BP Algorithm(1) In each iteration, new messages are computed for each possible state H (p) \ q denotes the neighbors of p other than q c is a normalization factor

BP Algorithm(2) After T iterations a belief vector is computed for each node The state the maximizes at each node is selected as the estimated level

BP Algorithm(3) If =1, set the color as a new foreground sample If =0, set the color as a new background sample Otherwise, choose the pair of foreground and background colors from the group of samples

BP Algorithm(4) Then, the uncertainty value u(p) is updated as and are weights for the selected pair of foreground and background samples

Iterative Belief Propagation for Image Matting

Extension to Video is small as definite foreground

Result and Comparisons Propose approach Bayesian

Extracted foreground and novel background

Foreground and background is similar Not work well

Start with initial trimap Rough trimap the user created Bayesian Proposed approach

Summary and conclusion Proposed a approach to solve image matting problem combines the problems of segmentation and matting into a unified formula Does not require a well specified trimap