1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978

2 Outline The matting problem Previous work New approaches: –The iterative approach Jue Wang, Michael F.Cohen –Closed form solution Anat Levin, Dani Lischinski,Yair Weiss Comparison and summary Bonus?

3 Outline The matting problem Previous work New approaches: –The iterative approach Jue Wang, Michael F.Cohen –Closed form solution Anat Levin, Dani Lischinski,Yair Weiss Comparison and summary Bonus?

4 The matting problem - Motivation  Image and video editing New backgroundComposite image

5 The matting problem - Motivation  Image and video editing Input imageNew image

6 The matting problem –The separation of an image I into I.Foreground object image F II.Background image B III.Alpha matte α – the opacity –Problem: extract F, B, α from image hairfur

7 Why is matting challenging? Under constrained problem: One equation, 3 unknowns  We need to constrain the problem!

8 Outline The matting problem Previous work New approaches: –The iterative approach Jue Wang, Michael F.Cohen –Closed form solution Anat Levin, Dani Lischinski,Yair Weiss Comparison and summary Bonus?

9 Previous work Two types: Known background Natural image matting matting

10 Known Background Blue screen Matting Still under-constrained –Solution: make more assumptions “Foreground contains no blue” Other foreground distribution assumption… Use two different backgrounds Main flaw: need to know the background… Blue backgroundComposite image

11 Natural Image Matting The assumptions: –Smoothness of the alpha matte –GMM for the Background and Foreground colors Initial estimate: trimap provided by the user Input imageTrimap Background Foreground Unknown

12 Natural Image Matting The algorithms framework: –Estimate F, B distributions from close pixels –Find best α by some method

13 Knockout –Extrapolate F,B from close neighborhood –Estimate α from calculated F, B values

14 Bayesian –Estimate F, B distributions in area –Find best α matching distributions

15 Bayesian –P(F), P(B) from image samples –P(C|F,B,α) using a distribution for C

16 Natural Image Matting Main flaw: Accurate trimap required Tedious to provide manually Hard to extract automatically  In particular, not feasible to videos Binary segmentation Adding unknown region Input imageTrimap

17 Great. So let’s get started…

18 Outline The matting problem Previous work New approaches: –The iterative approach Jue Wang, Michael F.Cohen –Closed form solution Anat Levin, Dani Lischinski,Yair Weiss Comparison and summary Bonus?

19 New Approach to Matting Trimap reduces to scribbles Two new methods –Iterative optimization approach Heuristic algorithmic optimization –A closed form solution Mathematical approach Trimap Scribbles

20 Iterative optimization approach Jue Wang Michael F. Cohen

21 Iterative approach

22 Iterative approach Score: fit to image data + alpha matte smoothness Iteratively propagating estimated results.

23 Iterative optimization - outline Initialize “work pixels” from scribbles Repeatedly: Expand work pixels Find best alpha matte Stop when finished

24 Initialization Introducing: –u i - uncertainty variable –U c – work pixels u i = 0 α = 0 U c = {user scribbles} u i = 0 α = 1 u i = 1 α = 0.5

25 Optimization U c = {user scribbles + 15 pixel radius} Our goal: find α matte for U c that minimizes the energy - Data Smoothness

26 VdVd Score for α p = α N Possible values for FN Possible values for B Image color I p

27 VdVd Fit measure of α p to I p Score for α p = α : F i, B j – possible values for F, B in the pixel w F i, w B j – corresponding weights

28 VdVd F Samples B Samples F i, B j – possible values for F, B in the pixel w F i, w B j – corresponding weights p α = 0.9 u = 0.2 α = 0.8 u = 0.3 α = 0.4 u = 0.5 α = 0.4 u = 0.4 α = 0.2 u = 0.3 α = 0.3 u = 0.3 α = 0.5 u = 1.0 What happens when there are not enough F/B samples?

29 VdVd Score for α p = α : Discretize and normalize

30 VsVs Matte smoothness :

31 Iterative optimization – step 2 U c = {user scribbles + 15 pixel radius} Our goal: find α matte for U c that minimizes the energy - U c  Graph Nodes = Pixels, Edges by 4-connectivity

32 Iterative optimization – step 2 GOAL: Minimize BELIEF PROPAGATION

33 Iterative optimization – step 2 GOAL: Minimize BELIEF PROPAGATION αlog p …… 12.3 p q m pq – message from p to q t=0 y Vector: p’s “opinion” for each possible α for q

34 Iterative optimization – step 2 GOAL: Minimize αlog p …… 12.2 p q t=1 y αlog p …… BELIEF PROPAGATION m pq – new message p  q m yp – previous message y  p

35 Iterative optimization – step 2 GOAL: Minimize BELIEF PROPAGATION p q t=2,3,4… y

36 Iterative optimization – step 2 GOAL: Minimize BELIEF PROPAGATION p q t=T (stopping time) y αLog p …… 11 αLog p …… 11.3 αLog p …… 11 αLog p …… 11

37 Iterative optimization – step 2 GOAL: Minimize BELIEF PROPAGATION p q t=T (stopping time) y αLog p …… 11 αLog p …… 11.3 αLog p …… 11 αLog p …… 11 Best state calculated for each node:

38 Iterative optimization – step 3 Found α matte for U c that minimizes the energy - Update F, B and uncertainty:

39 Iterative optimization - algorithm Initialize U c, F, B, u and alpha matte from scribbles Repeatedly: Expand U c by another 15 pixel radius Find best alpha matte (BP) Update F,B,u for new matte Stop when total uncertainty is minimal Initial matte Propagation of α matte Final matte

40 Iterative optimization - Results Input imageExtracted matte

41 Iterative optimization - Results Input image Extracted matte Composite image

42 Iterative optimization - Results The ambiguity bunny

43 Ambiguity bunny with trimap Iterative optimization - Results Scribbles resultTrimap result Ambiguity bunny with scribbles

44 Iterative optimization - Summary Minimal user input Applicable to video Sensitive to ambiguity in F, B Uses simple color-model Performance: –15-20 min. on a 640x480 image –Factor 50 reported by better implementation

45 Fantastic. Let’s go on…

46 Outline The matting problem Previous work New approaches: –The iterative approach Jue Wang, Michael F.Cohen –Closed form solution Anat Levin, Dani Lischinski,Yair Weiss Comparison and summary Bonus?

47 Closed form solution Anat Levin Dani Lischinski Yair Weiss

48 Closed form solution Assumption: local smoothness in F, B  cancel out unknowns from the matte equs. Solve for F,B and alpha using algebraic tricks.

49 Closed form solution Assumptions: –F,B locally smooth.  treat F,B as constant in a small window w

50 Closed form solution GOAL: Minimize - -Numerical stability -Bias to smoother matte wjwj

51 Closed form solution GOAL: –Minimize:

52 Closed form solution Minimize: 3N Variables (N = image size) We can rid a, b by algebraic manipulation

53 Closed form solution Minimize: Theorem: for we have Intuitively, L is some covariance matrix

54 Closed form solution Minimize: Proof: Rewrite in matrix form:

55 Closed form solution Minimize: Proof: Rewrite in matrix form: By mean-least-squares, best a,b pair for each window is:

56 Closed form solution Some more manipulation give the required result EXCITED? GET YOUR I LOVE MATH T-SHIRT, NOW FOR ONLY $19 99

57 Closed form solution For color images: –Simple: Do each channel separately –Smart: Assume one alpha for R,G,B. Use redundancy to allow a “color-line” model per window Color line model: OUT: F, B Constant within a window IN: F, B are on some line R G F1F1 F2F2

58 Closed form solution For color images: –Simple: Do each channel separately –Smart: Assume one alpha for R,G,B. Use redundancy to allow a “color-line” model per window

59 Closed form solution Now, as before, cost is: And a,b can be cancelled out. For color images: –Simple: Do each channel separately –Smart: Assume one alpha for R,G,B. Use redundancy to allow a “color-line” model per window

60 Closed form solution Now problem reduced to finding best α for: L is Huge  size NxN (N = # image pixels) But Sparse…

61 Closed form solution The algorithm: –Compute L –Solve for given the scribbles. Solving a sparse set of bilinear equations with constraints (Lagrange multipliers) –Find F, B given the matte Adding smoothness assumptions on F, B Improvements: –Use larger environment in low cost by “pyramids”

62 Closed form solution - Results Input image Extracted matte

63 Closed form solution - Results Input image with scribbles Problematic matte

64 Eigenvectors as guides Small eigenvectors of L are correlated with minimal matte L is positive definite. Eigenbasis: v 1,…,v N Eigenvalues: λ 1 > λ 2 > … > λ N > 0

65 Eigenvectors as guides Small eigenvectors of L are correlated with minimal matte

66 Eigenvectors as guides Small eigenvectors of L are correlated with minimal matte  can guide user scribbles Eigenvectors matching smallest eigenvalues Guided scribbles Resulting matte

67 Closed form solution - Summary Minimal user input Provable optimality (under assumptions) Assumes only smooth F,B (no color model) Applicable to video (as we speak…) Problematic with textures Performance: –20-40 seconds for a 200x300 image –Expensive in memory

68 Superb. Let’s sum up…

69 Outline The matting problem Previous work New approaches: –The iterative approach Jue Wang, Michael F.Cohen –Closed form solution Anat Levin, Dani Lischinski,Yair Weiss Comparison and summary Bonus?

70 Comparison Iterative approach Poisson Closed form solution Input image Matte ground truth

71 Main improvements Trimap based approaches New approaches User input TrimapScribbles Complex foreground Poor results. Exact trimap required Good results Video Not easily applicable.Applicable

72 Comparison Color ambiguity Iterative approach Closed form Sensitive Solvable by adding more scribbles

73 Comparison Improving results… Iterative approach Bayesian Closed form solution Ambiguity bunny

74 Comparison Optimality? Iterative approach Closed form Uses heuristics to optimize Provably optimal But for the specific (simplified) cost

75 Comparison Textures Iterative approach Closed form Assumes only Alpha matte smooth F,B must satisfy color-line model

76 Comparison Rough edges Iterative approach Closed form Assumes Alpha matte smooth Can handle rough edges Input image with scribbles  matte results 

77 Comparison Running time Iterative approach Closed form ~20 sec. 20/40 seconds Costly in memory (For medium size image)

78 Comparison Tests Iterative approach Closed form No quantitative results reported Extensively tested quantitative results

79 Outline The matting problem Previous work New approaches: –The iterative approach Jue Wang, Michael F.Cohen –Closed form solution Anat Levin, Dani Lischinski,Yair Weiss Comparison and summary Bonus?

80 Environment Matting and Compositing Douglas E. Zongker ~ Dawn M. Werner ~ Brian Curless ~ David H. Salsin

81 Environment Matting C = F + (1-  )B +   ~ Contribution of light from Environment that travels through the object R – reflectance image T – Texture image

82 Environment Matting? Alpha Matte Environment Matte Photograph

83 Environment Mattin Alpha Matte Environment Matte Photograph

84 Summary The matting problem Old methods: require trimap Two new methods from scribbles: –Iterative optimization Assume: matte smooth, F,B locally similar Use heuristic optimization for alpha –Close form solution Assume: F, B locally smooth (color-line model) Solve linear equations for alpha

85 ANY LAST

86