1. Computational Photography, Fall 2013
Matting
Computational Photography, Fall 2013
Connelly Barnes
Slides from Alexei Efros, James Hays

2. How does Superman fly? Super-human powers? OR Image Blending
Image Matting?
Matting vs Blending

3. Physics of Alpha Matting
Semi-transparent objects
Or, why segmentation isn’t good enough.
Pixels too large

4. Alpha channel Add one more channel:
Image(R,G,B,alpha) Sprite!
Encodes transparency (or pixel coverage):
Alpha = 1: opaque object (complete coverage)
Alpha = 0: transparent object (no coverage)
0<Alpha<1: semi-transparent (partial coverage)
Example: alpha = 0.7
Partial coverage or semi-transparency

5. Alpha Blending Icomp = aaIa + (1-aa)abIb acomp = aa + (1-aa)ab
So far we assumed that one image (background) is opaque.
If blending semi-transparent sprites (the “A over B” operation):
Icomp = aaIa + (1-aa)abIb
acomp = aa + (1-aa)ab
Note: sometimes alpha is premultiplied: im(aR,aG,aB,a):
Icomp = Ia + (1-aa)Ib
(same for alpha!)

6. Matting an Image Problem Definition: Hard problem
The separation of an image C into
A foreground object image Co,
a background image Cb,
and an alpha matte a
Co and a can then be used to composite the foreground object into a different image
Hard problem
Even if alpha is binary, this is hard to do automatically (background subtraction problem)
For movies/TV, manual segmentation of each frame is difficult
Need to make a simplifying assumption…

7. Average/Median Image
What can we do with this?

8. Background Subtraction- =

9. Crowd Synthesis (with Pooja Nath)
Do background subtraction in each frame
Find and record “blobs”
For synthesis, randomly sample the blobs, taking care not to overlap them

10. Background Subtraction
A largely unsolved problem…
One video
frame
Estimated
background
Difference
Image
Thresholded
Foreground
on blue

11. Blue Screen

12. Blue Screen matting Most common form of matting in TV studios & movies
Petros Vlahos invented blue screen matting in the 50s. His Ultimatte® is still the most popular equipment. He won an Oscar for lifetime achievement.
A form of background subtraction:
Need a known background
Compute alpha as SSD(C,Cb) > threshold
Or use Vlahos’ formula: a = 1-p1(B-p2G)
Hope that foreground object doesn’t look like background
no blue ties!
Why blue?
Why uniform?

13. The Ultimatte
p1 and p2

14. Blue screen for superman?

15. Semi-transparent mattes
What we really want is to obtain a true alpha matte, which involves semi-transparency
Alpha between 0 and 1

16. Matting Problem: Mathematical Definition

17. Why is general matting hard?

18. Solution #1: No Blue!

19. Solution #2: Gray or Flesh

20. Triangulation Matting (Smith & Blinn)
How many equations?
How many unknowns?
Does the background need to be constant color?

21. The Algorithm

22. Triangulation Matting Examples

23. More Examples

24. More examples

25. Intelligent Scissors
[Mortensen, SIGGRAPH 1995]

26. Intelligent Scissors

27. Intelligent Scissors
Initial Cost Matrix

28. Intelligent Scissors
Distance from Seed Point (Dynamic Programming)

29. Trimap
[Juan and Keriven 2005]

30. Poisson Matting
[Sun et al, SIGGRAPH 2004]

31. Poisson Matting
Matting Equation (not alpha-premultiplied)

32. Poisson Matting
Differentiate. Now what simplifying assumptions can we make?

33. Poisson Matting
Approximation. Solve in least square sense

34. Poisson Matting
Least squares (variational) problem. Now differentiate again.

35. Poisson Matting
Same equation as Project 2 – Poisson Image Editing

36. Poisson Matting Iteratively solve for (1) a, (2) F - B
Poisson Matting Paper

37. Poisson Matting

38. Problems with Matting Solution: Modify the Matting Equation
Images do not look realistic
Lack of Refracted Light
Lack of Reflected Light
Solution:
Modify the Matting Equation

39. Environment Matting and Compositing
slides by Jay Hetler
Douglas E. Zongker ~ Dawn M. Werner ~ Brian Curless ~ David H. Salsin
SIGGRAPH 99

40. Environment Matting Equation
C = F + (1- a)B + F
C ~ Color
F ~ Foreground color
B ~ Background color
a ~ Amount of light that passes through the foreground
F ~ Contribution of light from Environment that travels through the object
Alpha holds values from zero to one…

41. Explanation of F
R – reflectance image
T – Texture image

42. Series of structured backgrounds
Environment Mattes

43. Performance Calibration
Matting: minutes extraction time for each texture map (Pentium II 400Mhz)
Compositing: 4-40 frames per second
Real-Time?
3-6 hours

44. How much better is Environment Matting?
Alpha Matte Environment Matte Photograph

45. How much better is Environment Matting?
Alpha Matte Environment Matte Photograph

46. Shree K. Nayar Gurunandan G. Krishnan Columbia University
Fast Separation of Direct and Global Images Using High Frequency Illumination
Shree K. Nayar
Gurunandan G. Krishnan
Columbia University
Michael D. Grossberg
City College of New York
Ramesh Raskar
MERL
SIGGRAPH Conference
Boston, July 2006
Support: ONR, NSF, MERL

47. Direct and Global Illumination
participating
medium
D : Volumetric
translucent
surface E
E : Diffusion C
C : Subsurface A
A : Direct
surface
source B
B : Interrelection P
camera

48. Direct and Global Components: Interreflections
surface
source j
BRDF and geometry i
camera
direct
global
radiance

49. High Frequency Illumination Pattern
surface
source i
camera
fraction of activated source elements +

50. - High Frequency Illumination Pattern + i surface source camera
fraction of activated source elements

51. Separation from Two Images
direct
global

52. Other Global Effects: Subsurface Scattering
translucent
surface
source i j
camera

53. Other Global Effects: Volumetric Scattering
participating medium
surface
source i j
camera

54. Diffuse
Interreflections
Specular
Volumetric
Scattering
Subsurface
Diffusion

55. Scene

56. Scene
Direct
Global

57. More Real World Examples:

58. Eggs: Diffuse Interreflections
Direct
Global

59. Wooden Blocks: Specular Interreflections
Direct
Global

60. Kitchen Sink: Volumetric Scattering
Chandrasekar 50, Ishimaru 78
Direct
Global

61. Peppers: Subsurface Scattering
Direct
Global

62. Hand Direct Global Skin: Hanrahan and Krueger 93,
Uchida 96, Haro 01, Jensen et al. 01,
Cula and Dana 02, Igarashi et al. 05, Weyrich et al. 05
Direct
Global

63. Face: Without and With Makeup
Without Makeup
Global
Direct
With Makeup

64. Blonde Hair Direct Global Hair Scattering: Stamm et al. 77,
Bustard and Smith 91, Lu et al. 00
Marschner et al. 03
Direct
Global