1. Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park

2. Midterm Statistics

3. Outline Background Subtraction Matting & compositing

4. Moving in Time Moving only in time, while not moving in space, has many advantages –No need to find correspondences –Can look at how each ray changes over time –In science, always good to change just one variable at a time This approach has always interested artists (e.g. Monet) Modern surveillance video camera is a great source of information –There are now many such WebCams now, some running for several years!

5. Image Stack As can look at video data as a spatio-temporal volume –If camera is stationary, each line through time corresponds to a single ray in space –We can look at how each ray behaves –What are interesting things to ask? t 0 255 time

6. Example

7. Getting the right pixels Average image Median Image

8. Webcams Lots of cool potential projects –Weather morphing, weather extrapolation, etc. http://sv.berkeley.edu/view/

9. Input Video

10. Average Image What is happening?

11. Average/Median Image What can we do with this?

12. Background Subtraction - =

13. Crowd Synthesis 1.Do background subtraction in each frame 2.Find and record “blobs” 3.For synthesis, randomly sample the blobs, taking care not to overlap them

14. Background Subtraction A largely unsolved problem… Estimated background Difference Image Thresholded Foreground on blue One video frame

15. How does Superman fly? Super-human powers? OR Image Matting and Compositing?

16. Image Compositing

17. Compositing Procedure 1. Making Sprite (Image + Mask Image (alpha channel)) 2. adding the sprite on the background (Iinear Interpolation) α

18. Compositing Procedure 1-αα

19. Multiple Compositing 1. Extract Sprites (e.g using Intelligent Scissors in Photoshop) Composite by David Dewey 2. Blend them into the composite (in the right order)

20. Compositing: Two Issues Semi-transparent objects Pixels too large

21. Solution: alpha channel Add one more channel: –Image(R,G,B,alpha) Sprite! Encodes transparency (or pixel coverage): –Alpha = 1:opaque object (complete coverage) –Alpha = 0:transparent object (no coverage) –0<Alpha<1:semi-transparent (partial coverage) Example: alpha = 0.7 Partial coverage or semi-transparency

22. Multiple Alpha Blending So far we assumed that one image (background) is opaque. If blending semi-transparent sprites (the “A over B” operation): I comp =  a I a + (1-  a )  b I b  comp =  a + (1-  a )  b

23. “Pulling a Matte” Problem Definition: –The separation of an image C into A foreground object image C o, a background image C b, and an alpha matte  –C o and  can then be used to composite the foreground object into a different image Hard problem –Even if alpha is binary, this is hard to do automatically (background subtraction problem) –For movies/TV, manual segmentation of each frame is infeasible –Need to make a simplifying assumption…

24. Blue Screen

25. Blue Screen matting Most common form of matting in TV studios & movies Petros Vlahos invented blue screen matting in the 50s. His Ultimatte ® is still the most popular equipment. He won an Oscar for lifetime achievement. A form of background subtraction: –Need a known background –Compute alpha as SSD(C,Cb) > threshold Or use Vlahos’ formula:  = 1-p 1 (B-p 2 G) –Hope that foreground object doesn’t look like background no blue ties!

26. The Ultimatte p 1 and p 2

27. Blue screen for superman?

28. Topics for the Term Project

29. Feature based Image Metamorphosis Thaddeus Beier and Shawn Neely, “Feature-Based Image Metamorphosis “, SIGGRAPH 1992

30. Tour into the picture Anjyo, K., Horry, Y., and Arai, K. “Tour into the picture”, SIGGRAPH 1997

31. Texture synthesis A. A. Efros and T. K. Leung, "Texture Synthesis by Non-parametric Sampling", ICCV 99A. A. Efros and T. K. Leung, "Texture Synthesis by Non-parametric Sampling", ICCV 99 Wei and Levoy, “Fast Texture Synthesis using Tree-structured Vector Quantization “, SIGGRAPH 2000

32. Image Analogy A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Salesin, “Image Analogy”, SIGGRAPH 2001

33. Colorization Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller, ”Transferring color to grayscale images”, SIGGRAPH 2002

34. Image Inpainting M. Bertalmío, G. Sapiro, V. Caselles and C. Ballester. “Image Inpainting”, SIGGRAPH 2000

35. Image Blending Patrick Perez, Michel Gangnet, Andrew Blake, “Poisson Image Editing”, SIGGRAPH 2003 http://www.cs.tau.ac.il/~tommer/adv-graphics/ex1.htm source images Poisson seamless cloning

36. Painterly Rendering Aaron Hertzmann, "Painterly Rendering with Curved Brush Strokes of Multiple Sizes“, SIGGRAPH 1998

37. Painterly Rendering2 Michio Shiraishi and Yasushi Yamaguchi, “Image Moment-Based Stroke Placement”, ACM SIGGRAPH 99 Conference abstracts and applications, 1999