1. Video Matting from Depth Maps Jonathan Finger Oliver Wang University of California, Santa Cruz {jfinger, owang}@soe.ucsc.edu

2. Motivation Given a video, replace the background with something different  Isolate the find foreground in each frame

3. Our Method Use a depth camera to automate foreground extraction Use Bayesian matting Improve the matting algorithm to get more realistic video

4. The Matting Problem Separation of a foreground image from a background image

5. The Easy Direction Background (known) Foreground (known) Composite (unknown) 2 knowns, 1 unknown

6. The Hard Direction Background (unknown) Foreground (unknown) Composite (known) 1 known, 2 unknowns

7. The Matting Problem Actually there is another unknown  Represents areas that are a combination of foreground and background 0 1 transparent opaque ::

8. The Matting Problem =1=1  =0.5 =0=0 Foreground

9. The Matting Problem How do we isolate the foreground?  Use an alpha mask Alpha Mask  An image who's color represents foreground and background

10. The Matting Problem original alpha mask

11. The Masking Problem Basic pipeline Original composite Alpha mask Isolated foreground New background New composite

12. The Masking Problem But, how do you get an alpha mask?

13. Previous Work Blue Screen Matting Petro Vlahos (1964) Hollywood Special Effects pioneer Can isolate the foreground if the background is a constant color

14. Previous Work Background is known so it is easy to make a mask

15. The Matting Problem How can this be done with an unknown background?  Use a general matting algorithm input: original composite + trimap output: alpha mask

16. Trimaps A three color image (usually drawn by hand)  Black = 100% background  White = 100% foreground  Gray = unknown

17. Trimaps The matting algorithm fills in the gray area with estimated alpha values

18. Matting Algorithms The matting equation For each 2D location in the image, there is a given composite pixel C We are to find F, B, and  at each pixel where C =  F + (1 -  )B

19. Bayesian Matting Original compositeTrimap Foreground estimation Background estimationAlpha mask

20. Matting Algorithms alpha maskbackground removedclose up Knockout Ruzon and Tomasi Bayesian

21. Problem with Bayesian Matting These all require a manual trimap Our goal is to do this with video  We do not want to make trimaps by hand

22. Previous Work Defocus Video Matting (McGuire) Two cameras  one focused on the background  one focused on the foreground

23. Previous Work A trimap can be generated from the defocused foreground However, apertures have to be very specific and can be thrown off by lighting Also requires texture in the scene

24. Previous Work Bayesian Matting Using Learned Image Priors (Apostoloff, Fitzgibbon) Sequences of frames can be compared in order to find movement

25. Previous Work assumptions  foreground is moving  nothing else is moving

26. Our Contribution Automatically generated trimaps  Does not depend on lighting, texture or movement Improved Bayesian Matting using depth information Hella Trimaps

27. Overview Low res depth Original composite High res depthTrimap New composite New backgroundAlpha mask Supersample Bayesian matting Compose

28. Our Method Canesta depth camera  Uses infrared lasers to detect distances from the camera

29. Our Method Optical imageDepth image Canesta takes 64x64 resolution image  Optical images are 640x640 or more

30. Trimap Overview To get a trimap 1) Upsample depth image to resolution of optical image 2) Threshold to separate into two colors 3) Erode/dilate to create a gray border around the foreground

31. Upsampling Use Qing's supersampled depth method  Use edge cues from high resolution color image  Can increase the depth resolution to up to 100X

32. Thresholding Assumption  Foreground is in front of background  Threshold on a distance plane Done once for entire animation

33. Erode/Dilate Grow unknown area around edges

34. Improved Bayesian Matting Bayesian matting is ill defined when the foreground and background are similar colors Original image Alpha mask

35. Improved Bayesian Matting Use depth information in Bayesian Matting optimization step Original image Bayesian matting Depth map

36. Improved Bayesian Matting Bayesian MattingImproved Bayesian Method

37. Results video

38. Conclusion Video matting can be done without the user having to manually tweak any individual frames We were able to improve Bayesian Matting using depth information