1. Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San Diego

4. Related Work Studio Matting – Vlahos 71, Wallace 82, Smith and Blinn 96, Ben-Ezra 00, Debevec et. al 02, McGuire et al. 06 User Assisted Natural Matting – Ruzon and Tomasi 00, Chuang et al. 01, Chuang et al. 02, Sun et al. 04, Li et al. 04, Rother et al 04, Li et al. 05, Wang et al. 05, Wang and Cohen 05, Levin et al. 06 Automated Natural Matting – Wexler et al. 02, Zitnick et al. 04, McGuire et al. 05, Kolmogrov et al. 05 – Our Method

5. Contributions Extending the matting equation to use the variance of pixel measurements Using a camera array for alpha matting An automatic method for computing alpha mattes that is linear in the number of pixels A near real-time system for natural video matting – 2 to 5 FPS

6. Algorithm Overview Foreground Object Background Camera Array  

7. Using a Camera Array

8. Foreground and Background Parallax

9. Aligning to a Foreground Plane Foreground Plane     I0I0 InIn I0I0 InIn IiIi IiIi

10. Aligned Images

11. Variance across cameras Taking the Variance Background Foreground Transparent (Mixture) Transparent I0I0 InIn IiIi

12. Key Idea alpha is a function of: variance across cameras images

13. Traditional Alpha Matting [Smith and Blinn 96] I =  F + (1 -  )B I = observed image  = transparency F = foreground B = background

14. Alpha Matting with Multiple Images I =  F + (1 -  )B I, F, and B are continuous random variables  is a scalar

15. I =  F + (1 -  ) B Solving for alpha  2 var( F ) + (1 -  ) 2 var( B ) - var( I ) = 0 var( I ) = var[  F + (1 -  ) B ]

16. Solving for alpha and  F  2 var( F ) + (1 -  ) 2 var( B ) - var( I ) = 0 a  2 + b  + c = 0 a = var( F ) + var( B ) b = -2var( B ) c = var( B ) - var( I )

17. Triangulation Matting: A Special Case of our Method I1I1 I2I2 [Smith and Blinn 96]

18.  F is a function of: [mean(I),mean(B)] Variance and Mean Statistics We observe I F and B can’t be directly measured  is a function of: [var(I),var(F),var(B)] ? ? ?

19. Foreground = Low var(I) Background = High var(I) Threshold var(I) to get Trimap

20. Propagating Variance and Mean For each pixel in the unknown (gray) region: – var(F) = var(I nf ) where I nf is the nearest pixel in the foreground – var(B) = var(I nb ), mean(B) = mean(I nb ) where I nb is the nearest pixel in the background

21. Algorithm Steps 1. Calibrate camera array and film scene 2. Find the foreground depth 3. Compute var(I) and mean(I) 4. Threshold var(I) to get trimap 5. Estimate var(F), var(B), mean(B) 6. Compute  and  F

22. System and Implementation Details 8 640x480 (Bayer pattern) Basler Video Cameras 1 Desktop PC (P4 3.0 Ghz) Calibration: – Color w/ Macbeth chart – Bundle Adjustment Average Running Time: 2 to 5 FPS at 320 x 240

23. 500 2500 4500

24. Our methodUsing a known background

29. 500 2500 4500

30. Our method Using a known background

34. Limitations Self-occlusion: alpha is not view independent Known Background Our Method Need background variation, i.e., var(B) should be large X

35. Conclusions Standard matting equation extended to work with variance and mean statistics Running time is linearly proportional to the number of pixels Automatic, near real-time alpha matting using a camera array

36. Acknowledgments SIGGRAPH Reviewers Joe Marks, John Barnwell, Frédo Durand, Matthias Zwicker, Bennett Wilburn, Dan Morris, and Merrie Morris Our willing and patient “Actors”: Lavanya Sharan, Chien-i Tu, Bernd Bickel, Yuanzhen Li MERL (for my internship) NSF IGERT Grant (for additional funding)

37. http://graphics.ucsd.edu/papers/camera_array_matting/