1. Image-based rendering Michael F. Cohen Microsoft Research

2. Computer Graphics Image Output Model Synthetic Camera

3. Real Scene Computer Vision Real Cameras Model Output

4. Combined Model Real Scene Real Cameras Image Output Synthetic Camera

5. But, vision technology falls short Model Real Scene Real Cameras Image Output Synthetic Camera

6. … and so does graphics. Model Real Scene Real Cameras Image Output Synthetic Camera

7. Image Based Rendering Real Scene Real Cameras -or- Expensive Image Synthesis Images+Model Image Output Synthetic Camera

8. Ray q Constant radiance time is fixed q 5D 3D position 2D direction

9. All Rays q Plenoptic Function all possible images too much stuff!

10. Line q Infinite line q 4D 2D direction 2D position

11. Ray q Discretize q Distance between 2 rays Which is closer together?

12. Image q What is an image? q All rays through a point Panorama?

13. Image q 2D position of rays has been fixed direction remains

14. Image q Image plane q 2D position

15. q Image plane q 2D position Image

16. q Light leaving towards “eye” q 2D just dual of image Object

17. q All light leaving object

18. Object q 4D 2D position 2D direction

19. Object q All images

20. Lumigraph q How to organize capture render

21. Lumigraph - Organization q 2D position q 2D direction s 

22. Lumigraph - Organization q 2D position q 2 plane parameterization s u

23. Lumigraph - Organization q 2D position q 2 plane parameterization u s t s,t u,v v s,t u,v

24. Lumigraph - Organization  Hold s,t constant  Let u,v vary q An image s,t u,v

25. Lumigraph - Organization q Discretization higher res near object if diffuse captures texture lower res away captures directions s,t u,v

26. Lumigraph - Capture q Idea 1 Move camera carefully over s,t plane Gantry see Lightfield paper s,t u,v

27. Lumigraph - Capture q Idea 2 Move camera anywhere Rebinning see Lumigraph paper s,t u,v

28. Lumigraph - Rendering q For each output pixel determine s,t,u,v either find closest discrete RGB interpolate near values s,t u,v

29. Lumigraph - Rendering q For each output pixel determine s,t,u,v either use closest discrete RGB interpolate near values s u

30. Lumigraph - Rendering q Nearest closest s closest u draw it q Blend 16 nearest quadrilinear interpolation s u

31. High-Quality Video View Interpolation Using a Layered Representation Larry Zitnick Sing Bing Kang Matt Uyttendaele Simon Winder Rick Szeliski Interactive Visual Media Group Microsoft Research

32. Current practice Many cameras Motion Jitter vs. free viewpoint video

33. Current practice Many cameras Motion Jitter vs. free viewpoint video

34. Video view interpolation Fewer cameras Smooth Motion Automatic and Real-time rendering

35. Prior work: IBR (static) The Lumigraph Gortler et al., SIGGRAPH ‘96 Concentric Mosaics Shum & He, SIGGRAPH ‘99 Plenoptic Modeling McMillan & Bishop, SIGGRAPH ‘95 Light Field Rendering Levoy & Hanrahan, SIGGRAPH ‘96

36. Prior work: IBR (dynamic) Free-viewpoint Video of Humans Carranza et al., SIGGRAPH ‘03 Image-Based Visual Hulls Matusik et al., SIGGRAPH ‘00 Virtualized Reality TM Kanade et al., IEEE Multimedia ‘97 Dynamic Light Fields Goldlucke et al., VMV ‘02 Stanford Multi-Camera Array Project 3D TV Matusik & Pfister, SIGGRAPH ‘04

37. System overview OFFLINE ONLINE Video Capture Stereo Compression Selective Decompression Render File Representation Video Capture

38. concentrators hard disks controlling laptop controlling laptop cameras

39. Calibration Zhengyou Zhang, 2000

40. Input videos

41. Key to view interpolation: Geometry Stereo Geometry Camera 1Camera 2 Image 1 Image 2 Virtual Camera

42. Match Score Good Bad Image correspondence Correct Image 1 Image 2 Leg Wall Incorrect

43. Image 1 Image 2 Local matching Low texture

44. Number of states = number of depth levels Image 2Image 1 Global regularization Create MRF (Markov Random Field): A F E D C B color A ≈ color B → z A ≈ z B Each segment is a node z A ≈ z P, z Q, z S P Q R S T U A A

45. Iteratively solve MRF

46. Depth through time

47. Matting Interpolated view without matting Background Surface Foreground Surface Camera Foreground Alpha Background Bayesian Matting Chuang et al. 2001 Strip Width Background Foreground

48. Rendering with matting Matting No Matting

49. Representation Main Layer: Color Depth Main Boundary Boundary Layer: Color Depth Alpha Background Foreground Strip Width

50. “Massive Arabesque” videoclip