1. Image or Object? Michael F. Cohen Microsoft Research

2. Is this real? Photo by Patrick Jennings (patrick@synaptic.bc.ca), Copyright 1995, 96, 97 Whistler B. C. Canada

3. A mental model Photo by Patrick Jennings (patrick@synaptic.bc.ca), Copyright 1995, 96, 97 Whistler B. C. Canada ?

4. Sue Vision

5. Is this real? Michael F. Cohen, Shenchang Eric Chen, John R. Wallace, and Donald P. Greenberg 1988, A Progressive Refinement Approach to Fast Radiosity Image Generation.

6. Bob Graphics

7. Computer Graphics Image Output Model Synthetic Camera

8. Real Scene Computer Vision Real Cameras Model Output

9. Computer Graphics Meets Computer Vision

11. Combined Model Real Scene Real Cameras Image Output Synthetic Camera

12. But, vision technology falls short Model Real Scene Real Cameras Image Output Synthetic Camera

13. … and so does graphics. Model Real Scene Real Cameras Image Output Synthetic Camera

14. Image Based Rendering Real Scene Real Cameras -or- Expensive Image Synthesis Images+Model Image Output Synthetic Camera

15. Ray q Constant radiance time is fixed q 5D 3D position 2D direction

16. All Rays q Plenoptic Function all possible images too much stuff!

17. Line q Infinite line q 4D 2D direction 2D position

18. Ray q Discretize q Distance between 2 rays Which is closer together?

19. Image q What is an image? q All rays through a point Panorama?

20. Image q 2D position of rays has been fixed direction remains

21. Image q Image plane q 2D position

22. q Image plane q 2D position Image

23. q Light leaving towards “eye” q 2D just dual of image Object

24. q All light leaving object

25. Object q 4D 2D position 2D direction

26. Object q All images

27. Object q Computer Vision?

28. Plenoptic Function q 5D q Rays

29. Lumigraph / Lightfield q Outside convex space q 4D Stuff Empty

30. Lumigraph q Inside convex space q 4D Empty Stuff

31. Lumigraph q How to organize capture render

32. Lumigraph - Organization q 2D position q 2D direction s 

33. Lumigraph - Organization q 2D position q 2 plane parameterization s u

34. Lumigraph - Organization q 2D position q 2 plane parameterization u s t s,t u,v v s,t u,v

35. Lumigraph - Organization  Hold s,t constant  Let u,v vary q An image s,t u,v

36. Lumigraph - Organization q Discretization higher res near object if diffuse captures texture lower res away captures directions s,t u,v

37. Lumigraph - Capture q Idea 1 Move camera carefully over s,t plane Gantry see Lightfield paper s,t u,v

38. Lumigraph - Capture q Idea 2 Move camera anywhere Rebinning see Lumigraph paper s,t u,v

39. Lumigraph - Rendering q For each output pixel determine s,t,u,v either find closest discrete RGB interpolate near values s,t u,v

40. Lumigraph - Rendering q For each output pixel determine s,t,u,v either use closest discrete RGB interpolate near values s u

41. Lumigraph - Rendering q Nearest closest s closest u draw it q Blend 16 nearest quadrilinear interpolation s u

42. Lumigraph - Rendering q Depth Correction closest s intersection with “object” best u closest u s u

43. Lumigraph - Rendering q Depth Correction quadralinear interpolation new “closest” like focus s u

44. Lumigraph - Rendering q Fast s,t,u,v finding scanline interpolate texture mapping shear warp s u

45. Lumigraph - Rendering q Fast s,t,u,v finding scanline interpolate texture mapping shear warp s u s1,u1 s2,u2

46. Lumigraph - Rendering q Fast s,t,u,v finding scanline interpolate texture mapping shear warp s u

47. Lumigraph - Rendering q Fast s,t,u,v finding scanline interpolate texture mapping shear warp s u

48. Lumigraph - Demo q Lumigraph Lion, Fruit Bowl, Visible Woman, Path Tracing

49. Lightfield - Demo q Digital Michelangelo Project Marc Levoy, Stanford University Lightfield (“night”) assembled by Jon Shade

50. Surface Lightfields q Turn 4D parameterization around q Leverage coherence

51. Surface Lightfields q Wood et al, SIGGRAPH 2000

52. 3D Representations q Image is 2D q Lumigraph is 4D q What happened to 3D? 3D Lumigraph subset Concentric mosaics

53. 3D Lumigraph q One row of s,t plane i.e., hold t constant s,t u,v

54. 3D Lumigraph q One row of s,t plane i.e., hold t constant thus s,u,v a “row of images” s u,v

55. Concentric Mosaics q Replace “row” with “circle” of images

56. Concentric Mosaics

57. q From above

58. Concentric Mosaics q From above

59. Concentric Mosaics q Panorama

60. 2.5D Representations q Image is 2D q Lumigraph is 4D q 3D 3D Lumigraph subset Concentric mosaics q 2.5D Layered Depth Images View Dependent Surfaces

61. Layered Depth Image Image Depth Layered 2.5 D ?

62. Layered Depth Image q Rendering from LDI Incremental in LDI X and Y Guaranteed to be in back-to-front order

63. Layered Depth Image q Rendering from LDI Incremental in LDI X and Y Guaranteed to be in back-to-front order

64. Layered Depth Image (LDI)

65. Tiled LDIs q Multiresolution q Directionally dependent

66. View Dependent Surfaces

67. Summary q 5D: Plenoptic Function (Ray) q 4D: Lumigraph / Lightfield q 3D: Lumigraph Subset q 3D: Concentric Mosaics q 2.5D: Layered Depth Image q 2.5D: View Dependent Models q 2D: Image

68. Thanks q Peter-Pike Sloan (Lumigraph) q Jonathan Shade (Lightfield, LDI) q Marc Levoy, Stanford University Michaelangelo data set