1. CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai

2. Outline Light field rendering Plenoptic sampling (light field sampling) 3D light field (concentric mosaics) Others

3. Q: How many images are needed for anti-aliased light field rendering? Review: Plenoptic Sampling

4. Q: How many images are needed for anti-aliased light field rendering? Review: Plenoptic Sampling A: formulate this as high-dimensional signal (4D Plenoptic function) reconstruction and sampling problem

5. Review: Between Two Planes Z vt t v Z1Z1 Z1Z1 Z2Z2 Z2Z2

6. Review: Minimal Sampling Rate 1/∆t 1/∆v Image resolution Sample interval

7. Review: Minimal Sampling Rate 1/∆T max >=Ω v *(f/z min -f/z max ) ΩvΩv

8. Review: Minimal Sampling Rate Minimal sampling rate depends on: - texture of object (Ω v ) - focal length (f) - depth complexity (z min, z max ) 1/∆T max >=Ω v *(f/z min -f/z max )

9. 3D Plenoptic Function Image/panorama is 2D Light field/lumigraph is 4D What happens to 3D?

10. 3D Plenoptic Function Image/panorama is 2D Light field/lumigraph is 4D What happens to 3D? - 3D light field subset - Concentric mosaic [Siggraph99]

11. 3D light field One row of s,t plane i.e., hold t constant s,t u,v

12. 3D light field One row of s,t plane i.e., hold t constant thus s,u,v a “row of images” s u,v

13. Concentric mosaics [Shum and He] Polar coordinate system: - hold r constant - thus (θ,u,v)

14. Concentric mosaics Why concentric mosaic? - easy to capture - relatively small in storage size - inside looking out

15. Concentric mosaics From above How to capture images?

16. Concentric mosaics From above How to capture images?

17. Concentric mosaics From above How to render a new image?

18. Concentric mosaics From above How to render a new image?

19. Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

20. Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

21. Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

22. Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays How about this ray?

23. Concentric mosaics From above object How to retrieve the closest rays?

24. Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays?

25. Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays? What’s the optimal interpolation radius?

26. Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays? What’s the optimal interpolation radius? 2r min r max /(r min +r max )

27. Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays?

28. Concentric mosaics From above object (s,t) interpolation plane How to retrieval the closest rays?

29. Concentric mosaics From above object (s,t) interpolation plane How to retrieval the closest rays?

30. Concentric mosaics From above object (s,t) interpolation plane How to synthesize the color of rays?

31. Concentric mosaics From above object (s,t) interpolation plane How to synthesize the color of rays? - bilinear interpolation

32. Concentric mosaics From above

33. Concentric mosaics From above

34. Concentric mosaics From above

35. Concentric mosaics What are limitations?

36. Concentric mosaics What are limitations? - limited rendering region? - large vertical distortion

37. Concentric mosaics What are limitations? - limited rendering region? - large vertical distortion

38. 2.5 D representation Image is 2D Light field/lumigraph is 4D 3D - a subset of light field - concentric mosaics 2.5D - layered depth image [Shade et al, SIGGRAPH98] - view-dependent surfaces

39. Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: - image with depths

40. Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: - rays with colors and depths

41. Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: (r,g,b,depth) - image with depths - rays with colors and depths

42. Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image

43. Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Guaranteed to be in back-to-front order - Forward warping one pixel with depth

44. Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Guaranteed to be in back-to-front order - Forward warping one pixel with depth

45. Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image - Incremental in X and Y - Guaranteed to be in back-to-front order - Forward warping one pixel with depth How to deal with occlusion/visibility problem?

46. How to form LDIs Synthetic world with known geometry and texture - from multiple depth images - modified ray tracer Real images - reconstruct geometry from multiple images (e.g., voxel coloring, stereo reconstruction) - form LDIs using multiple images and reconstructed geometry

47. 2.5 D representation Image is 2D Light field/lumigraph is 4D 3D - a subset of light field - concentric mosaics 2.5D - layered depth image [Shade et al, SIGGRAPH98] - view-dependent surfaces

48. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

49. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

50. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

51. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

52. View-dependent texture mapping [Debevec et al 98]

53. View-dependent texture mapping Subject's 3D proxy points V C 0 C 2 C 3 C 1  0  1 D  2  3 - Virtual camera at point D - Textures from camera C i mapped onto triangle faces - Blending weights in vertex V - Angle θ i is used to compute the weight values: w i = exp(-θ i 2 /2σ 2 )

54. 2.5 D representation Image is 2D Light field/lumigraph is 4D 3D - a subset of light field - concentric mosaics 2.5D - layered depth image [Shade et al, SIGGRAPH98] - view-dependent surfaces

55. Videos: view-dependent texture mapping

56. The Image-Based Rendering Problem Synthesize novel views from reference images Static scenes, fixed lighting Flexible geometry and camera configurations

57. The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Geometric Fidelity # of Images VDTM LF

58. The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Geometric Fidelity # of Images VDTM LF ULR

59. The ULR Algorithm [Siggraph01] Designed to work over a range of image and geometry configurations Designed to satisfy desirable properties Geometric Fidelity # of Images VDTM LF ULR

60. Desired Camera “Light Field Rendering,” SIGGRAPH ‘96 u0u0 s0s0 u s Desired color interpolated from “nearest cameras”

61. Desired Camera “Light Field Rendering,” SIGGRAPH ‘96 u s Desired Property #1: Epipole consistency

62. Desired Camera “The Scene” “The Lumigraph,” SIGGRAPH ‘96 u Potential Artifact

63. “The Scene” “The Lumigraph,” SIGGRAPH ‘96 Desired Property #2: Use of geometric proxy Desired Camera

64. “The Lumigraph,” SIGGRAPH ‘96 “The Scene” Desired Camera

65. “The Lumigraph,” SIGGRAPH ‘96 “The Scene” Rebinning Note: all images are resampled. Desired Camera Desired Property #3: Unstructured input images

66. “The Lumigraph,” SIGGRAPH ‘96 “The Scene” Desired Property #4: Real-time implementation Desired Camera

67. View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98 “The Scene” Occluded Out of view

68. Desired Camera “The Scene” Desired Property #5: Continuous reconstruction View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

69. Desired Camera “The Scene” θ1θ1 θ2θ2 θ3θ3 View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

70. Desired Camera “The Scene” θ1θ1 θ2θ2 θ3θ3 Desired Property #6: Angles measured w.r.t. proxy View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

71. “The Scene” Desired Camera

72. “The Scene” Desired Property #7: Resolution sensitivity

73. Unstructured Lumigraph Rendering 1.Epipole consistency 2.Use of geometric proxy 3.Unstructured input 4.Real-time implementation 5.Continuous reconstruction 6.Angles measured w.r.t. proxy 7.Resolution sensitivity

74. Demo