1. Surface Light Fields for 3D Photography Daniel Wood Daniel Azuma Wyvern Aldinger Brian Curless Tom Duchamp David Salesin Werner Stuetzle

2. 3D Photography Goals Rendering and editing Inputs Photographs and geometry Requirements Estimation and compression

3. View-dependent texture mapping Debevec et al. 1996, 1998 Pulli et al. 1997

4. View-dependent texture mapping Debevec et al. 1996, 1998 Pulli et al. 1997

5. Two-plane light field Levoy and Hanrahan 1996 Gortler et al. 1996

6. Surface light fields Walter et al. 1997 Miller et al. 1998 Nishino et al. 1999

7. Lumisphere-valued “texture” maps Lumisphere

8. Overview Data acquisition Estimation and compression Rendering Editing

9. Overview Data acquisition Estimation and compression Rendering Editing

10. Scan and reconstruct geometry Reconstructed geometryRange scans (only a few shown...)

11. Take photographs Camera positionsPhotographs

12. Register photographs to geometry Geometry Photographs

13. Register photographs to geometry User selected correspondences (rays)

14. Parameterizing the geometry Base mesh Scanned geometry Map

15. Sample base mesh faces Base meshDetailed geometry

16. Assembling data lumispheres Data lumisphere

17. Overview Data acquisition Estimation and compression Rendering Editing

18. Pointwise fairing Faired lumisphere Data lumisphere

19. Pointwise fairing results Input photograph Pointwise faired (177 MB)

20. Pointwise fairing Many input data lumispheres Many faired lumispheres

21. Compression Small set of prototypes

22. Compression / Estimation Small set of prototypesMany input data lumispheres

23. Reflected reparameterization

26. Before After

27. Median removal + Reflected Median (“diffuse”) Median-removed (“specular”) +

28. Median removal Median valuesSpecularResult

29. Function quantization Codebook of lumispheres Input data lumisphere

30. Lloyd iteration Input data lumispheres

31. Lloyd iteration Codeword

32. Lloyd iteration Perturb codewords to create larger codebook

33. Lloyd iteration Form clusters around each codeword

34. Lloyd iteration Optimize codewords based on clusters

35. Lloyd iteration Create new clusters

36. Function quantization results Input photograph Function quantized (1010 codewords, 2.6 MB)

37. Principal function analysis Subspace of lumispheres Input data lumisphere Prototype lumisphere

38. Principal function analysis Approximating subspace Prototype lumisphere

39. Principal function analysis

41. Principal function analysis results Input photograph PFA compressed (Order 5 - 2.5 MB)

42. Compression comparison Pointwise fairing (177 MB) Function quantization (2.6 MB) Principal function analysis (2.5 MB)

43. Comparison with 2-plane light field (uncompressed) Pointwise-faired surface light field (177 MB) Uncompressed lumigraph / light field (177 MB)

44. Comparison with 2-plane light field (compressed) Compressed (PFA) surface light field (2.5 MB) Vector-quantized lumigraph / light field (8.1 MB)

45. Overview Data acquisition Estimation and compression Rendering Editing

46. View-dependent level-of-detail

47. Render texture domain and coordinates in false color

48. Evaluate surface light field

49. Interactive renderer screen capture

50. Overview Data acquisition Estimation and compression Rendering Editing

51. Lumisphere filtering Original surface light fieldGlossier coat

52. Lumisphere filtering

53. Rotating the environment Original surface light fieldRotated environment

54. Deformation OriginalDeformed

55. Deformation

56. Summary 1.Estimation and compression Function quantization Principal function analysis 2.Rendering From compressed representation With view-dependent level-of-detail 3.Editing Lumisphere filtering Geometric deformations and transformations

57. Future work Better geometry-to-image registration More complex surfaces (mirrored, refractive, fuzzy…) under more complex illumination Derive geometry from images Combining FQ and PFA

58. Acknowledgements Marc Levoy and Pat Hanrahan –(Thanks for the use of the Stanford Spherical Gantry) Michael Cohen and Richard Szeliski National Science Foundation

59. The end

60. Geometry (fish) Reconstruction: 129,000 faces Memory for reconstruction: 2.5 MB Base mesh: 199 faces Re-mesh (4x subdivided): 51,000 faces Memory for re-mesh: 1 MB Memory with view-dependence: 7.5 MB

61. Light field data and rep (fish) Time to acquire: 1 hour Input images: 661 Raw data size: ~500 MB Lumisphere representation: 3 times subdivided octehedron Lumisphere size: 258 directions

62. Lumigraph (fish) Lumigraph images: 400x400 Lumigraph viewpoints per slab: 8x8 Number of slabs: 6 Lumigraph size (w/o geom): 184 MB Lumigraph VQ dimension: 16384 Lumigraph VQ codewords: 2x2x2x2x3 Compressed size (w/o geom): 8.1 MB

63. Compression (fish) Pointwise faired: Memory = 177 MBRMS error = 9 FQ (2000 codewords) Memory = 3.4 MBRMS error = 23 PFA (dimension 3) Memory = 2.5 MBRMS error = 24 PFA (dimension 5) Memory = 2.9 MBRMS error = ?

64. Pre-processing times (fish) Compute times on ~450 MHz P-III… Range scanning time: 3 hours Geometry registration: 2 hours Image to geometry alignment: 6 hours MAPS (sub-optimal): 5 hours Assembling data lumispheres: 24 hours Pointwise fairing: 30 minutes FQ codebook construction (10%): 30 hours FQ encoding: 4 hours PFA “codebook” construction (0.1%): 20 hours PFA encoding: 2 hours

65. Breakdown and rendering (fish) For PFA dimension 3… Direction mesh: 11 KB Normal maps: 680 KB Median maps: 680 KB Index maps: 455 KB Weight maps: 680 KB Codebook: 3 KB Geometry w/o view dependence: <1 MB Geometry w/ view dependence: 7.5 MB Rendering platform: 550 MHz PIII, linux, Mesa Rendering performance: 6-7 fps (typical)

66. Construct codebook using Lloyd iteration Iterate until convergence: 1.Assign all data lumispheres to closest codeword, forming clusters. 2.Compute new codeword for each cluster by “cluster-wise” fairing. Then split all codewords and start over.

67. Data extrapolation PhotographSurface light field

68. Comparison with 2-plane light field (uncompressed) Pointwise-faired surface light field (177 MB) Uncompressed 2-plane light field (177 MB)

69. Comparison with 2-plane light field (compressed) Principal function analysis surface light field (2.5 MB) Vector-quantized 2-plane light field (8.1 MB)

70. Details Input photograph Pointwise fairing (177 MB) Function quantization (3.4 MB) Principal function analysis (2.5 MB)