1. Surface Light Fields for 3D Photography Daniel N. Wood University of Washington SIGGRAPH 2001 Course

2. Collaborators ( and co-authors on SIGGRAPH 2000 paper ) Daniel Azuma Wyvern Aldinger Brian Curless Tom Duchamp David Salesin Werner Stuetzle University of Washington

3. Outline 1.Surface light field representation 2.Compression primer 3.Surface light fields for 3D photography With details of compression And a preliminary look at a new compression algorithm

4. Surface light fields

5. Lumisphere-valued “texture” maps Lumisphere

6. Surface light fields in flatland s 

7. s 

8. s 

9. Compression primer Singular value decomposition (or principal components analysis) –Handling color –Using regions –Reflection parameterization Vector quantization Others… (Wavelets, DCT)

10. Singular value decomposition SLF UVTVT 

11. SVD in two matrices Eigen Textures Eigen Lumispheres SLF

12. First eigenvectors Eigen Textures Eigen Lumispheres SLF First eigentexture and corresponding first eigenlumisphere

13. Truncated SVD Outer product of first eigen-texture and first eigen- lumisphere is closest rank 1 (separable) matrix. = Eigen-lumisphere Eigen-texture ~ ~

14. Handling color Separate matrices Colors in surface texture (columns) Colors in directions (rows)

15. Handling color Separate matrices Nishino et al.Wood et al.Chen et al. Colors in surface texture (columns) Colors in directions (rows)

16. SVD in color Eigen Textures Eigen Lumispheres SLF

17. Zooming in… Eigen Textures Eigen Lumispheres SLF

18. Zoom into important vectors......... SLF

19. Reconstruction using SVD OriginalRank 1Rank 7 (1:20)

20. Surface light field structure Rows are points on surface What are the columns? And, can they be made more coherent?

21. Surface light field structure Rows are points on surface Increasing column coherence: 1.Break into regions, and / or 2.Use reflection parameterization

22. Separate SVD for regions Eigen Textures Eigen Lumispheres Eigen Textures Eigen Lumispheres

23. Reconstruction using regions OriginalOne region (Rank 5) Two regions (Rank 5)

24. Reflection reparameterization

27. Reflection in flatland* *sort of Un-reflectedReflected

28. Reflection doesn’t happen in the plane  Un-reflectedReflected 

29. Reflection in 3 space Un-reflected Reflected

30. Reflection in “flatland” Un-reflectedReflected Original Rank 5

31. Other compression strategies Discrete cosine transform [ Miller et al. 1999 – Eurographics Workshop on Rendering] Vector quantization [ Wood et al. 2000 - SIGGRAPH ] Wavelet decomposition [ Magnor and Girod 2000 - SPIE VCIP]

32. Vector quantization (unreflected) UncompressedCodebookVector- quantized

33. Vector quantization ( reflected ) UncompressedCodebookVector- quantized

34. Surface light fields for 3D photography ( SIGGRAPH 2000 ) Goals Rendering and editing Inputs Photographs and geometry Requirements Estimation and compression

35. Overview Data acquisition Estimation and compression Rendering Editing

36. Overview Data acquisition Estimation and compression Rendering Editing

37. Scan and reconstruct geometry Reconstructed geometryRange scans (only a few shown...)

38. Take photographs Camera positionsPhotographs

39. Register photographs to geometry Geometry Photographs

40. Register photographs to geometry User selected correspondences (rays)

41. Parameterizing the geometry Base mesh Scanned geometry Map

42. Sample base mesh faces Base meshDetailed geometry

43. Assembling data lumispheres Data lumisphere

44. Overview Data acquisition Estimation and compression Rendering Editing

45. Pointwise fairing Faired lumisphere Data lumisphere

46. Pointwise fairing results Input photograph Pointwise faired (177 MB)

47. Pointwise fairing Many input data lumispheres Many faired lumispheres

48. Compression Small set of prototypes

49. Compression / Estimation Small set of prototypesMany input data lumispheres

50. Median removal + Reflected Median (“diffuse”) Median-removed (“specular”) +

51. Median removal Median valuesSpecularResult

52. Function quantization Codebook of lumispheres Input data lumisphere

53. Lloyd iteration Input data lumispheres

54. Lloyd iteration Codeword

55. Lloyd iteration Perturb codewords to create larger codebook

56. Lloyd iteration Form clusters around each codeword

57. Lloyd iteration Optimize codewords based on clusters

58. Lloyd iteration Create new clusters

59. Function quantization results Input photograph Function quantized (1010 codewords, 2.6 MB)

60. Principal function analysis Subspace of lumispheres Input data lumisphere Prototype lumisphere

61. Principal function analysis Approximating subspace Prototype lumisphere

62. Principal function analysis

64.  ++  + Median PFA decomposition

65. Principal function analysis  + + … Median PFA decomposition = Final approximation

66. Principal function analysis results Input photograph PFA compressed (Order 5 - 2.5 MB)

67. Compression comparison Pointwise fairing (177 MB) Function quantization (2.6 MB) Principal function analysis (2.5 MB)

68. Rewind… what we didn’t want to do

69. Instead, a middle ground Regularly sampled directions, but not all there.

70. New method: Principal components with missing data SLF with missing data Hallucinate data Use either fairing or pair-wise present covariance matrix to fill holes Hole-filled SLF

71. New method: Principal components with missing data SLF with missing dataLow rank approximation Hallucinate data Find best low rank approximation Hole-filled SLF

72. New method: Principal components with missing data SLF with missing dataLow rank approximation Hallucinate data Improve hallucination

73. Preliminary comparison Principal function analysis Order 3, RMS error 26.9 SVD with missing data Order 3, RMS error 26.1

74. Overview Data acquisition Estimation and compression Rendering Editing

75. Interactive renderer screen capture

76. Overview Data acquisition Estimation and compression Rendering Editing

77. Lumisphere filtering Original surface light fieldGlossier coat

78. Lumisphere filtering

79. Rotating the environment Original surface light fieldRotated environment

80. Deformation OriginalDeformed

81. Deformation

82. Summary 1.Estimation and compression Function quantization Principal function analysis 2.Rendering From compressed representation With view-dependent level-of-detail 3.Editing Lumisphere filtering Geometric deformations and transformations

83. Future work Better geometry-to-image registration Derive geometry from images More complex surfaces (mirrored, refractive, fuzzy…) under more complex illumination

84. Acknowledgements Marc Levoy and Pat Hanrahan –(Thanks for the use of the Stanford Spherical Gantry) Michael Cohen and Richard Szeliski National Science Foundation

85. Nearly the end

86. For more information http://graphics.cs.washington.edu/projects/slf Talks, papers, … and raw data.