Surface Light Fields for 3D Photography Daniel Wood Daniel Azuma Wyvern Aldinger Brian Curless Tom Duchamp David Salesin Werner Stuetzle

3D Photography Goals Rendering and editing Inputs Photographs and geometry Requirements Estimation and compression

View-dependent texture mapping Debevec et al. 1996, 1998 Pulli et al. 1997

View-dependent texture mapping Debevec et al. 1996, 1998 Pulli et al. 1997

Two-plane light field Levoy and Hanrahan 1996 Gortler et al. 1996

Surface light fields Walter et al Miller et al Nishino et al. 1999

Lumisphere-valued “texture” maps Lumisphere

Overview Data acquisition Estimation and compression Rendering Editing

Overview Data acquisition Estimation and compression Rendering Editing

Scan and reconstruct geometry Reconstructed geometryRange scans (only a few shown...)

Take photographs Camera positionsPhotographs

Register photographs to geometry Geometry Photographs

Register photographs to geometry User selected correspondences (rays)

Parameterizing the geometry Base mesh Scanned geometry Map

Sample base mesh faces Base meshDetailed geometry

Assembling data lumispheres Data lumisphere

Overview Data acquisition Estimation and compression Rendering Editing

Pointwise fairing Faired lumisphere Data lumisphere

Pointwise fairing results Input photograph Pointwise faired (177 MB)

Pointwise fairing Many input data lumispheres Many faired lumispheres

Compression Small set of prototypes

Compression / Estimation Small set of prototypesMany input data lumispheres

Reflected reparameterization

Before After

Median removal + Reflected Median (“diffuse”) Median-removed (“specular”) +

Median removal Median valuesSpecularResult

Function quantization Codebook of lumispheres Input data lumisphere

Lloyd iteration Input data lumispheres

Lloyd iteration Codeword

Lloyd iteration Perturb codewords to create larger codebook

Lloyd iteration Form clusters around each codeword

Lloyd iteration Optimize codewords based on clusters

Lloyd iteration Create new clusters

Function quantization results Input photograph Function quantized (1010 codewords, 2.6 MB)

Principal function analysis Subspace of lumispheres Input data lumisphere Prototype lumisphere

Principal function analysis Approximating subspace Prototype lumisphere

Principal function analysis

Principal function analysis results Input photograph PFA compressed (Order MB)

Compression comparison Pointwise fairing (177 MB) Function quantization (2.6 MB) Principal function analysis (2.5 MB)

Comparison with 2-plane light field (uncompressed) Pointwise-faired surface light field (177 MB) Uncompressed lumigraph / light field (177 MB)

Comparison with 2-plane light field (compressed) Compressed (PFA) surface light field (2.5 MB) Vector-quantized lumigraph / light field (8.1 MB)

Overview Data acquisition Estimation and compression Rendering Editing

View-dependent level-of-detail

Render texture domain and coordinates in false color

Evaluate surface light field

Interactive renderer screen capture

Overview Data acquisition Estimation and compression Rendering Editing

Lumisphere filtering Original surface light fieldGlossier coat

Lumisphere filtering

Rotating the environment Original surface light fieldRotated environment

Deformation OriginalDeformed

Deformation

Summary 1.Estimation and compression Function quantization Principal function analysis 2.Rendering From compressed representation With view-dependent level-of-detail 3.Editing Lumisphere filtering Geometric deformations and transformations

Future work Better geometry-to-image registration More complex surfaces (mirrored, refractive, fuzzy…) under more complex illumination Derive geometry from images Combining FQ and PFA

Acknowledgements Marc Levoy and Pat Hanrahan –(Thanks for the use of the Stanford Spherical Gantry) Michael Cohen and Richard Szeliski National Science Foundation

The end

Geometry (fish) Reconstruction: 129,000 faces Memory for reconstruction: 2.5 MB Base mesh: 199 faces Re-mesh (4x subdivided): 51,000 faces Memory for re-mesh: 1 MB Memory with view-dependence: 7.5 MB

Light field data and rep (fish) Time to acquire: 1 hour Input images: 661 Raw data size: ~500 MB Lumisphere representation: 3 times subdivided octehedron Lumisphere size: 258 directions

Lumigraph (fish) Lumigraph images: 400x400 Lumigraph viewpoints per slab: 8x8 Number of slabs: 6 Lumigraph size (w/o geom): 184 MB Lumigraph VQ dimension: Lumigraph VQ codewords: 2x2x2x2x3 Compressed size (w/o geom): 8.1 MB

Compression (fish) Pointwise faired: Memory = 177 MBRMS error = 9 FQ (2000 codewords) Memory = 3.4 MBRMS error = 23 PFA (dimension 3) Memory = 2.5 MBRMS error = 24 PFA (dimension 5) Memory = 2.9 MBRMS error = ?

Pre-processing times (fish) Compute times on ~450 MHz P-III… Range scanning time: 3 hours Geometry registration: 2 hours Image to geometry alignment: 6 hours MAPS (sub-optimal): 5 hours Assembling data lumispheres: 24 hours Pointwise fairing: 30 minutes FQ codebook construction (10%): 30 hours FQ encoding: 4 hours PFA “codebook” construction (0.1%): 20 hours PFA encoding: 2 hours

Breakdown and rendering (fish) For PFA dimension 3… Direction mesh: 11 KB Normal maps: 680 KB Median maps: 680 KB Index maps: 455 KB Weight maps: 680 KB Codebook: 3 KB Geometry w/o view dependence: <1 MB Geometry w/ view dependence: 7.5 MB Rendering platform: 550 MHz PIII, linux, Mesa Rendering performance: 6-7 fps (typical)

Construct codebook using Lloyd iteration Iterate until convergence: 1.Assign all data lumispheres to closest codeword, forming clusters. 2.Compute new codeword for each cluster by “cluster-wise” fairing. Then split all codewords and start over.

Data extrapolation PhotographSurface light field

Comparison with 2-plane light field (uncompressed) Pointwise-faired surface light field (177 MB) Uncompressed 2-plane light field (177 MB)

Comparison with 2-plane light field (compressed) Principal function analysis surface light field (2.5 MB) Vector-quantized 2-plane light field (8.1 MB)

Details Input photograph Pointwise fairing (177 MB) Function quantization (3.4 MB) Principal function analysis (2.5 MB)