1. Lightfields, Lumigraphs, and Image-based Rendering

2. 2D to 4D More than can be captured in one imageMore than can be captured in one image All light passing through a region of spaceAll light passing through a region of space General (5D) vs. unoccluded space (4D)General (5D) vs. unoccluded space (4D) “Rebinning” pixels

3. Image-Based Modeling and Rendering Generate new views of a scene directly from existing viewsGenerate new views of a scene directly from existing views “Pure” IBR (such as lightfields): no model of scene“Pure” IBR (such as lightfields): no model of scene Other IBR techniques try to obtain higher quality with less storage by building a modelOther IBR techniques try to obtain higher quality with less storage by building a model

4. Lightfields Advantages:Advantages: – Simpler computation vs. traditional CG – Cost independent of scene complexity – Cost independent of material properties and other optical effects Disadvantages:Disadvantages: – Static geometry – Fixed lighting – High storage cost

5. Using Lightfields Obtain 2D slices of 4D data setObtain 2D slices of 4D data set Arbitrary views: take other 2D slicesArbitrary views: take other 2D slices Challenges:Challenges: – Capture – Parameterization – Compression – Rendering

6. Capturing Lightfields Need a 2D set of (2D) imagesNeed a 2D set of (2D) images Choices:Choices: – Camera motion: human vs. computer – Constraints on camera motion – Coverage and sampling uniformity – Aliasing

7. Point / anglePoint / angle Two points on a sphereTwo points on a sphere Points on two planesPoints on two planes Original images and camera positionsOriginal images and camera positions Lightfield Parameterization

8. Compression Compress individual images (JPEG, etc.)Compress individual images (JPEG, etc.) Adapt video compression to 2D arraysAdapt video compression to 2D arrays Decomposition into basis functionsDecomposition into basis functions Vector quantizationVector quantization

9. Rendering How to select rays?How to select rays? InterpolationInterpolation Taking advantage of hardwareTaking advantage of hardware – Graphics hardware – Compression hardware

10. Implementations Lightfields, SIGGRAPH 96, Levoy and HanrahanLightfields, SIGGRAPH 96, Levoy and Hanrahan Lumigraphs, SIGGRAPH 96, Gortler et al.Lumigraphs, SIGGRAPH 96, Gortler et al. Unstructured lumigraphs, SIGGRAPH 01, Buehler et al.Unstructured lumigraphs, SIGGRAPH 01, Buehler et al.

11. Light Field Rendering Capture:Capture: – Computer-controlled camera rig – Move camera to grid of locations on a plane

12. Light Field Rendering Parameterization:Parameterization: – Two planes, evenly sampled: “light slab” – In general, planes in arbitrary orientations – In practice, one plane = camera locations Minimizes resamplingMinimizes resampling

13. Light Field Coverage

14. Multi-Slab Light Fields

15. Rendering For each desired ray:For each desired ray: – Compute intersection with (u,v) and (s,t) planes – Take closest ray Variants: interpolationVariants: interpolation – Bilinear in (u,v) only – Bilinear in (s,t) only – Quadrilinear in (u,v,s,t)

16. Light Field Compression Based on vector quantizationBased on vector quantization Preprocessing: build a representative codebook of 4D tilesPreprocessing: build a representative codebook of 4D tiles Each tile in lightfield represented by indexEach tile in lightfield represented by index Example: 2x2x2x2 tiles, 16 bit index = 24:1 compressionExample: 2x2x2x2 tiles, 16 bit index = 24:1 compression

17. Lightfield Compression by Vector Quantization Advantages:Advantages: – Fast at runtime – Reasonable compression Disadvantages:Disadvantages: – Preprocessing slow – Manually-selected codebook size – Does not take advantage of structure

18. The Lumigraph Capture: move camera by handCapture: move camera by hand Camera intrinsics assumed calibratedCamera intrinsics assumed calibrated Camera pose recovered from markersCamera pose recovered from markers

19. Lumigraph Postprocessing Obtain rough geometric modelObtain rough geometric model – Chroma keying (blue screen) to extract silhouettes – Octree-based space carving Resample images to two-plane parameterizationResample images to two-plane parameterization

20. Lumigraph Rendering Use rough depth information to improve rendering qualityUse rough depth information to improve rendering quality

21. Lumigraph Rendering Use rough depth information to improve rendering qualityUse rough depth information to improve rendering quality

22. Lumigraph Rendering Without using geometry Using approximate geometry

23. Unstructured Lumigraph Rendering Further enhancement of lumigraphs: do not use two-plane parameterizationFurther enhancement of lumigraphs: do not use two-plane parameterization Store original pictures: no resamplingStore original pictures: no resampling Hand-held camera, moved around an environmentHand-held camera, moved around an environment

24. Unstructured Lumigraph Rendering To reconstruct views, assign penalty to each original rayTo reconstruct views, assign penalty to each original ray – Distance to desired ray, using approximate geometry – Resolution – Feather near edges of image Construct “camera blending field”Construct “camera blending field” Render using texture mappingRender using texture mapping

25. Unstructured Lumigraph Rendering Blending fieldRendering

26. Other Lightfield Acquisition Devices Spherical motion of camera around an objectSpherical motion of camera around an object Samples space of directions uniformlySamples space of directions uniformly Second arm to move light source – measure BRDFSecond arm to move light source – measure BRDF

27. Other Lightfield Acquisition Devices Acquire an entire light field at onceAcquire an entire light field at once Video ratesVideo rates Integrated MPEG2 compression for each cameraIntegrated MPEG2 compression for each camera (Bennett Wilburn, Michal Smulski, Mark Horowitz)