Dynamically Reparameterized Light Fields Aaron Isaksen, Leonard McMillan (MIT), Steven Gortler (Harvard) Siggraph 2000 Presented by Orion Sky Lawlor cs497yzy 2003/4/24

Introduction Lightfield Aquisition Image Reconstruction Synthetic Aperture

Introduction Rendering cool pictures is hard Rendering them in realtime is even harder (Partial) Solution: Image-based rendering Acquire or pre-render many images At display time, recombine existing images somehow Standard sampling problems: Aliasing, acquisition, storage

Why use Image-based Rendering? Captures arbitrarily complex material/light interactions Spatially varying glossy BRDF Global, volumetric, subsurface,... Display speed independent of scene complexity Excellent for natural scenes Non-polygonal description avoids Difficulty doing sampling & LOD Cracks, watertight, manifold,...

Why not use Image-based? Must acquire images beforehand Fixed scene & lighting Often only the camera can move Predetermined sampling rate Undersampling, aliasing problems Predetermined set of views Can’t look in certain directions! Acquisition painful or expensive Must store many, many images Yet access must be quick

How do Lightfields not Work? At every point in space, take a picture (or environment map): 3D Space, 2D Images => 5D Display is just image lookup!

Why don’t Lightfields work like that? These images all contain duplicate rays, again and again 3D Space, 2D Images => 5D

How do Lightfields actually Work? We can thus get away with just one layer of cameras: 2D Cameras 2D Images => 4D Lightfield Reconstructed novel viewpoint Only assumption: Rays are unchanged along path Display means interpolating several views

Camera Array Geometry (Illustration: Isaksen, MIT)

Introduction Lightfield Aquisition Image Reconstruction Synthetic Aperture

How do you make a Lightfield? Synthetic scene Render from different viewpoints Real scene Sample from different viewpoints In either case, need Fairly dense sampling Lots of data, compression useful Good antialiasing, over both the image plane (pixels), and camera plane (apertures)

XY Motion Control Camera Mount (Isaksen, MIT)

8 USB Digital Cameras, covers removed (Jason Chang, MIT)

Lens array (bug boxes!) on a flatbed scanner (Jason Chang, MIT)

(Lightfield: Isaksen, MIT)

Introduction Lightfield Aquisition Image Reconstruction Synthetic Aperture

Lightfield Reconstruction To build a view, just look up light along each outgoing ray: Camera Array Reconstructed novel viewpoint Need both direction and camera interpolation

Two-Plane Parameterization Parameterize any ray via its intersection with two planes: Focal plane, for ray direction Camera plane May need 6 pairs of planes to capture all sides of a 3D object (Slide: Levoy & Hanrahan, Stanford)

Camera and Direction Interpolation (Slide: Levoy & Hanrahan, Stanford)

Mapping camera views to screen Can map camera view to new viewpoint using texture mapping (since everything’s linear) (Figure: Isaksen, MIT) New Camera Old Camera Focal Plane

Lightfield Reconstruction (again) To build a view, just look up light along each outgoing ray: Camera Array Reconstructed novel viewpoint Reconstruction done via graphics hardware & laws of perspective

Related: Lenticular Display Replace cameras with directional emitters, like many little lenses: Reconstructed novel viewpoint Reconstruction done in free space & laws of optics Lens array Image Optional Blockers (Isaksen)

Related: Holography A Hologram is just a sampling plane with directional emission: Reconstructed novel viewpoint Reconstruction done in free space & coherent optics Holographic film Interference patterns on film act like little diffraction gratings, and give directional emission. Reference Beam (Hanrahan)

Introduction Lightfield Aquisition Image Reconstruction Synthetic Aperture

Camera Aperture & Focus Non-pinhole cameras accept rays from a range of locations: Stuff’s in focus here Stuff’s blurry out here Lens One pixel on CCD or film

Camera Aperture Can vary effective lens size by changing physical aperture (“hole”) On a camera, this is the f-stop Small ApertureBig Aperture Not much blurring— long depth of field Lots of depth blurring—short depth of field

Synthetic Aperture Can build a larger aperture in postprocessing, by combining smaller apertures Big Assembled Aperture Note: you can assemble a big aperture out of small ones, but not split a small aperture from a big one—it’s easy to blur, but not to un-blur. Same depth blurring as with a real aperture!

Synthetic Aperture Example Vary reconstructed camera’s aperture size: a larger synthetic aperture means a shorter “depth of field”— shorter range of focused depths. (Illustration: Isaksen, MIT)

Camera Focal Distance Can vary real focal distance by changing the camera’s physical optics FarNear

Synthetic Aperture Focus With a synthetic aperture, can vary focus by varying direction Synthetic FarSynthetic Near Note: this is only works exactly in the limit of small source apertures, but works OK for finite apertures.

Synthetic Aperture Focus: Aliasing Aliasing artifacts can be caused by focal plane mismatch Synthetic FarSynthetic Near Point sampling along this plane causes aliasing artifacts Blurring along this plane due to source focal length

Variable Focal Plane Example Vary reconstructed camera’s focal length: just a matter of changing the directions before aperture assembly. (Illustration: Isaksen, MIT)

Advantages of Synthetic Aperture: Can simulate a huge aperture Impractical with a conventional camera Can even tilt focal plane Impossible with conventional optics! (Illustration: Isaksen, MIT)

Conclusions Lightfields are a unique way to represent the world Supports arbitrary light transport Equivalent to holograms & lenticular displays Isaksen et al.’s synthetic aperture technique allows lightfields to be refocused Opportunity to extract more information from lightfields