CS 395: Adv. Computer Graphics Light Fields and their Approximations Jack Tumblin

for a given scene, describe:for a given scene, describe: –ALL rays through –ALL pixels, of –ALL cameras, at –ALL wavelengths, for –ALL time F(x,y,z, , ,, t) Plenoptic Function (Adelson, Bergen `91)

for a given scene, describe:for a given scene, describe: –ALL rays through –ALL pixels, of –ALL cameras, at –ALL wavelengths, for –ALL time F(x,y,z, , ,, t)

Plenoptic Function (Adelson, Bergen `91) for a given scene, describe:for a given scene, describe: –ALL rays through –ALL pixels, of –ALL cameras, at –ALL wavelengths, –ALL time F(x,y,z, , ,, t)

Plenoptic Function (Adelson, Bergen `91) ! Complete !! Complete ! –Geometry, Lighting, BRDF,… ! NOT REQUIRED ! Preposterously Huge: (?!?! 7-D function sampled at image resolution !?!?), butPreposterously Huge: (?!?! 7-D function sampled at image resolution !?!?), but Hugely Redundant:Hugely Redundant: –Wavelength  RGB triplets, ignore Time, and Restrict eyepoint movement: maybe ~3 or 4D ? –Very Similar images—use Warping rules? –Exploit Movie Storage/Compression Methods

The Big Idea: Image-Based Modeling & Rendering Computer Graphics: given scene description, synthesize imagesComputer Graphics: given scene description, synthesize images Digital Photography: given scene, capture imagesDigital Photography: given scene, capture images Computer Vision: given images, assumptions,… (& hope) describe sceneComputer Vision: given images, assumptions,… (& hope) describe scene Image-Based Modeling & Rendering: Collect images to estimate Plenoptic Function QUICKLY interpolate / recall images (more storage, but less computing…)Image-Based Modeling & Rendering: Collect images to estimate Plenoptic Function QUICKLY interpolate / recall images (more storage, but less computing…)

OLDEST IBR: Shadow Maps (1984) Fast Shadows from Z-buffer hardware: 1) Make the “Shadow Map”: –Render image seen at light source, BUT –Keep ONLY the Z-buffer values (depth) 2) Render Scene from Eyepoint: –Pixel + Z depth gives 3D position of surface; –Project 3D position into Shadow map –If Shadow Map depth < 3D depth, SHADOW! MANY sampling hacks (See Eric Haines) nVidia / OpenGL implementations available

Early IBR: QuickTime VR (Chen, Williams ’93) 1) Four Planar Images  1 Cylindrical Panorama: Re-sampling Required! Planar Pixels: equal distance on x,y plane (tan -1  ) Cyl. Pixels: horiz: equal angle on cylinder (  ) vert: equal distance on y (tan -1  ) Cyl. Pixels: horiz: equal angle on cylinder (  ) vert: equal distance on y (tan -1  )

Early IBR: QuickTime VR (Chen, Williams ’93) 1) Four Planar Images  1 Cylindrical Panorama: IN:OUT:

Early IBR: QuickTime VR (Chen, Williams ’93) 2) Windowing, Horizontal-only Reprojection: IN:OUT:

Plenoptic Array: ‘The Matrix Effect’ Simple arc, line, or ring array of camerasSimple arc, line, or ring array of cameras Synchronized shutterSynchronized shutter Warp/blend between images to change viewpoint on ‘time-frozen’ scene:Warp/blend between images to change viewpoint on ‘time-frozen’ scene:

View Interpolation: How? Traditional Stereo Disparity Map: pixel-by-pixel search for correspondenceTraditional Stereo Disparity Map: pixel-by-pixel search for correspondence (Lighter: more Horiz. Separation)

View Interpolation: How? Store Depth at each pixel: reprojectStore Depth at each pixel: reproject Coarse or Simple 3D model:Coarse or Simple 3D model:

View Interpolation Problems: Visibility ChangesVisibility Changes Disocclusion, or ‘holes in the picture’Disocclusion, or ‘holes in the picture’ Partial Answers: More PicturesMore Pictures Interpolate Nearby PixelsInterpolate Nearby Pixels Better: Multiple Samples (LDI Trees SIGG99 )Better: Multiple Samples (LDI Trees SIGG99 ) L view R view

Plenoptic Modeling (McMillan`95) Alerted CGI community to notion of Plenoptic FunctionAlerted CGI community to notion of Plenoptic Function Solved ‘depth order’ problem: read images in occlusion-compatible orderSolved ‘depth order’ problem: read images in occlusion-compatible order Demonstrated reprojection from two spherical projectionsDemonstrated reprojection from two spherical projections

Seitz: ‘View Morphing’ SIGG` )Manually set some corresp.points (eye corners, etc.) 2) pre-warp and post-warp to match points in 3D, 3) Reproject for Virtual cameras

Seitz: ‘View Morphing’ SIGG`96

Seitz: ‘View Morphing’ SIGG`96

Seitz: ‘View Morphing’ SIGG`96

Seitz: ‘View Morphing’ SIGG`96

Light Fields / Lumigraphs Subset of Plenoptic Function:Subset of Plenoptic Function: –Let Convex Surface surround scene –Place in-facing camera at every surface point –Captures ALL viewing rays for object! –(But does NOT separate external light efx)

Light Fields / Lumigraphs Two-plane parameterization: –Generalized camera: TWO image planes –Sets angle AND position – u v s t L(u,v,s,t)

Light Fields / Lumigraphs. u

Hierarchical Layered Depth Images A ‘stack’ of unstructured 3-D sample pointsA ‘stack’ of unstructured 3-D sample points Stores, retrieves data at multiple resolutionsStores, retrieves data at multiple resolutions Eliminates redundant storageEliminates redundant storage Can be updated dynamically with new data (from renderer or camera)Can be updated dynamically with new data (from renderer or camera) Demonstrated for real-time walkthroughs…Demonstrated for real-time walkthroughs…

Hybrid Approaches Simple Initial Geometry, ‘decorated’ by image-based models (Debevec`96)Simple Initial Geometry, ‘decorated’ by image-based models (Debevec`96) ‘Voxel-Carving’‘Voxel-Carving’ ‘Image-based Visual Hull’ SIGG. 2000…‘Image-based Visual Hull’ SIGG. 2000…

IBR-Motivating Opinions “Computer Graphics: Hard” –Complex! geometry, texture, lighting, shadows, compositing, BRDF, interreflections, etc. etc., etc., … –Irregular! Visibility,Topology, Render Eqn.,… –Isolated! Tough to use real objects in CGI –Slow! compute-bound, off-line only,… “Digital Imaging: Easy” –Simple! More quality? Just pump more pixels! –Regular! Vectorized, compressible, pipelined… –Accessible! Use real OR synthetic (CGI) images! –Fast! Scalable, Image reuse, a path to interactivity…

Difficulties Sampling / Reconstruction problemsSampling / Reconstruction problems Brute Force: Bandwidth & Memory HogBrute Force: Bandwidth & Memory Hog Easier Display  Difficult EditingEasier Display  Difficult Editing Increased Flexibility  Reduced RegularityIncreased Flexibility  Reduced Regularity –more complex storage / retrieval, –more computing, –less deterministic (cache miss, etc.)

Interesting Recent Work ‘Surface Light Fields for 3D Photography’ Wood et al., SIGG Light Fields for 3D Photography’ Wood et al., SIGG ‘Plenoptic Stitching’ Aliaga et al. SIGG Stitching’ Aliaga et al. SIGG ‘Single-view Modeling’ Seitz,CVPR Modeling’ Seitz,CVPR2001