CSL 859: Advanced Computer Graphics Dept of Computer Sc. & Engg. IIT Delhi

Image-Based Rendering So far: So far: Geometry -> images Geometry -> images Object space model, even volumetric Object space model, even volumetric Image-based rendering: Image-based rendering: Image -> Another image Image -> Another image Zoom, Pan etc. Zoom, Pan etc. Just image processing? Just image processing?

Images with depth Quicktime VR: Quicktime VR: 2D panoramic photograph 2D panoramic photograph Spin around, zoom in and out Spin around, zoom in and out Can add objects closer to viewer Can add objects closer to viewer Tour into the picture Tour into the picture Assign depth to parts of the image Assign depth to parts of the image One might add objects hidden behind some object in the image One might add objects hidden behind some object in the image Layered depth images Layered depth images

Image Based Rendering Store image from every conceivable view Store image from every conceivable view Rendering would reduce to database query Rendering would reduce to database query Generality demand infinite sized database Generality demand infinite sized database Could store enough images Could store enough images Given a desired viewpoint (viewmatrix) Given a desired viewpoint (viewmatrix) Choose an image from a saved view near the desired view Choose an image from a saved view near the desired view Warp the image Warp the image Or, interpolate from nearby known viewpoints Or, interpolate from nearby known viewpoints

Ray Equation

Correspondence Warp x 1 to x 2 +

General 3D Warp [Courtesy L Mcmillan]

Occlusion Determination Project the desired center-of- projection onto the reference image Project the desired center-of- projection onto the reference image

Occlusion Determination Draw towards the projected point Draw towards the projected point Guarantees painter’s ordering Guarantees painter’s ordering Independent of the scene's contents Independent of the scene's contents Generalizes to non- planar viewing surfaces Generalizes to non- planar viewing surfaces

Reconstruction

Radiances in a Scene Account for all rays Account for all rays Origin Origin 3 dimensions 3 dimensions Direction Direction 2 dimensions 2 dimensions Space of rays is 5 dimensional Space of rays is 5 dimensional

Panorama All rays from a single point

Plenoptic Function p = P(Θ, Φ, x, y, z, λ, t) All rays from all points Courtesy L. Mcmillan

Radiances in a Scene II Account for all rays Account for all rays Origin Origin 3 dimensions 3 dimensions Direction Direction 2 dimensions 2 dimensions Space of rays is 5 dimensional Space of rays is 5 dimensional Radiance is constant along ray Radiance is constant along ray 4 dimensional space 4 dimensional space Subject to occlusion Subject to occlusion

Capturing Radiances Capture images from many places Capture images from many places Camera positioning Camera positioning Parameterize the 4D space Parameterize the 4D space Camera position and 2D image? Camera position and 2D image? Sample the 4D space Sample the 4D space Coverage and sampling uniformity Coverage and sampling uniformity Aliasing Aliasing Too much data Too much data

Representing Scene Radiance Like texture map Like texture map Except ray origin is not fixed Except ray origin is not fixed Source and destination of ray varies Source and destination of ray varies 2 coordinates (u,v) for ray origin 2 coordinates (u,v) for ray origin 2 coordinates (s,t) for ray destination 2 coordinates (s,t) for ray destination s t u v [Light-field: Hanrahan & Levoy]

Sampling Coverage θ r θ r With four slabs the (r,θ) space is well covered (for an outside looking in case) Ray Source Ray Target

Stanford Multi-camera Array 640 × 480 pixels × 30 fps × 128 cameras 640 × 480 pixels × 30 fps × 128 cameras Synchronized timing Synchronized timing Continuous streaming Continuous streaming Flexible arrangement Flexible arrangement

Light Field as Array of Images

For each pixel (x, y) For each pixel (x, y) Compute ray Compute ray Map to (u,v,s,t) Map to (u,v,s,t) Look up “4D” texture Look up “4D” texture Store as many 2D textures Store as many 2D textures Quadri-linear interpolation Quadri-linear interpolation Rendering of Light Fields

Good and Bad Advantages: Advantages: Simpler computation vs. traditional CG Simpler computation vs. traditional CG Cost independent of scene complexity Cost independent of scene complexity Cost independent of material properties and other optical effects Cost independent of material properties and other optical effects Disadvantages: Disadvantages: Static geometry Static geometry Fixed lighting Fixed lighting High storage cost High storage cost