Image-based Rendering of Real Objects with Complex BRDFs

Intensity of One Pixel s 1 (  ,   ) q p Consider the measured intensity at one pixel I 1 (  1,  1 ) as the isotropic point source is moved over the surface. I 1 (  ,   ) Note similarity to (Levoy, Hanrahan, 1996) (Levoy, Hanrahan, 1996) (Gortler et al, 1996) (Gortler et al, 1996)

Phong Intensity of One Pixel: I 1 (  ,   ) This is effectively a 2-D slice of a point’s BRDF except for Shadowing Shadowing 1/r 2 falloff from the source 1/r 2 falloff from the source

Intensity of One Pixel: I 1 (  ,   ) This is effectively a 2-D slice of a point’s BRDF except for Shadowing Shadowing 1/r 2 falloff from the source 1/r 2 falloff from the source

Image Acquisition

Intensity Over Second Surface s 1 (  ,   ) s 2 (  ,   ) p Now, consider moving an isotropic point source over a second surface and measuring the intensity of the same pixel: I 2 (  ,   ) I 1 (  ,   ) I 2 (  ,   )

I 1 (  ,   ) and I 2 (  ,   ) Inner Sphere: I 1 (  ,   ) Outer Sphere: I 2 (  ,   )

Relation Between Intensity Maps When the surface point p, When the surface point p, s 1 (     ) and s 2 (     ) are collinear (in correspondence), the measured pixel intensities are simply related by the relative 1/r 2 losses. s 1 (  ,   ) s 2 (  ,   ) p

Depth Estimation s 1 (  ,   ) s 2 (  ,   ) p( ) This correspondence can be expressed as a change of coordinates  2 (     ; )and  2 (      ; ) parameterized by depth This correspondence can be expressed as a change of coordinates  2 (     ; ) and  2 (      ; ) parameterized by depth. We can then estimate by minimizing We can then estimate by minimizing: O( )= [I 2 (  2 ( ),  2 ( )) - r 2 I 1 (  1,  1 ) ] 2 d  1 d  1

A Reconstructed Depth Map 143 Images on each surface

Rendering Synthetic Images: Point Sources New light position Intersection with the sphere Intersect light ray through P with sphere. Intersect light ray through P with sphere. Find triangle of light sources containing P. Find triangle of light sources containing P. Interpolate pixel intensities of images corresponding to the triangle vertices. Interpolate pixel intensities of images corresponding to the triangle vertices. For a given image point, there is a scene point: P For a given image point, there is a scene point: P P

Rendered Images

Rendered Image: A Sea Shell Isotropic point light source located between acquisition spheres.

Rendered Image: A Pear Two light sources Two light sources Point source to the left Point source to the left 3 by 5 cm area source 3 by 5 cm area source to the right to the right

Video Compositing of Real Objects Video Frame #567 Radiance Map Frame #567

Video Compositing Background Image #2313Object Image #2313

Video Compositing Composite Frame #567

Lighting Sensitive Displays Shree Nayar Peter Belhumeur Terry Boult Columbia Yale Lehigh Computer Vision Laboratory Columbia University Sponsor: NSF ITR

Displays Everywhere

But, Displays are Passive brightness contrast

display content Lighting Sensitive Display (LSD) Senses the Environmental Illumination Modifies Displayed Content Accordingly illumination : Perception : Reaction

State of the Art brightness contrast photodetector adjustment Heijligers 62 ; Thomas 63; Gibson 64; Korda 65; Biggs 65; Szermy 68 Newman 72; Constable 78; Fitzgibbon 82; Antwerp 85; Otenstein 93

Display’s Illumination Field display content display content (s,t) (u,v) L(s,t,u,v,  Wide Range of Sources: Sunlight, Overcast, Halogen, Fluorescent... Arbitrarily Complex : Point/Extended/Multiple Sources, Scene Radiance... (s,t) (u,v) L(s,t,u,v,  Four-Dimensional Ray Manifold

Methods for Sensing the Illumination Field photodetectors optical fibers hemispherical camera ? ?

probe video Compact Hemispherical Illumination Probe compact wide angle optics color video camera neutral density filters

LSD Prototype Sony 15” LCD Flat Display Hemispherical Probe Camera Matting Wooden Frame

Content Modification : Rendering Power Efficiency Brighter in Sunlight Dimmer Indoors Compensation Spatially Varying Brightness Spatially Varying Color Photorealism Consistent Colors and Shadings Consistent Highlights and Shadows All Modifications in Real-Time

Rendering Using Explicit Models: 2D+ v s1 n s2 O viewer source display rendered image content: surface Algorithms: Ray Tracing, Radiosity

Rendering Using Explicit Models: 3D v s1 n s2 O viewer source display rendered image content: shape, BRDF Algorithms: Ray Tracing, Radiosity

Image based Rendering probe camera capture camera Off-line Scene Capture (with Kudelka and Swaminathan)

Efficient Representation and Rendering Image Bases E source directions Captured Images i=1 i=4096 I k=1 k=10 blocks x basis 30 x 40 x 10 Lighting Coefficient Vectors L source directions i=1 i=4096 SVD

Efficient Representation and Rendering Compressed Coefficient Vectors Coeff. Bases U source directions Coefficient Vectors L i=1 i=4096 V q=1 b=200 blocks x basis 30 x 40 x 10 SVD source directions i=1 i=4096

Real-Time Rendering Compressed Coefficient Vector Coeff Eigenvectors U Illumination Field Vs Coefficient Vector Image Eigenvectors E Display I Compressed Coefficient Vectors V s X X X UVs

Efficient Representation and Rendering Captured Data Compute Local Subspaces Local Bases and Coefficients source direction Image Reconstruction Display Illumination Field Display 4 Gb 10 Mb 8 fps (laptop)

Efficient Representation and Rendering Captured Data Compute Local Subspaces Local Bases and Coefficients source direction Image Reconstruction Display Illumination Field Display 4 Gb 10 Mb 8 fps (laptop)

Face

Still Life: Scene Capture

Still Life

Summary Lighting Sensitive Display: Senses Environmental Illumination Modifies Displayed Content Applications: Compensation: Computers, PDA’s, Televisions, Billboards Photorealism: Digital Art, E-Commerce, Future Homes

Capturing Scenes for Image based Rendering probe camera capture camera