Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs Computer Science Division University of California at Berkeley Computer Science Division University of California at Berkeley Yizhou Yu, Paul Debevec, Jitendra Malik & Tim Hawkins

Image-based Modeling and Rendering 1st Generation---- vary viewpoint but not lighting –Recover geometry ( explicit or implicit ) –Acquire photographs –Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, (Light Field Rendering) etc. 1st Generation---- vary viewpoint but not lighting –Recover geometry ( explicit or implicit ) –Acquire photographs –Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, (Light Field Rendering) etc.

Image-based Modeling and Rendering Photographs are not Reflectance Maps ! 2nd Generation---- vary viewpoint and lighting for non-diffuse scenes –Recover geometry –Recover reflectance properties –Render using light transport simulation Photographs are not Reflectance Maps ! 2nd Generation---- vary viewpoint and lighting for non-diffuse scenes –Recover geometry –Recover reflectance properties –Render using light transport simulation Illumination Radiance Reflectance

Previous Work BRDF Measurement in the Laboratory –[ Ward 92 ], [Dana, Ginneken, Nayar & Koenderink 97] Isolated Objects under Direct Illumination –[ Sato, Wheeler & Ikeuchi 97 ] Isolated Objects under General Illumination –[ Yu & Malik 98], [ Debevec 98] BRDF Measurement in the Laboratory –[ Ward 92 ], [Dana, Ginneken, Nayar & Koenderink 97] Isolated Objects under Direct Illumination –[ Sato, Wheeler & Ikeuchi 97 ] Isolated Objects under General Illumination –[ Yu & Malik 98], [ Debevec 98]

The Problem General case of multiple objects under mutual illumination has not been studied.

Global Illumination Reflectance Properties Radiance Images Geometry Illumination

Inverse Global Illumination Reflectance Properties Radiance Images Geometry Illumination

Input Radiance Images [ Debevec & Malik 97]

In Detail...

Geometry and Camera Positions

Light Sources

Synthesized Images Original LightingNovel Lighting

OutlineOutline Diffuse surfaces under mutual illumination Non-diffuse surfaces under direct illumination Non-diffuse surfaces under mutual illumination Diffuse surfaces under mutual illumination Non-diffuse surfaces under direct illumination Non-diffuse surfaces under mutual illumination

Lambertian Surfaces under Mutual Illumination B i, B j, E i measured Form-factor F ij known Solve for diffuse albedo Source Target

Parametric BRDF Model [ Ward 92 ] Isotropic Kernel Anisotropic Kernel N H ( 3 parameters) ( 5 parameters)

Non-diffuse Surfaces under Direct Illumination N H P1P1 P2P2 P1P1 P2P2

Non-diffuse Surfaces under Mutual Illumination L P i A j is not known. ( unlike diffuse case, where L P i A j = L C k A j ) CvCv CkCk AjAj PiPi LPiAjLPiAj LCkAjLCkAj LCvPiLCvPi Source Target

Solution: iteratively estimate specular component. Initialize Repeat –Estimate BRDF parameters for each surface –Update and

Estimation of Specular Difference S Estimate specular component of by Monte Carlo ray-tracing using current guess of reflectance parameters. Similarly for Difference gives S CvCv CkCk AjAj PiPi LPiAjLPiAj LCkAjLCkAj LCvPiLCvPi LPiAjLPiAj LCkAjLCkAj

Recovering Diffuse Albedo Maps Specular properties assumed uniform across each surface, but diffuse albedo allowed to vary. Specular properties assumed uniform across each surface, but diffuse albedo allowed to vary.

ResultsResults A simulated cubical room

Results for the Simulated Case Diffuse AlbedoSpecular Roughness

ResultsResults A real conference room

Real vs. Synthetic for Original Lighting Real Synthetic

Diffuse Albedo Maps of Identical Posters in Different Positions Poster APoster BPoster C

Inverting Color Bleed Input Photograph Output Albedo Map

Real vs. Synthetic for Novel Lighting Real Synthetic

VideoVideo

AcknowledgmentsAcknowledgments Thanks to David Culler and the Berkeley NOW project, Tal Garfinkel, Gregory Ward Larson, Carlo Sequin. Supported by ONR BMDO, the California MICRO program, Philips Corporation, Interval Research Corporation and Microsoft Graduate Fellowship. Thanks to David Culler and the Berkeley NOW project, Tal Garfinkel, Gregory Ward Larson, Carlo Sequin. Supported by ONR BMDO, the California MICRO program, Philips Corporation, Interval Research Corporation and Microsoft Graduate Fellowship.

ConclusionsConclusions A digital camera can undertake all the data acquisition tasks involved. Both specular and high resolution diffuse reflectance properties can be recovered from photographs. Reflectance recovery can re-render non-diffuse real scenes under novel illumination as well as from novel viewpoints. A digital camera can undertake all the data acquisition tasks involved. Both specular and high resolution diffuse reflectance properties can be recovered from photographs. Reflectance recovery can re-render non-diffuse real scenes under novel illumination as well as from novel viewpoints.