Paul Debevec, Tim Hawkins, Chris Tchou, H.P. Duiker, Westley Sarokin, and Mark Sagar Acquiring the Reflectance Field of a Human Face UC Berkeley / USC Institute for Creative Technologies / LifeF/X

Rendering with Natural Light Fiat Lux

Related Work Hanrahan and Krueger. Reflection from Layered Surfaces due to Subsurface Scattering. SIGGRAPH 93 Bregler et al. Video Rewrite. SIGGRAPH 97 Guenter et al. Making Faces. SIGGRAPH 98 Pighin et al. Synthesizing Realistic Facial Expressions from Photographs. SIGGRAPH 98 Sagar et al. The Jester. SIGGRAPH 99 ET Marschner et al. Reflectance Measurements of Human Skin Hanrahan and Krueger. Reflection from Layered Surfaces due to Subsurface Scattering. SIGGRAPH 93 Bregler et al. Video Rewrite. SIGGRAPH 97 Guenter et al. Making Faces. SIGGRAPH 98 Pighin et al. Synthesizing Realistic Facial Expressions from Photographs. SIGGRAPH 98 Sagar et al. The Jester. SIGGRAPH 99 ET Marschner et al. Reflectance Measurements of Human Skin. 1999

The Reflectance Field

R i ( u i,v i,  i,  i ) incident light field

The Reflectance Field R r ( u r,v r,  r,  r ) R i ( u i,v i,  i,  i ) incident light field radiant light field

The Reflectance Field R i ( u i,v i,  i,  i ; u r,v r,  r,  r ) 8D reflectance field

R i ( u i,v i,  i,  i ; u r,v r,  r,  r ) 4D Slices of the 8D Reflectance Field

The Light Stage

The Light Stage: 60-second exposure

Light Stage Data Original Resolution: 64  32 Lighting through image recombination: Haeberli ‘92, Nimeroff ‘94, Wong ‘97

Video

Light Stage Results Environments from the Light Probe Image Gallery Environments from the Light Probe Image Gallery

Reflectance Functions

Lighting Reflectance Functions normalized light map reflectance function lighting product rendered pixel 1 1 DCT Basis Smith and Rowe. Compressed domain processing of JPEG-encoded images. 1996

Interactive Lighting Demo SIGGRAPH 2000 Creative Applications Laboratory

Changing the Viewpoint

Reflection of Light from Skin Specular Component: Color of light, shiny, brighter near grazing, maintains polarization Subsurface Component: Color of skin, diffuse, desaturated near grazing, scrambles polarization Specular Component: Color of light, shiny, brighter near grazing, maintains polarization Subsurface Component: Color of skin, diffuse, desaturated near grazing, scrambles polarization (After Hanrahan and Krueger 93) After Hanrahan ‘93

Separating Reflectance Components using Crossed Polarizers Normal Image Subsurface Component Specular Component Colorspace techniques - Sato ‘94, Nayar ‘97

Transforming a Reflectance Function Subsurface Component Specular Component Final RF Comparison RF Shifted and Scaled Specular => Torrance- Sparrow microfacet distribution Surface Normal Estimate Original RF

Point-Source Comparison Original Image Novel Viewpoint

Spatially-Varying Reflectance Parameters Surface Normals n Diffuse Albedo  d Specular Intensity k s Specular Roughness 

Video

Compositing Test Original Image Light Probe Rendered Face Composite

4. Ongoing Work Animate the faces Capture more spectral samples Use high-speed cameras to achieve real-time capture Animate the faces Capture more spectral samples Use high-speed cameras to achieve real-time capture

5. Conclusion We have presented:  The light stage apparatus for capturing slices of the reflectance field of the human face  A technique for correctly relighting faces and objects with arbitrary illumination  A technique for extrapolating human reflectance to novel viewpoints We have presented:  The light stage apparatus for capturing slices of the reflectance field of the human face  A technique for correctly relighting faces and objects with arbitrary illumination  A technique for extrapolating human reflectance to novel viewpoints

Thanks Digital Media Innovation Program Interactive Pictures Corporation Alias|Wavefront UCB Digital Digital Media/New Genre Program ONR/BMDO Cornell Program of Computer Graphics Berkeley Millennium Project Digital Media Innovation Program Interactive Pictures Corporation Alias|Wavefront UCB Digital Digital Media/New Genre Program ONR/BMDO Cornell Program of Computer Graphics Berkeley Millennium Project and Shawn Brixey, Bill Buxton, Larry Rowe, Jessica Vallot, Patrick Wilson, Melanie Levine, Eric Paulos, Christine Waggoner, Holly Cim, Eliza Ra, Bryan Musson, David Altenau, Marc Levoy, Maryann Simmons, Henrik Wann Jensen, Don Greenberg, Pat Hanrahan, Randal Kleiser, Chris Bregler, Michael Naimark, Dan Maas, Steve Marschner, and Kevin Binkert. and Shawn Brixey, Bill Buxton, Larry Rowe, Jessica Vallot, Patrick Wilson, Melanie Levine, Eric Paulos, Christine Waggoner, Holly Cim, Eliza Ra, Bryan Musson, David Altenau, Marc Levoy, Maryann Simmons, Henrik Wann Jensen, Don Greenberg, Pat Hanrahan, Randal Kleiser, Chris Bregler, Michael Naimark, Dan Maas, Steve Marschner, and Kevin Binkert.

Overview 1.Related work 2.Relighting faces (from static viewpoints) 3.Changing the viewpoint 4.Ongoing work 5.Conclusion 1.Related work 2.Relighting faces (from static viewpoints) 3.Changing the viewpoint 4.Ongoing work 5.Conclusion

Modeling Indirect Illumination from Clothing

Light Stage Results

Modulated Images Original Resolution: 64  32

In-plane Reflectometry Measurements  Subsurface exhibits chromaticity falloff  Specular is monochromatic with Torrance- Sparrow microfacet behavior  Subsurface exhibits chromaticity falloff  Specular is monochromatic with Torrance- Sparrow microfacet behavior Subsurface Subsurface + Specular ii ii rr rr ii ii rr rr

Reflectometry Experiment

The Jester SIGGRAPH 99 Electronic Theater Mark Sagar et al. - LifeF/X, Inc. Performance and Text: Jessica Vallot

Changing the Viewpoint Model from Structured Lighting

Traditional Approach  Derive reflectance parameters for each point on the face’s surface  Map the parameters onto a geometric model of the face  Render using traditional methods  Derive reflectance parameters for each point on the face’s surface  Map the parameters onto a geometric model of the face  Render using traditional methods

Reflectance Function Mosaic

Goals  Investigate spatially varying skin reflectance for realistic skin rendering  Develop lighting techniques for illuminating real actors  Investigate spatially varying skin reflectance for realistic skin rendering  Develop lighting techniques for illuminating real actors

1. Relighting Faces

Basic Approach Directly record how the face, as a whole, interacts with light, i.e. Record how the face transforms an incident field of illumination into a radiant field of illumination, i.e. Acquire the face’s 8-D Reflectance Field as fully as possible Directly record how the face, as a whole, interacts with light, i.e. Record how the face transforms an incident field of illumination into a radiant field of illumination, i.e. Acquire the face’s 8-D Reflectance Field as fully as possible R = R( R i,R r ) = R( u i,v i,  i,  i ; u r,v r,  r,  r )