1. www.debevec.org Paul Debevec, Tim Hawkins, Chris Tchou, H.P. Duiker, Westley Sarokin, and Mark Sagar Acquiring the Reflectance Field of a Human Face UC Berkeley / USC Institute for Creative Technologies / LifeF/X

2. Rendering with Natural Light Fiat Lux

4. Related Work Hanrahan and Krueger. Reflection from Layered Surfaces due to Subsurface Scattering. SIGGRAPH 93 Bregler et al. Video Rewrite. SIGGRAPH 97 Guenter et al. Making Faces. SIGGRAPH 98 Pighin et al. Synthesizing Realistic Facial Expressions from Photographs. SIGGRAPH 98 Sagar et al. The Jester. SIGGRAPH 99 ET Marschner et al. Reflectance Measurements of Human Skin. 1999 Hanrahan and Krueger. Reflection from Layered Surfaces due to Subsurface Scattering. SIGGRAPH 93 Bregler et al. Video Rewrite. SIGGRAPH 97 Guenter et al. Making Faces. SIGGRAPH 98 Pighin et al. Synthesizing Realistic Facial Expressions from Photographs. SIGGRAPH 98 Sagar et al. The Jester. SIGGRAPH 99 ET Marschner et al. Reflectance Measurements of Human Skin. 1999

5. The Reflectance Field

8. R i ( u i,v i,  i,  i ) incident light field

9. The Reflectance Field R r ( u r,v r,  r,  r ) R i ( u i,v i,  i,  i ) incident light field radiant light field

10. The Reflectance Field R i ( u i,v i,  i,  i ; u r,v r,  r,  r ) 8D reflectance field

11. R i ( u i,v i,  i,  i ; u r,v r,  r,  r ) 4D Slices of the 8D Reflectance Field

12. The Light Stage

13. The Light Stage: 60-second exposure

14. Light Stage Data Original Resolution: 64  32 Lighting through image recombination: Haeberli ‘92, Nimeroff ‘94, Wong ‘97

15. Video

16. Light Stage Results Environments from the Light Probe Image Gallery www.debevec.org Environments from the Light Probe Image Gallery www.debevec.org

17. Reflectance Functions

18. Lighting Reflectance Functions normalized light map reflectance function lighting product rendered pixel 1 1 DCT Basis Smith and Rowe. Compressed domain processing of JPEG-encoded images. 1996

19. Interactive Lighting Demo SIGGRAPH 2000 Creative Applications Laboratory

21. Changing the Viewpoint

22. Reflection of Light from Skin Specular Component: Color of light, shiny, brighter near grazing, maintains polarization Subsurface Component: Color of skin, diffuse, desaturated near grazing, scrambles polarization Specular Component: Color of light, shiny, brighter near grazing, maintains polarization Subsurface Component: Color of skin, diffuse, desaturated near grazing, scrambles polarization (After Hanrahan and Krueger 93) After Hanrahan ‘93

23. Separating Reflectance Components using Crossed Polarizers Normal Image Subsurface Component Specular Component Colorspace techniques - Sato ‘94, Nayar ‘97

24. Transforming a Reflectance Function Subsurface Component Specular Component Final RF Comparison RF Shifted and Scaled Specular => Torrance- Sparrow microfacet distribution Surface Normal Estimate Original RF

25. Point-Source Comparison Original Image Novel Viewpoint

26. Spatially-Varying Reflectance Parameters Surface Normals n Diffuse Albedo  d Specular Intensity k s Specular Roughness 

27. Video

28. Compositing Test Original Image Light Probe Rendered Face Composite

29. 4. Ongoing Work Animate the faces Capture more spectral samples Use high-speed cameras to achieve real-time capture Animate the faces Capture more spectral samples Use high-speed cameras to achieve real-time capture

30. 5. Conclusion We have presented:  The light stage apparatus for capturing slices of the reflectance field of the human face  A technique for correctly relighting faces and objects with arbitrary illumination  A technique for extrapolating human reflectance to novel viewpoints We have presented:  The light stage apparatus for capturing slices of the reflectance field of the human face  A technique for correctly relighting faces and objects with arbitrary illumination  A technique for extrapolating human reflectance to novel viewpoints

31. Thanks Digital Media Innovation Program Interactive Pictures Corporation Alias|Wavefront UCB Digital Digital Media/New Genre Program ONR/BMDO Cornell Program of Computer Graphics Berkeley Millennium Project Digital Media Innovation Program Interactive Pictures Corporation Alias|Wavefront UCB Digital Digital Media/New Genre Program ONR/BMDO Cornell Program of Computer Graphics Berkeley Millennium Project and Shawn Brixey, Bill Buxton, Larry Rowe, Jessica Vallot, Patrick Wilson, Melanie Levine, Eric Paulos, Christine Waggoner, Holly Cim, Eliza Ra, Bryan Musson, David Altenau, Marc Levoy, Maryann Simmons, Henrik Wann Jensen, Don Greenberg, Pat Hanrahan, Randal Kleiser, Chris Bregler, Michael Naimark, Dan Maas, Steve Marschner, and Kevin Binkert. and Shawn Brixey, Bill Buxton, Larry Rowe, Jessica Vallot, Patrick Wilson, Melanie Levine, Eric Paulos, Christine Waggoner, Holly Cim, Eliza Ra, Bryan Musson, David Altenau, Marc Levoy, Maryann Simmons, Henrik Wann Jensen, Don Greenberg, Pat Hanrahan, Randal Kleiser, Chris Bregler, Michael Naimark, Dan Maas, Steve Marschner, and Kevin Binkert.

35. Overview 1.Related work 2.Relighting faces (from static viewpoints) 3.Changing the viewpoint 4.Ongoing work 5.Conclusion 1.Related work 2.Relighting faces (from static viewpoints) 3.Changing the viewpoint 4.Ongoing work 5.Conclusion

36. Modeling Indirect Illumination from Clothing

37. Light Stage Results

39. Modulated Images Original Resolution: 64  32

40. In-plane Reflectometry Measurements  Subsurface exhibits chromaticity falloff  Specular is monochromatic with Torrance- Sparrow microfacet behavior  Subsurface exhibits chromaticity falloff  Specular is monochromatic with Torrance- Sparrow microfacet behavior Subsurface Subsurface + Specular ii ii rr rr ii ii rr rr

41. Reflectometry Experiment

42. The Jester SIGGRAPH 99 Electronic Theater Mark Sagar et al. - LifeF/X, Inc. Performance and Text: Jessica Vallot

43. Changing the Viewpoint Model from Structured Lighting

44. Traditional Approach  Derive reflectance parameters for each point on the face’s surface  Map the parameters onto a geometric model of the face  Render using traditional methods  Derive reflectance parameters for each point on the face’s surface  Map the parameters onto a geometric model of the face  Render using traditional methods

45. Reflectance Function Mosaic

46. Goals  Investigate spatially varying skin reflectance for realistic skin rendering  Develop lighting techniques for illuminating real actors  Investigate spatially varying skin reflectance for realistic skin rendering  Develop lighting techniques for illuminating real actors

47. 1. Relighting Faces

48. Basic Approach Directly record how the face, as a whole, interacts with light, i.e. Record how the face transforms an incident field of illumination into a radiant field of illumination, i.e. Acquire the face’s 8-D Reflectance Field as fully as possible Directly record how the face, as a whole, interacts with light, i.e. Record how the face transforms an incident field of illumination into a radiant field of illumination, i.e. Acquire the face’s 8-D Reflectance Field as fully as possible R = R( R i,R r ) = R( u i,v i,  i,  i ; u r,v r,  r,  r )