1. Image-based Rendering of Real Objects with Complex BRDFs

2. Intensity of One Pixel s 1 (  ,   ) q p Consider the measured intensity at one pixel I 1 (  1,  1 ) as the isotropic point source is moved over the surface. I 1 (  ,   ) Note similarity to (Levoy, Hanrahan, 1996) (Levoy, Hanrahan, 1996) (Gortler et al, 1996) (Gortler et al, 1996)

3. Phong Intensity of One Pixel: I 1 (  ,   ) This is effectively a 2-D slice of a point’s BRDF except for Shadowing Shadowing 1/r 2 falloff from the source 1/r 2 falloff from the source

4. Intensity of One Pixel: I 1 (  ,   ) This is effectively a 2-D slice of a point’s BRDF except for Shadowing Shadowing 1/r 2 falloff from the source 1/r 2 falloff from the source

5. Image Acquisition

6. Intensity Over Second Surface s 1 (  ,   ) s 2 (  ,   ) p Now, consider moving an isotropic point source over a second surface and measuring the intensity of the same pixel: I 2 (  ,   ) I 1 (  ,   ) I 2 (  ,   )

7. I 1 (  ,   ) and I 2 (  ,   ) Inner Sphere: I 1 (  ,   ) Outer Sphere: I 2 (  ,   )

8. Relation Between Intensity Maps When the surface point p, When the surface point p, s 1 (     ) and s 2 (     ) are collinear (in correspondence), the measured pixel intensities are simply related by the relative 1/r 2 losses. s 1 (  ,   ) s 2 (  ,   ) p

9. Depth Estimation s 1 (  ,   ) s 2 (  ,   ) p( ) This correspondence can be expressed as a change of coordinates  2 (     ; )and  2 (      ; ) parameterized by depth This correspondence can be expressed as a change of coordinates  2 (     ; ) and  2 (      ; ) parameterized by depth. We can then estimate by minimizing We can then estimate by minimizing: O( )= [I 2 (  2 ( ),  2 ( )) - r 2 I 1 (  1,  1 ) ] 2 d  1 d  1

10. A Reconstructed Depth Map 143 Images on each surface

11. Rendering Synthetic Images: Point Sources New light position Intersection with the sphere Intersect light ray through P with sphere. Intersect light ray through P with sphere. Find triangle of light sources containing P. Find triangle of light sources containing P. Interpolate pixel intensities of images corresponding to the triangle vertices. Interpolate pixel intensities of images corresponding to the triangle vertices. For a given image point, there is a scene point: P For a given image point, there is a scene point: P P

12. Rendered Images

13. Rendered Image: A Sea Shell Isotropic point light source located between acquisition spheres.

14. Rendered Image: A Pear Two light sources Two light sources Point source to the left Point source to the left 3 by 5 cm area source 3 by 5 cm area source to the right to the right

15. Video Compositing of Real Objects Video Frame #567 Radiance Map Frame #567

16. Video Compositing Background Image #2313Object Image #2313

17. Video Compositing Composite Frame #567

18. Lighting Sensitive Displays Shree Nayar Peter Belhumeur Terry Boult Columbia Yale Lehigh Computer Vision Laboratory Columbia University Sponsor: NSF ITR

19. Displays Everywhere

20. But, Displays are Passive brightness contrast

21. display content Lighting Sensitive Display (LSD) Senses the Environmental Illumination Modifies Displayed Content Accordingly illumination : Perception : Reaction

22. State of the Art brightness contrast photodetector adjustment Heijligers 62 ; Thomas 63; Gibson 64; Korda 65; Biggs 65; Szermy 68 Newman 72; Constable 78; Fitzgibbon 82; Antwerp 85; Otenstein 93

23. Display’s Illumination Field display content display content (s,t) (u,v) L(s,t,u,v,  Wide Range of Sources: Sunlight, Overcast, Halogen, Fluorescent... Arbitrarily Complex : Point/Extended/Multiple Sources, Scene Radiance... (s,t) (u,v) L(s,t,u,v,  Four-Dimensional Ray Manifold

24. Methods for Sensing the Illumination Field photodetectors optical fibers hemispherical camera ? ?

25. probe video Compact Hemispherical Illumination Probe compact wide angle optics color video camera neutral density filters

26. LSD Prototype Sony 15” LCD Flat Display Hemispherical Probe Camera Matting Wooden Frame

27. Content Modification : Rendering Power Efficiency Brighter in Sunlight Dimmer Indoors Compensation Spatially Varying Brightness Spatially Varying Color Photorealism Consistent Colors and Shadings Consistent Highlights and Shadows All Modifications in Real-Time

28. Rendering Using Explicit Models: 2D+ v s1 n s2 O viewer source display rendered image content: surface Algorithms: Ray Tracing, Radiosity

29. Rendering Using Explicit Models: 3D v s1 n s2 O viewer source display rendered image content: shape, BRDF Algorithms: Ray Tracing, Radiosity

30. Image based Rendering probe camera capture camera Off-line Scene Capture (with Kudelka and Swaminathan)

31. Efficient Representation and Rendering Image Bases E source directions 40 30 16 Captured Images i=1 i=4096 I 40 30 16 k=1 k=10 blocks x basis 30 x 40 x 10 Lighting Coefficient Vectors L source directions i=1 i=4096 SVD

32. Efficient Representation and Rendering Compressed Coefficient Vectors Coeff. Bases U source directions Coefficient Vectors L i=1 i=4096 V q=1 b=200 blocks x basis 30 x 40 x 10 SVD source directions i=1 i=4096

33. Real-Time Rendering Compressed Coefficient Vector Coeff Eigenvectors U Illumination Field Vs Coefficient Vector Image Eigenvectors E Display I Compressed Coefficient Vectors V s X X X UVs

34. Efficient Representation and Rendering Captured Data Compute Local Subspaces Local Bases and Coefficients source direction Image Reconstruction Display Illumination Field Display 4 Gb 10 Mb 8 fps (laptop)

35. Efficient Representation and Rendering Captured Data Compute Local Subspaces Local Bases and Coefficients source direction Image Reconstruction Display Illumination Field Display 4 Gb 10 Mb 8 fps (laptop)

36. Face

37. Still Life: Scene Capture

38. Still Life

41. Summary Lighting Sensitive Display: Senses Environmental Illumination Modifies Displayed Content Applications: Compensation: Computers, PDA’s, Televisions, Billboards Photorealism: Digital Art, E-Commerce, Future Homes

42. Capturing Scenes for Image based Rendering probe camera capture camera