1. CS 395/495-25: Spring 2003 IBMR: Week 9B Image-Based Physics: Measuring Light & Materials Jack Tumblin jet@cs.northwestern.edu

2. Reminders ProjA graded: Good Job! 90,95, 110ProjA graded: Good Job! 90,95, 110 ProjB graded: Good! minor H confusions.ProjB graded: Good! minor H confusions. MidTerm graded: novel solutions encouraged.MidTerm graded: novel solutions encouraged. ProjC due Friday, May 16: grading done.ProjC due Friday, May 16: grading done. ProjD posted, due Friday May 30ProjD posted, due Friday May 30 Take-Home Final Exam: Assign on Thurs June 5, due June 11Take-Home Final Exam: Assign on Thurs June 5, due June 11

3. IBMR: Measure,Create, Modify Light How can we measure ‘rays’ of light? Light Sources? Scattered rays? etc. Shape,Position,Movement, BRDF,Texture,Scattering EmittedLight Reflected,Scattered, Light … Cameras capture subset of these rays. Digital light sources (Projectors) can produce a subset of these rays.

4. Cleaner Formulation:Cleaner Formulation: –Orthographic camera, –positioned on sphere around object/scene –Orthographic projector, –positioned on sphere around object/scene –(and wavelength and time) F(x c,y c,  c,  c,x l,y l  l,  l,, t) ‘Full 8-D Light Field’ (10-D, actually: time, ) camera projector

5. Light Measurement Terms Flux W = power, Watts, # photons/secFlux W = power, Watts, # photons/sec Irradiance E = Watts/area = dW/dAIrradiance E = Watts/area = dW/dA Radiance L = (Watts/area)/sr = (dW/dA)/srRadiance L = (Watts/area)/sr = (dW/dA)/sr BRDF: Measure EMITTED radiance that results from INCOMING irradiance from just one direction: BRDF = F r = L e / E i = (Watts/area) / (Watts/area  sr)BRDF: Measure EMITTED radiance that results from INCOMING irradiance from just one direction: BRDF = F r = L e / E i = (Watts/area) / (Watts/area  sr)

6. IBMR Tools Digital Light Input:Digital Light Input: –Light meter: measure visible irradiance E –Camera: pixels measure Radiance L i ; flux arriving at lens from one (narrow solid) angle Digital Light Output:Digital Light Output: –Luminaires: point lights, extended(area) sources –Emissive Surfaces: CRT, LCD surface –Projectors: laser dot,stripe,scan; video display Light Modifiers (Digital?):Light Modifiers (Digital?): –Calibration objects, shadow sources, etc. –Lenses,diffusers, filters, reflectors, collimators... –?Where are the BRDF displays / printers?

7. What’s wrong with Images? What We Want What We Get

8. ???? 0 255 Domain of Human Vision: from ~10 -6 to ~10 +8 cd/m 2 Range of Typical Displays: from ~1 to ~100 cd/m 2 starlightmoonlight office light daylightflashbulb 10 -6 10 -2 110100 10 +4 10 +8 Problem:Map Scene to Display

12. High-Contrast Image Capture? An open problem! (esp. for video...)An open problem! (esp. for video...) Direct (expensive) solution:Direct (expensive) solution: –Flying Spot Radiometer: brute force instrument, costly, slow, delicate –Novel Image Sensors: line-scan cameras, logarithmic CMOS circuits, cooled detectors, rate-based detectors... Most widely used idea: multiple exposuresMost widely used idea: multiple exposures Elegant paper ( Debevec1996 ) describes how:Elegant paper ( Debevec1996 ) describes how:

13. starlightmoonlightoffice lightdaylightflashbulb Use Overlapped Exposure Values

14. starlightmoonlightoffice lightdaylightflashbulb

15. Use Overlapped Exposure Values starlightmoonlightoffice lightdaylightflashbulb

16. Use Overlapped Exposure Values starlightmoonlightoffice lightdaylightflashbulb What is the camera response curve? What are the pixel intensities? ?

17. Use Overlapped Exposure Values starlightmoonlightoffice lightdaylightflashbulb It’s a null-space problem! Input intensities = unknown vector x Input intensities = unknown vector x Film response R(x) = known film value vector b Film response R(x) = known film value vector b Multiple, Aligned, known Exposures: Multiple, Aligned, known Exposures: R 0 (x*2 0 ) = b 0 R 1 (x*2 1 ) = b 1 R 2 (x*2 2 ) = b 2 R 3 (x*2 3 ) = b 3... Rearrange, stack, solve with SVD. (see paper) Rearrange, stack, solve with SVD. (see paper)

18. Camera Abilities / Limitations Nonlinear Intensity Response: S-shapedNonlinear Intensity Response: S-shaped Low-Contrast Devices: Noise limitedLow-Contrast Devices: Noise limited Varied Spectral Response: RGB 1 != RGB 2...Varied Spectral Response: RGB 1 != RGB 2... Color Sensing Strategies:Color Sensing Strategies: –3-chip cameras: best, but expensive! –Mosaic sensor: trades resolution for color Nonuniform sensitivity & geometryNonuniform sensitivity & geometry –Lens limitations (vignetting, radial distortion, bloom/scatter, uneven focus,...) –CCD Sensor geometry: VERY exact, repeatable

19. Display Abilities / Limitations Nonlinear Intensity Response: r-shapedNonlinear Intensity Response: r-shaped Low-Contrast DevicesLow-Contrast Devices –scattering usually sets upper bounds –Best Contrast: laser projectors, some DLP devices, specialized devices...) Varied Spectral Response: RGB 1 != RGB 2...Varied Spectral Response: RGB 1 != RGB 2... Color Reproducing Strategies: varied...Color Reproducing Strategies: varied... Nonuniform sensitivity & geometry:Nonuniform sensitivity & geometry: –CRTs: e-beam cos(  ), distortion, focus, convergence... –LCDs, DLPs: VERY exact, (but pixels die, etc.)

20. Light Modifiers?   Discuss! Low-Contrast BRDF ‘Devices’ to measure light?Low-Contrast BRDF ‘Devices’ to measure light? –‘Light Probe’ mirror sphere BRDF = ? –Diffuse reflectances limited to about 0.02  0.95 –Diffractive materials: complex BRDF may be useful... –(Transmissive LCDs?) ?Can you name more? PRECISELY Linear ‘Response’ to light... BRDFs are fixed ratios; no intensity dependence!PRECISELY Linear ‘Response’ to light... BRDFs are fixed ratios; no intensity dependence! Smudge, nick may modify BRDF drasticallySmudge, nick may modify BRDF drastically Shadows? Precision? Inter-reflections?Shadows? Precision? Inter-reflections? PRECISE input/output symmetryPRECISE input/output symmetry--BUT-- Scattering WITHIN material can be trouble...Scattering WITHIN material can be trouble...

21. What is the complete IBMR toolset? Camera(s) + light probe, etc:  arbitrary Radiance meter.Camera(s) + light probe, etc:  arbitrary Radiance meter. Sphere of Projectors/CRTs:  arbitrary Irradiance source.Sphere of Projectors/CRTs:  arbitrary Irradiance source. Some (as yet unknown) device:  arbitrary BRDF / light ray modifierSome (as yet unknown) device:  arbitrary BRDF / light ray modifier Is our toolset complete complete? have we spanned the IBMR problem?...

22. Missing the most important tool… Human Visual System.Human Visual System. –the receiver/user for MOST IBMR data. –Eye is a very poor light meter, but very good at sensing BRDF and (some) shape. –Eye senses change; integration used to estimate the world –Eye permits tradeoffs of geometry vs. surface appearance –Eye permits selective radiance distortions, especially to illumination:

24. Details Everywhere; segmented partial - ordering of intensities. Local changes matter. Absolute intensities don’t matter much, but boundaries, shading, & CHANGES do. ---WANTED:--- visually important information in machine-readable form. Picture: Copy Appearance

25. Two Big Missing Pieces Computer controlled BRDF.Computer controlled BRDF. –Can we really do without it? –are cameras and projectors enough to ‘import the visible world’ into our computers? BRDF is not enough:BRDF is not enough: –Subsurface scattering is crucial aspect of photo’d images –? how can we model it? measure it? use it?

26. Scattering Difficulties: For many surfaces, single-point BRDFs do not exist Angles Depend on refractive index, scattering, cell wall structures, Depends on total area of cell wall interfaces Dicotyledon leaf structure

27. Subsurface Scattering Models Classical: Kubelka-Monk(1930s, for paint; many proprietary variants), CG approach: Hanrahan & Krueger(1990s) Recent (2001, Jensen) Marble BSSRDF Marble BRDF

28. Subsurface Scattering Models Classical: Kubelka-Monk(1930s, for paint; many proprietary variants), CG approach: Hanrahan & Krueger(1990s) Recent (2001, Jensen) Skin BSSRDF (approximated) Skin BRDF (measured)

29. BSSRDF Model Approximates scattering result as embedded point sources below a BRDF surface: BSSRDF: “A Practical Model for Subsurface Light Transport” Henrik Wann Jensen, Steve Marschner, Marc Levoy, Pat Hanrahan, SIGGRAPH’01 (online) online

30. BSSRDF Model Embedded point sources below a BRDF surfaceEmbedded point sources below a BRDF surface Ray-based, tested, Physically-Measurable ModelRay-based, tested, Physically-Measurable Model ?Useful as a predictive model for IBMR data??Useful as a predictive model for IBMR data? Wann Jensen et al., 2001

31. END