1 Practical Scene Illuminant Estimation via Flash/No-Flash Pairs Cheng Lu and Mark S. Drew Simon Fraser University {clu,

2 Flash/No-flash Imagery – a Brief History diCarlo, Xiao, & Wandell, CIC 2001 Combine flash/no-flash images to produce a pure-flash image.  Use dim=3 FDM + knowledge of flash SPD and sensor curves to estimate surface reflectance  most likely ambient illuminant Raskar et al., Non-Realistic Rendering 2004 Filling in night-time imagery with daytime image info. Copy edges from cloned image region into edge-map of target background; re- integrate. Blake et al., Poisson Image Editing, Siggraph 2004 Szeliski et al., Siggraph 2004 Transfer lower-noise information from flash image to higher-noise ambient-light image. Find shadow-mask, copy edges inside shadow from flash image into ambient image, re-integrate. Drew,Lu,Finlayson, Removing Shadows using Flash/Noflash Image Edges, ICME 2006

3 This paper: Estimate Ambient Illuminant, using Flash/No-flash Pairs Like diCarlo&Wandell approach, but replace knowledge of camera sensor curves with a camera RGB-based calibration using difference of with-flash and no-flash images. How? - Spectral sharpening - Subtract “both” – “no-flash”  pure-flash image - Log’s - Project difference of flash minus ambient into geometric-mean chromaticity color space  Calibrate such to get illuminant chromaticity.

4 What’s the point?: Can estimate scene (ambient) illuminant without knowing: - Flash SPD - Camera sensors - Surface reflectance

5 Why estimate the illuminant? White balance, plus many computer vision applications == intrinsic images without illumination. - Simple - Fast What’s good about this method?

6 The set-up: 2 images, one under ambient lighting, & another under flash. Under Ambient: Image “A”.Under Both: Image “B”. +

7 The Key: Pure-Flash Image  The ambient light from “A” is also in “B”.  Therefore if we subtract the two, we have “F”: the pure-flash image. Under Flash: Image “F”: + - = ) (

8 Incidentally, note that there are now extra shadows, from the flash (since it’s offset from the lens). Image “F”: the scene as imaged under Flash light only.

9 1. Lambertian surface: RGB = Shading = normal  effective light-direction Illuminant Surface Sensors Simple Image Formation Model  will guide us. Assumptions: 1., 2., 3.

10 2. Narrow-band sensors: so then is exactly a single-spike sensor:

11 3. Planckian light: But, can violate 1., 2., 3. and still succeed. (in Wien’s approximation) Gives

12 -Now take Log’s, to pull apart multiplications: Camera-dep’t vector Intensity and shading Surface Color- temperature of light where

13 Camera- dep’t vector Surface Color- temperature of light So form geometric-mean chromaticity: -We’d like to remove intensity/shading term: In logs: where

14 -The point: As temp (light color) changes, move along straight line. -But, we have “A” and “F” images:  Subract them, and use same chromaticity trick  Only illumination is left!

15 Log-difference Geometric-Mean Chromaticity  So log-log delivers inverse-temperature difference: -Calibrate for 1/T A -1/T F, then in new scene obtain T A ! {

16 What does this look like? Moved to 2D; color-matching functions in geo-mean chromaticity. (9 Planckians, Macbeth ColorChecker, spike sensors, xenon flash SPD)

17 Sony DXC930 sensors, Daylights+F2, actual xenon flash SPD: “Reference locus” How to proceed: -Sharpen - Find closest cluster

18 Effect of sharpening: Poor clusters Better clusters  #’ing Kodak DCS420:

19 Test: can we determine the illuminant? 102 illuminants, Sony camera, Munsell patches 102 illuminants, Sony camera, Macbeth patches Estimate illum. from Munsell to Macbeth  Nearly 100% correctly identified.

20 Application: White Balance 4 calibration illuminants, HP camera, Macbeth chart (each cluster has 24 dots) No flash  With flash  - Sharpen - Sample image at 24 locations evenly over image -Same (“daylight”) color balance for training and for testing Image under CWF; CWF+Xenon

21 Overlaps best with CWF, so use white patch of Macbeth under CWF for white balance: “Auto” balance – Wrong. “Fluor” balance – Correct. Our color-balance– Much closer.

22 Thanks! To Natural Sciences and Engineering Research Council of Canada