1. 1 Automatic Compensation for Camera Settings for Images Taken under Different Illuminants Cheng Lu and Mark S. Drew Simon Fraser University {clu, mark}@cs.sfu.ca

2. 2 Flash/No-flash Imagery – What About Camera Settings? (or, more generally, pairs of images with two different illuminants). Growing body of research on combining flash/no-flash image pairs to carry out tasks in: - Computer Vision and in - Color Science

3. 3 + “Removing Shadows using Flash/Noflash Image Edges” One use: “Removing Shadows using Flash/Noflash Image Edges” [Lu, Drew, & Finlayson, ICME 2006]

4. 4

5. 5

6. 6 ( )

7. 7 But need to ensure that - really gives just the image under pure-flash lighting. If settings are different, won’t work, without compensation!

8. 8 Strategy:  Wish to compensate for  exposure time,  ISO,  aperture,  focal length,  white balance.  Can use a 2nd-order “masking model” (i.e., polynomial) on such parameters  How do we know how to compensate?  Make shadow disappear for difference of adjusted images, by matrixing,  Map pairs of settings to matrix via masking model.

9. 9 Strategy, cont’d:  Simplify matrix: Adjust magnitude in each color channel so as to eliminate shadow in: (with-flash) – (no-flash), over large set of image pairs.  Train polynomial model.  Apply polynomial model to new image pairs.

10. 10 Assumptions:  Additivity and proportionality of (transformed) camera parameters  2 nd order polynomial model  9 parameters. (Compare CMY overprinting: )

11. 11 Example of image pairs: No scaling Scaled to max=255 Ambient light (“A”) Ambient + flash ( “Both”, “B”)

12. 12 Now subtract: No, see shadow in pure-flash image! So use in-shadow, out-of-shadow regions to obtain 3 color-channel multipliers 

13. 13  We need 3-vector of scaling coefficients A  A so boxes match, in difference image. Call in-shadow region “s”, out-of- shadow “ns”:

14. 14  Now what is M: A  A as a function of camera settings?  use polynomial model (like for printers) -- uses log’s and assumes additivity and proportionality of values. Parameters:

15. 15 Training : 1. Fix focal length, use tripod. 2. Use “auto” setting; and acquire actual settings used from stored image meta-data. 3. Use EV (exposure value) = same for all shutter speed/aperture combinations that give same exposure. In APEX system (Additive Photographic Exposure System), EV=AV+TV; AV=ApertureValue=log 2 f 2, TV=TimeValue=-log 2 t 4. ISO automatic 5. White balance  encapsulate the effect of white balancing by using use the mean value for each RGB channel in the masking model.

16. 16 6. Ok, we generate values in M: A  A by selecting in/out-of-shadow areas by hand. What model should we use for mapping settings to M?  Use log’s of ratios, in 2 nd - order model: 9 parameters a1,a2,a3,b1,b2,b3, c1,c2 c3, so use least- squares. Then apply same model to new image pair.

17. 17 3N x 13N x 99 x 1 ??

18. 18 Experiments 5 lighting sources: –Direct sunlight, cloudy daylight, a tungsten light lamp and incandescent lamp, and xenon flash light. Images captured in 5 situations: 125 training image pairs; 125 tests using take-one-out re-calc. of M: re-compute 9 param’s, predict M, apply. Sophisticated experimental setup :

19. 19 Ambient images Ambient+flash images Pure flash images Success! – no shadows

20. 20 Thanks! To Natural Sciences and Engineering Research Council of Canada