1. ICCV 2003 Colour Workshop 1 Recovery of Chromaticity Image Free from Shadows via Illumination Invariance Mark S. Drew 1, Graham D. Finlayson 2, & Steven D. Hordley 2 2 School of Information Systems, University of East Anglia, UK 1 School of Computing Science, Simon Fraser University, Canada

2. ICCV 2003 Colour Workshop 2 Overview Introduction Shadow Free Greyscale images - Illuminant Invariance at a pixel -- 1D image Shadow Free Chromaticity Images - Better-behaved 2D-colour image invariant to lighting Application - For shadow-edge-map aimed at re-integrating to obtain full colour, shadow-free image

3. ICCV 2003 Colour Workshop 3 The Aim: Shadow Removal We would like to go from a colour image with shadows to the same colour image, but without the shadows.

4. ICCV 2003 Colour Workshop 4 Why Shadow Removal? For Computer Vision, Image Enhancement, Scene Re-lighting, etc. - e.g., improved object tracking, segmentation etc. Two successive video frames Motion map, original colour space  Motion map, invariant colour space snake

5. ICCV 2003 Colour Workshop 5 What is a shadow? Region Lit by Sunlight and Sky-light Region Lit by Sky-light only A shadow is a local change in illumination intensity and (often) illumination colour.

6. ICCV 2003 Colour Workshop 6 Removing Shadows So, if we can factor out the illumination locally (at a pixel) it should follow that we remove the shadows. Can we factor out illumination locally? That is, can we derive an illumination-invariant colour representation at a single image pixel? Yes, provided that our camera and illumination satisfy certain restrictions ….

7. ICCV 2003 Colour Workshop 7 Conditions for Illumination Invariance Assumptions (but works anyway…!): (1) If sensors can be represented as delta functions (they respond only at a single wavelength) (2) and illumination is restricted to the Planckian locus (3) then we can find a 1D coordinate, a function of image chromaticities, which is invariant to illuminant colour and intensity (4) this gives us a greyscale representation of our original image, but without the shadows (so takes us a third of the way to the goal of this talk!)  (5) But the greyscale value in fact lives in a 2D log- chromaticity colour space, (so takes us a 2/3 of the way) [and exponentiating goes back to a rank-3 colour].

8. ICCV 2003 Colour Workshop 8 Chromaticity: 2D chromaticity is much more information than 1D greyscale: Can we obtain a shadowless chromaticity image? grey colour chromaticity

9. ICCV 2003 Colour Workshop 9 Image Formation Camera responses depend on 3 factors: light (E), surface (S), and sensor (Q)  is Lambertian shading

10. ICCV 2003 Colour Workshop 10 Q 2 ( ) Sensitivity Q 1 ( )Q 3 ( ) = Delta functions “select” single wavelengths: Using Delta-Function Sensitivities

11. ICCV 2003 Colour Workshop 11 Characterizing Typical Illuminants Most typical illuminants lie on, or close to, the Planckian locus (the red line in the figure) So, let’s represent illuminants by their equivalent Planckian black-body illuminants...

12. ICCV 2003 Colour Workshop 12 Here I controls the overall intensity of light, T is the temperature, and c 1, c 2 are constants Planckian Black-body Radiators For typical illuminants, c 2 >> T. So, Wien’s approximation:

13. ICCV 2003 Colour Workshop 13 How good is this approximation? 2500 Kelvin 10000 Kelvin 5500 Kelvin

14. ICCV 2003 Colour Workshop 14 For delta-function sensors and Planckian illumination we have: Back to the image formation equation Surface Light

15. ICCV 2003 Colour Workshop 15 Band-ratio chromaticity G R B Plane G=1 Perspective projection onto G=1 Let us define a set of 2D band-ratio chromaticities: p is one of the channels, (Green, say)

16. ICCV 2003 Colour Workshop 16 Let’s take log’s: Band-ratios remove shading and intensity with Gives a straight line: Shading and intensity are gone.

17. ICCV 2003 Colour Workshop 17 Calibration: find invariant direction Log-ratio chromaticities for 6 surfaces under 14 different Planckian illuminants, HP912 camera Macbeth ColorChecker: 24 patches

18. ICCV 2003 Colour Workshop 18 Deriving the Illuminant Invariant This axis is invariant to shading + illuminant intensity/colour

19. ICCV 2003 Colour Workshop 19 Algorithm: Plot, and subtract mean for each colour patch: SVD (2 nd eigenvector) gives invariant direction.

20. ICCV 2003 Colour Workshop 20 Algorithm, cont’d: Form greyscale I’ in log-space: exponentiate:

21. ICCV 2003 Colour Workshop 21 Obtaining invariant Chromaticity image (1): We observe: line in 2D chromaticity space is still 2D, if we use projector, rather than rotation: 2-vector

22. ICCV 2003 Colour Workshop 22 Obtaining invariant Chromaticity image (2): However, we have removed all lighting!  put back offset in e-direction equal to regression on top 1% brightness pixels:

23. ICCV 2003 Colour Workshop 23 Obtaining invariant Chromaticity image (3): offset in e-direction: We are most familiar with L 1 -chromaticity

24. ICCV 2003 Colour Workshop 24 Obtaining invariant Chromaticity image (4): In terms of L 1 -chromaticity: orig. recovered

25. ICCV 2003 Colour Workshop 25 Obtaining invariant Chromaticity image (5): Projection line becomes a rank ~3 curve in L 1 chromaticity space

26. ICCV 2003 Colour Workshop 26 Obtaining invariant Chromaticity image (6): We can do better on fitting recovered chromaticity to original — regress on brightest quartile:

27. ICCV 2003 Colour Workshop 27 Improves chromaticity: orig. recovered regressed

28. ICCV 2003 Colour Workshop 28 Some Examples colour chromaticity recovered

29. ICCV 2003 Colour Workshop 29 Main Advantage: chromaticity invariant (in [0,1]) is better- behaved than greyscale invariant –– better for shadow-free re-integration (ECCV02)

30. ICCV 2003 Colour Workshop 30 Acknowledgements The authors would like to thank the Natural Sciences and Engineering Research Council of Canada, and Hewlett-Packard Incorporated for their support of this work.