1. ECCV 2002 Removing Shadows From Images G. D. Finlayson 1, S.D. Hordley 1 & M.S. Drew 2 1 School of Information Systems, University of East Anglia, UK 2 School of Computer Science, Simon Fraser University, Canada

2. ECCV 2002 Overview Introduction Shadow Free Grey-scale images - Illuminant Invariance at a pixel Shadow Free Colour Images - Removing shadow edges using illumination invariance - Re-integrating edge maps Results and Future Work

3. ECCV 2002 The Aim: Shadow Removal We would like to go from a colour image with shadows, to the same colour image, but without the shadows.

4. ECCV 2002 Why Shadow Removal? For Computer Vision - improved object tracking, segmentation etc. For Image Enhancement - creating a more pleasing image For Scene Re-lighting - to change for example, the lighting direction

5. ECCV 2002 What is a shadow? Region Lit by Sunlight and Sky-light Region Lit by Sky-light only A shadow is a local change in illumination intensity and (often) illumination colour.

6. ECCV 2002 Removing Shadows So, if we can factor out the illumination locally (at a pixel) it should follow that we remove the shadows. So, can we factor out illumination locally? That is, can we derive an illumination-invariant colour representation at a single image pixel? Yes, provided that our camera and illumination satisfies certain restrictions ….

7. ECCV 2002 Conditions for Illumination Invariance (1) If sensors can be represented as delta functions (they respond only at a single wavelength) (2) and illumination is restricted to the Planckian locus (3) then we can find a 1-D co-ordinate, a function of image chromaticities, which is invariant to illuminant colour and intensity (4) this gives us a grey-scale representation of our original image, but without the shadows (it takes us a third of the way to the goal of this talk!)

8. ECCV 2002 Image Formation Camera responses depend on 3 factors: light (E), surface (S), and sensor (R, G, B)

9. ECCV 2002 G( ) Sensitivity B( )R( ) = Delta functions “select” single wavelengths: Using Delta Function Sensitivities

10. ECCV 2002 Characterising Typical Illuminants Most typical illuminants lie on, or close to, the Planckian locus (the red line in the figure) So, let’s represent illuminants by their equivalent Planckian black-body illuminants...

11. ECCV 2002 Here I controls the overall intensity of light, T is the temperature, and c 1, c 2 are constants Planckian Black-body Radiators But, for typical illuminants, c 2 >> T. So, Planck’s eqn. is approximated as:

12. ECCV 2002 How good is this approximation? 2500 Kelvin 10000 Kelvin 5500 Kelvin

13. ECCV 2002 For, delta function sensors and Planckian Illumination we have: Back to the image formation equation SurfaceLight Or, taking the log of both sides...

14. ECCV 2002 Where subscript s denotes dependence on reflectance and k,a,b and c are constants. T is temperature. Summarising for the three sensors Constant independent of sensor Variable dependent only on reflectance Variable dependent on illuminant

15. ECCV 2002 That is: there exists a weighted difference of log-opponent chromaticities that depends only on surface reflectance Factoring out the illumination First, let’s calculate log-opponent chromaticities: Then, with some algebra, we have:

16. ECCV 2002 An example - delta function sensitivities B W R Y G P Narrow-band (delta-function sensitivities) Log-opponent chromaticities for 6 surfaces under 9 lights

17. ECCV 2002 Deriving the Illuminant Invariant Log-opponent chromaticities for 6 surfaces under 9 lights This axis is invariant to illuminant colour Rotate chromaticities

18. ECCV 2002 Normalized sensitivities of a SONY DXC-930 video camera A real example with real camera data Log-opponent chromaticities for 6 surfaces under 9 different lights

19. ECCV 2002 Deriving the invariant Log-opponent chromaticities for 6 surfaces under 9 different lights The invariant axis is now only approximately illuminant invariant (but hopefully good enough) Rotate chromaticities

20. ECCV 2002 Some Examples

21. ECCV 2002 A Summary So Far With certain restrictions, from a 3-band colour image we can derive a 1-d grey-scale image which is: - illuminant invariant - and so, shadow free

22. ECCV 2002 What’s left to do? To complete our goal we would like to go back to a 3- band colour image, without shadows We will look next at how the invariant representation can help us to do this...

23. ECCV 2002 Looking at edge information Consider an edge map of the colour image... And an edge map of the 1-d invariant image... These are approximately the same, except that the invariant edge map has no shadow edges

24. ECCV 2002 Removing Shadow Edges From these two edge maps we can remove shadow edges thus: Edges =  I orig &  I inv (Valid edges are in the original image, and in the invariant image)

25. ECCV 2002 Using Shadow Edges So, now we have the edge map of the image we would like to obtain (edge map of the original image with shadows edges set to zero) So, can we go from this edge information back to the image we want? (can we re-integrate the edge information?).

26. ECCV 2002 Re-integrating Edge Information Of course, re-integrating a single edge map will give us a grey-scale image: So, we must apply any procedure to each band of the colour image separately: Red Green Blue Original Colour Channels Edge Maps of Channels Shadow Edges Removed Re-integrated

27. ECCV 2002 Re-Integrating Edge Information The re-integration problem has been studied by a number of researchers: - Horn - Blake et al - Weiss ICCV ‘01 (Least-Squares) -... - Land et al (Retinex) The aim is typically to derive a reflectance image from an image in which illumination and reflectance are confounded.

28. ECCV 2002 Weiss’ Method Weiss used a sequence of time varying images of a fixed scene to determine the reflectance edges of the scene His method works by determining, from the image sequence, edges which correspond to a change in reflectance (Weiss’ definition of a reflectance edge is an edge which persists throughout the sequence) Given reflectance edges, Weiss re-integrates the information to derive a reflectance image In our case, we can borrow Weiss’ re-integration procedure to recover our shadow-free image.

29. ECCV 2002 Re-integrating Edge Information Let I j (x,y) represent the log of a single band of a colour image  x is the derivative operator in the x direction  y is the derivative operator in the y direction We first calculate: T is the operator that sets shadow edges to zero This summarises the process of detecting and removing shadow edges

30. ECCV 2002 Re-integrating Edge Information To recover the shadow free, image we want to invert this Equation To do this, we first form the Poisson Equation We solve this (subject to Neumann boundary conditions) as follows:

31. ECCV 2002 Re-integrating Edge Information We solve by applying the inverse Laplacian Note: the inverse operator has no Threshold Applying this process to each of the three channels recovers a log image without shadows.

32. ECCV 2002 A Summary of Re-integration 1. I orig = Original colour image, I inv = Invariant image 2. For j=1,2,3 I j orig = jth band of I orig 3. Remove Shadow Edges: Edges =  I j orig &  I inv 4. Differentiate the thresholded edge map 5. Re-integrate the image 6. Goto 3

33. ECCV 2002 Some Remarks The re-integration step is unique up to an additive constant (a multiplicative constant in linear image space Fixing this constant amounts to applying a correction for illumination colour to the image. Thus we choose suitable constants to correct for the prevailing scene illuminant In practice, the method relies upon having an effective thresholding step T, that is, on effectively locating the shadow edges. As we will see, our shadow edge detection is not yet perfect

34. ECCV 2002 Shadow Edge Detection The Shadow Edge Detection consists of the following steps: 1. Edge detect a smoothed version of the original (by channel) and the invariant images 2. Threshold to keep strong edges in both images 3. Shadow Edge = Edge in Original & NOT in Invariant 4. Applying a suitable Morphological filter to thicken the edges resulting from step 3. Canny or SUSAN This typically identifies the shadow edges plus some false edges

35. ECCV 2002 An Example Detected Shadow Edges Original Image Invariant Image Shadow Removed

36. ECCV 2002 A Second Example Detected Shadow Edges Original Image Invariant Image Shadow Removed

37. ECCV 2002 More Examples Detected Shadow Edges Original Image Invariant Image Shadow Removed

38. ECCV 2002 More Examples Detected Shadow Edges Original Image Invariant Image Shadow Removed

39. ECCV 2002 A Summary We have presented a method for removing shadows from images The method uses an illuminant invariant 1-d image representation to identify shadow edges From the shadow free edge map we re-integrate to recover a shadow free colour image Initial results are encouraging: we are able to remove shadows, even when shadow edge definition is not perfect

40. ECCV 2002 Future Work We are currently investigating ways to more reliably identify shadow edges... … or to derive a re-integration which is more robust to errors (Retinex?) Currently deriving the illuminant invariant image requires some knowledge of the capture device’s characteristics - We show in the paper how to determine these characteristics empirically and we are working on making this process more robust

41. ECCV 2002 Acknowledgements The authors would like to thank Hewlett-Packard Incorporated for their support of this work.