Shadow Removal Using Illumination Invariant Image Graham D. Finlayson, Steven D. Hordley, Mark S. Drew Presented by: Eli Arbel

Shadow Removal Seminar 2 Outline Introduction Removing Shadows Reconstruction Illumination Invariant Images Summary

Shadow Removal Seminar 3 Introduction Why shadow removal ? – Computer Vision – Image Enhancement – Illumination Re-rendering

Shadow Removal Seminar 4 Introduction, cont ’ d

Shadow Removal Seminar 5 Introduction, cont ’ d What is shadow ? Region lit by sunlight and skylight Region lit by skylight only A shadow is a local change in illumination intensity and (often) illumination color.

Shadow Removal Seminar 6 Introduction, cont ’ d Assumptions for shadow removal: – Only Hard shadows can be removed – No overlapping of object and shadow boundaries – Planckian light source – Narrow band sensors of the capturing device

Shadow Removal Seminar 7 Outline Introduction Removing Shadows Reconstruction Illumination Invariant Images Summary

Shadow Removal Seminar 8 Method For Removing Shadows An RGB image is input Shadow identification is based on edge detection

Shadow Removal Seminar 9 Discriminating Edges Can we factor out illumination changes (intensity and color) ? More on that later … Yes, under some assumptions …

Shadow Removal Seminar 10 Discriminating Edges, cont ’ d Input Image RGB Channels Edge Maps Illumination Invariant Image Illumination Invariant Image Edge Map

Shadow Removal Seminar 11 Discriminating Edges - Formally Let us denote one of the three channel edge maps as  (x,y) And denote the invariant image edge map as  gs(x,y) we apply a Thresholding operator on each of the channel edge maps as follows: Where ||  (x,y)|| and ||  gs(x,y)|| are the gradient magnitudes of channel edge map and illumination invariant edge map respectively

Shadow Removal Seminar 12 Discriminating Edges, cont ’ d Input Image RGB Channels Edge Maps Illumination Invariant Image Illumination Invariant Image Edge Map Thresholded edge maps

Shadow Removal Seminar 13 Outline Introduction Removing Shadows Reconstruction Illumination Invariant Images Summary

Shadow Removal Seminar 14 Reconstructing the Image For each channel, we now have an edge map in which shadow edges are removed: T – Thresholding operator – Derivative operator in x direction – Derivative operator in y direction

Shadow Removal Seminar 15 Re-integrating Edge Information We would like to integrate So first, we calculate the Laplacian out of the gradient:

Shadow Removal Seminar 16 Re-integrating Edge Information – cont ’ d Now we solve by applying the Inverse Laplacian: This is a private case of the Wiess reconstruction process where we have only two filters, and.

Shadow Removal Seminar 17 More on Reconstruction The re-integration step recover uniquely up to a multiplicative (additive) constant – DC. A heuristic approach is used to find this constant. For each shadow-free channel image: Consider the top 1-percentile pixels Compute their average Map this value to white

Shadow Removal Seminar 18 Some results

Shadow Removal Seminar 19 Some results – cont ’ d

Shadow Removal Seminar 20 Some results – cont ’ d

Shadow Removal Seminar 21 Some results – cont ’ d

Shadow Removal Seminar 22 Outline Introduction Removing Shadows Reconstruction Illumination Invariant Images Summary

Shadow Removal Seminar 23 Illumination Invariant Image – Theoretical Analysis Sensor response at any pixel can be formulated as: R= Reflectance L = Illumination S = Sensor Sensitivity

Shadow Removal Seminar 24 Assumption 1: Capturing Device Sensors Sensor response is narrow band, i.e. a Dirac Function:

Shadow Removal Seminar 25 Assumption 1: Capturing Device Sensors – cont ’ d

Shadow Removal Seminar 26 Assumption 2: Planckian Light Source Scene illumination is assumed to be Planckian, i.e. it falls very near to the Planckian locus:

Shadow Removal Seminar 27 Assumption 2: Planckian Light Source – cont ’ d Planck's law of black body radiation: The spectral intensity of electromagnetic radiation from a black body at temperature T as a function of wavelength:

Shadow Removal Seminar 28 Assumption 2: Planckian Light Source – cont ’ d Planck ’ s Law is a good approximation for incandescent and daylight illuminants 2500k 5500k CIE D55

Shadow Removal Seminar 29 Assumption 2: Planckian Light Source – cont ’ d To model varying illumination power, we add an intensity constant I: In addition, it can be shown that, thus:

Shadow Removal Seminar 30 Assumption 2: Planckian Light Source – cont ’ d

Shadow Removal Seminar 31 Towards Color Constancy at a Pixel Depends on illuminant intensity Depends on Surface reflectance Depends on Illuminant color

Shadow Removal Seminar 32 Towards Color Constancy at a Pixel – cont ’ d Simplifying notations:

Shadow Removal Seminar 33 Towards Color Constancy at a Pixel – Dropping the Intensity term

Shadow Removal Seminar 34 Color Constancy at a Pixel The relations: Can be written in matrix notation:

Shadow Removal Seminar 35 Color Constancy at a Pixel – cont ’ d - Camera sensor response Reminder: We just solved the one-dimensional color constancy problem at a pixel !

Shadow Removal Seminar 36 Color Constancy at a Pixel Examples Log-Chromaticity Differences for seven surfaces under 10 Planckian illuminants

Shadow Removal Seminar 37 Color Constancy at a Pixel Examples – cont ’ d Log-Chromaticity Differences for the Macbeth Color Checker with HP912 Digital Still Camera

Shadow Removal Seminar 38 Color Constancy at a Pixel Examples – cont ’ d Log-Chromaticity Differences for the Macbeth Color Checker with Nikon D-100

Shadow Removal Seminar 39 Illumination Invariant Images - Examples

Shadow Removal Seminar 40 Outline Introduction Removing Shadows Reconstruction Illumination Invariant Images Summary

Shadow Removal Seminar 41 Summary A method for shadow removal in single image using 1-D illumination invariant image presented Shadow-free edge-maps are re-integrated using Wiess reconstruction method 1-D Illumination invariant image is obtained relying on physical properties of lightness and camera sensors

Shadow Removal Seminar 42 References G. D. Finlayson, S.D. Hordley and M.S. Drew. Removing Shadows From Images G. D. Finlayson, S. D. Hordley and M. S. Drew. Removing shadows from images. Presentation for ECCV02, Grahm. D. Finlayson, Steven. D. Hordley. Color Constancy at a Pixel. Model-Based Object Tracking in Road Traffic Scenes, Dieter Koller ואבישי אדלר אורי בריט. הסרת צל מסדרת תמונות ומתמונה בודדת