School of Computer Science Simon Fraser University November 2009 Sharpening from Shadows: Sensor Transforms for Removing Shadows using a Single Image Mark S. DrewHamid Reza Vaezi Joze

Outline Image Formation Invariant Image Formation Finding invariant direction by calibration Finding invariant direction by minimizing entropy Sharpening Matrix Proposed Method Optimization problem Result 2

Shadow Removal Method To generate shadowless images, there are two steps: 1. Finding Illuminant Invariant image (grayscale) 2. Creating colored shadowless images using edges in main image and invariant image. [Finlayson et al. (ECCV2002)] 3 Main ImageShadowless ImageInvariant Image This Paper

Image Formation 4 Surface reflection Camera sensitivity Light spectral Camera Response:

Image Formation Simplification 5 1. Camera sensors represented as delta functions. 2. Illumination is restricted to the Planckian locus. 3. Wien’s approximation for temperature range 2500K to 10000K.  We have:

Invariant Image Formation 6 Using the simplified model we form band-ratio chromaticities r k by dividing  R and  B by  G and taking the logarithm: As temperature T changes, 2d-vectors r k,k=R,B, will follow a straight line in 2d chromaticity space. For all surfaces, the lines will be parallel, with slope (e k –e G ). Surface Dependent Camera Dependent

Invariant Image Formation 7 The invariant image, then, is formed by projecting 2-d colors into the direction orthogonal to the 2-vector (e k – e G ). So, the problem is reduced to finding the direction. Why we are interested? Shadow is nothing just the surface in different illumination condition. (they should be in a line)

Finding Invariant Direction 8 Calibrating Camera to find the invariant direction. [Finlayson et al. (ECCV2002)] Need many images under different illumination. Good for camera company not images with unknown camera. HP912 Digital Still Camera: Log-chromaticities of 24 patches; 7 patches, imaged under 9 illuminants.

Finding Invariant Direction 9 Without calibrating the camera, can use entropy of projection to find the invariant direction [Finlayson et al. (2004)] : Correct direction – smaller entropy Wrong direction – higher entropy

Sharpening Transform Matrix 10 Convert a given set of sensor sensitivity functions into a new set that will improve the performance of any color- constancy algorithm that is based on an independent adjustment of the sensor response channels. Transform the camera sensors to made them more narrow band, which is one of the assumption that we made. It also could apply to the image instead of sensors.

Proposed Method Select shadow and non shadow pixels for the same surface material. 2. Find the sharpening matrix which makes the chromaticities of selected pixels as linear as possible in log-log plane = an optimization problem. 3. Transform the main image by sharpening matrix. 4. Create illumination invariant image by entropy-minimization method [Finlayson et al. (2004)]

Shadow and Non Shadow Regions 12 The user selects the shadow and non shadow region of a surface. For future work this could be automatic. According to invariant formation in ideal condition, the chromaticities of these point in log-log plane should be in a line. User Defined

Optimization Problem 13 To find best sharpening matrix M 3x3 in order to make the chromaticity as linear as possible: m 11 m 12 m 13 m 21 m 22 m 23 m 31 m 32 m 33 sum is 1 Linear combination more than 1-β Colors don’t change completely

Objective Function 14 F return the minimum entropy of log chromaticities projected to all directions. rank is meant to encourage a non- rank-reducing matrix M. entropy Log chromaticities Minimum entropy For this M

Sharpening Matrix 15 Sharpening Matrix Shadow and non shadow region chromaticity Less linear More linear

Results 16 DifferenceInvariantSharpenedOriginal

Good vs. poor sharpening matrix 17 More linear minimum Obj. Func. =.0942 Obj. Func. =.0487

Result 18

Result 19

Conclusion 20 We proposed a new schema for generating illumination invariant for removing shadow. The contribution of this paper is using sharpener matrix to get better shadow removal. The method use single images which is more practical compared to camera calibration methods which needs bunch of images in different illumination condition.

References 21 Sharpening Matrix: G.D. Finlayson, M.S. Drew, and B.V. Funt. Spectral sharpening: sensor transformations for improved color constancy. J. Opt. Soc. Am. A, 11(5):1553–1563, May Illumination invariant image: G.D. Finlayson, S.D. Hordley, and M.H. Brill. Illuminant invariance at a single pixel. In 8th Color Imaging Conference: Color, Science, Systems and Applications., pages 85–90, Shadow removal method: G.D. Finlayson, S.D. Hordley, and M.S. Drew. Removing shadows from images. In ECCV 2002: European Conference on Computer Vision, pages 4:823–836, Lecture Notes in Computer Science Vol Entropy minimization method: G.D. Finlayson, M.S. Drew, and C. Lu. Intrinsic images by entropy minimization. In ECCV 2004: European Conference on Computer Vision, pages 582–595, Lecture Notes in Computer Science Vol

Questions? Thank you. 22 Thanks! To Natural Sciences and Engineering Research Council of Canada