Shadow removal algorithms Shadow removal seminar Pavel Knur

Deriving intrinsic images from image sequences Yair Weiss July 2001.

History “ intrinsic images ” by Barrow and Tenenbaum, 1978

Constraints Fixed viewpoint Works only for static objects Cast shadows

Classic ill-posed problem Denote – the input image – the reflectance image – the illumination image Number of Unknowns is twice the number of equations.

The problem Given a sequence of T images in which reflectance is constant over the time and only the illumination changes, can we solve for a single reflectance image and T illumination images ? Still completely ill-posed : at every pixel there are T equations and T+1 unknowns.

Maximum-likelihood estimation Log domain :

Assumptions When derivative filters are applied to natural images, the filter outputs tend to be sparse.

Laplacian distribution Can be well fit by laplacian distribution

Claim 1 Denote : N filters – Filter outputs – Filtered reflectance image – ML estimation of filtered reflectance image is given by

Estimated reflectance function Recover ML estimation of r is reversed filter of

ML estimation algorithm

ML estimation algorithm – cont. Ones we have estimated

Claim 2 What if does not have exactly a Laplasian distribution ? Let Then estimated filtered reflectance are within with probability at least:

Claim 2 - proof If more than 50% of the samples of are within of some value, then by definition of median, the median must be within of that value.

Example 1 Einstein image is translated diagonally 4 pixels per frame

Example 2 64 images with variable lighting from Yale Face Database

Illumination Normalization with Time- Dependent Intrinsic Images for Video Surveillance Y.Matsushita,K.Nishito,K.Ikeuchi Oct. 2004

Illumination Normalization algorithm Preprocessing stage for robust video surveillance. Causes –Illumination conditions –Weather conditions –Large buildings and trees Goal –To “ normalize ” the input image sequence in terms of incident lighting.

Constraints Fixed viewpoint Works only for static objects Cast shadows

Background images Remove moving objects from the input image sequence Input images Background images Off-line

Estimation of Intrinsic Images Denote input image time-varying reflectance image time-varying illumination image reflectance image estimated by ML illumination image estimated by ML Filters Log domain Input images Background images Off-line Estimation of Intrinsic Images

Estimation of Intrinsic Images – cont. In Weiss ’ s original work The goal is to find estimation of and Input images Background images Off-line Estimation of Intrinsic Images

Estimation of Intrinsic Images – cont. Basic idea: Estimate time-varying reflectance components by canceling the scene texture from initial illumination images Define: Input images Background images Off-line Estimation of Intrinsic Images

Estimation of Intrinsic Images – cont. Finally : Where : is reversed filter of Input images Background images Off-line Estimation of Intrinsic Images

Shadow Removal Denote - background image - illuminance-invariant image Input images Background images Off-line Estimation of Intrinsic Images

Illumination Eigenspace PCA – Principle component analysis Basic components - Input images Background images Off-line Estimation of Intrinsic Images Illumination Eigenspace

Illumination Eigenspace – cont. Average is P is MxN matrix where –N – number of pixels in illumination image –M – number of illumination images Covariance matrix Q of P is Input images Background images Off-line Estimation of Intrinsic Images Illumination Eigenspace

Direct Estimation of Illumination Images Pseudoillumination image Direct Estimation is Where –F is a projection function onto the j ’ s eigenvector - Input images Background images Off-line Estimation of Intrinsic Images Illumination Eigenspace

Direct Estimation of Illumination Images Results Input images Background images Off-line Estimation of Intrinsic Images Illumination Eigenspace

Shadow interpolation probability density function cumulative probability function shadowed area lit area mean optimum threshold value Input images Background images Off-line Estimation of Intrinsic Images Illumination Eigenspace Shadow Interpolation

The whole algorithm Input images Background images Off-line Estimation of Intrinsic Images Illumination Eigenspace / Illumination Images Normalization Shadow Interpolation

Example

Questions ?

References [1] Y.Weiss, ” Deriving Intrinsic Images from Image Sequences ”, Proc. Ninth IEEE Int ’ l Conf. Computer Vision, pp , July [2] Y.Matsushita,K.Nishito,K.Ikeuchi, “ Illum ination Normalization with Time- Dependent Intrinsic Images for Video Surveillance ”,Oct