Shadow Detection and Removal Image Processing Seminar Shadow Detection and Removal Using Video Advisor: Dr. Hagit Hel-Or By: Ido Yerushalmy February 2008

Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Why remove a shadow? The problem: shadows, like real objects, usually differ from their surroundings and move with the object creating them – making it difficult to distinguish between the 2 types Removing the shadows can help perform segmentation, pattern matching and moving objects extraction Knowing the luminance of the scene allows adding CG sections in a realistic manner JPEG Format Incentives Quant. Table Shadow Characteristics MPEG Format Survey GOP correlation ML Algorithm Double Quantization ML enhancement Summary Summary

Shadow From Video Incentives Types of shadows A cast shadow is the area projected on the scene by an object. It can be divided into: Umbra – direct light is totally blocked Penumbra – light partially blocked Shadow Characteristics Survey ML Algorithm ML enhancement Summary

Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Modeling appearance of a point Incentives For simplicity, assume all surfaces are matt (Lambertian world) Each point we see is the result of its reflectance and the amount of luminance it receives In a mathematical notation: I(x,y) = R(x,y) * L(x,y) Shadow Characteristics Quant. Table MPEG Format Survey GOP correlation ML Algorithm Double Quantization ML enhancement Summary Summary Image = Reflectance x Luminance

Modeling appearance of a point– Cont. Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Modeling appearance of a point– Cont. Incentives We can furthermore model the umbra and penumbra cases: Shadow Characteristics Quant. Table Quant. Table Quant. Table MPEG Format MPEG Format MPEG Format Survey GOP correlation GOP correlation GOP correlation ML Algorithm Double Quantization Double Quantization Double Quantization ML enhancement Illuminate Summary Summary Summary Summary Penumbra Umbra Where: CA – ambient light intensity CP – light source intensity L – direction of light source N(x,y) – object surface normal K(x,y) – softening due to penumbra

Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Survey of common methods Incentives Algorithms assume the following: Light source is white, spread evenly and strong The reference image is assumed static, textured and planar Shadow Characteristics Quant. Table Quant. Table Survey MPEG Format MPEG Format GOP correlation GOP correlation ML Algorithm Double Quantization Double Quantization ML enhancement Summary Summary Summary

Statistical non-parametric algorithms Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Statistical non-parametric algorithms Incentives Such statistical algorithms assume that shadows have similar chromaticity but lower brightness than the background model. A statistical learning procedure is used to automatically determine the appropriate thresholds. Shadow Characteristics Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format GOP correlation GOP correlation GOP correlation ML Algorithm Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary

Deterministic non-model based with color exploitation Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Deterministic non-model based with color exploitation Incentives Shadow Characteristics Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format Uses the HSV color representation GOP correlation GOP correlation GOP correlation ML Algorithm Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary

Deterministic non-model based with color exploitation – cont. Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Deterministic non-model based with color exploitation – cont. Incentives Shadow Characteristics Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format A shadow cast does not change the Hue significantly Shadows often lower the saturation and value of the points GOP correlation GOP correlation GOP correlation ML Algorithm Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary

Deterministic non-model based with color exploitation – cont. Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Deterministic non-model based with color exploitation – cont. Incentives Shadow Characteristics Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format Formally, it can be described as: GOP correlation GOP correlation GOP correlation ML Algorithm Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary Where: I and B are the input image and its computed background Alpha defines the “power” of the shadow and Beta is used only to avoid misclassifications due to noise

Deterministic non-model based with spatial redundancy exploitation Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Deterministic non-model based with spatial redundancy exploitation Incentives Shadow Characteristics Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format Three criteria are tested to detect shadows: The presence of a "darker" uniform regions hints a cast shadow Presence of high difference in luminance with relation to the reference frame The presence of static and moving edges. Static edges hint static shadows. Looking at the width of the edges can reveal if this is a penumbra shadow, since it usually have a softer (= wider) edge Drawback: This method is not enough generic and has too many assumptions GOP correlation GOP correlation GOP correlation ML Algorithm Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary

Maximum Likelihood Algorithm Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Maximum Likelihood Algorithm Incentives Shadow Characteristics Assumptions on the world: Lambertian (all surfaces are matt) A single reflectance image (static camera), and many illuminations images. Fits a surveillance camera model Formally denoted by: Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary For simplicity reasons we will work in the log domain:

Maximum Likelihood Algorithm - cont Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Maximum Likelihood Algorithm - cont Incentives Statistical knowledge from previous works: When derivative filters are applied to natural images, the filter output tends to be sparse Shadow Characteristics Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation Double Quantization Double Quantization Double Quantization ML enhancement Natural Image Horizontal Derivative Hist. Summary Summary Summary Summary

Maximum Likelihood Algorithm - cont Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Maximum Likelihood Algorithm - cont Incentives Shadow Characteristics Quant. Table Quant. Table Quant. Table What is this histogram similar to? Survey MPEG Format MPEG Format MPEG Format Horizontal Derivative Hist. Laplace ML Algorithm GOP correlation GOP correlation GOP correlation Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary Given an image, lets try to find its reflectance part by fitting the derivative’s histogram to the expected Laplace. This can be done by taking the pixel wise median over several frames (Maximum Likelihood algorithm)

Maximum Likelihood Algorithm - cont Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Maximum Likelihood Algorithm - cont Incentives Shadow Characteristics Quant. Table Quant. Table Quant. Table In a formal manner: Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation Where: On – is the nth derivative filter The median is for each pixel over a series of frames Note: this is true as long as the reflectance image is indeed constant Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary Retrieving the luminance image is now easy: l(x,y,t) = i(x,y,t)-r(x,y)

Maximum Likelihood Algorithm - cont Shadow From Video Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Maximum Likelihood Algorithm - cont Incentives Shadow Characteristics Quant. Table Quant. Table Quant. Table Retrieving the reflectance (and luminance) images: Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary

Maximum Likelihood Algorithm - cont Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Maximum Likelihood Algorithm - cont Incentives Shadow Characteristics Quant. Table Quant. Table Quant. Table Another example: Survey MPEG Format MPEG Format MPEG Format Frame 2 Frame 11 ML reflectance ML illumination 2 ML illumination 11 ML Algorithm GOP correlation GOP correlation GOP correlation Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary How do you produce such a sequence of images?

Maximum Likelihood Algorithm - cont Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Maximum Likelihood Algorithm - cont Incentives Shadow Characteristics A common way is to use a device like this: Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation Double Quantization Double Quantization Double Quantization ML enhancement Summary Summary Summary Summary

ML enhancement This algorithm is based on the one we just explored Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Shadow From Video ML enhancement Incentives This algorithm is based on the one we just explored Uses a static camera view (same as before) Does not assume a Lambertian world, resulting in a changing reflectance image. Shadow Characteristics Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation ML Enhancement Double Quantization Double Quantization Double Quantization Summary Summary Summary Summary

ML enhancement For example: Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Shadow From Video ML enhancement Incentives For example: The difference in reflectance at the border between white stripes and a black road is not constant, since the white stripes change their reflectance as a factor of time (light direction). This causes such areas to be considered (falsely) as part of the illumination image. Shadow Characteristics Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation ML Enhancement Double Quantization Double Quantization Double Quantization Summary Summary Summary Summary

Using JPEG Characteristics Shadow From Video Using JPEG Characteristics ML enhancement – cont. Incentives Shadow Characteristics Note that the problematic areas are where textures exist Overcoming the problem: Start by generating a basic (ML) reflectance image Try to move the textures from the illumination image to the reflectance image. One way to do it is: Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation ML Enhancement Double Quantization Double Quantization Double Quantization Summary Summary Summary Summary

ML enhancement – cont. This (almost) solves the problem we saw before: Using JPEG Characteristics Using JPEG Characteristics Shadow From Video Using JPEG Characteristics ML enhancement – cont. Incentives Shadow Characteristics This (almost) solves the problem we saw before: Quant. Table Quant. Table Quant. Table Survey MPEG Format MPEG Format MPEG Format ML Algorithm GOP correlation GOP correlation GOP correlation ML Enhancement Double Quantization Double Quantization Double Quantization Summary Summary Summary Summary

Using JPEG Characteristics Shadow From Video Using JPEG Characteristics Using JPEG Characteristics Using JPEG Characteristics Summary Incentives JPEG Format Shadow Characteristics Quant. Table Quant. Table Quant. Table Quant. Table Shadow removal can be helpful as a pre-processing stage in segmentation and pattern matching algorithms Shadow modeling is difficult. Some assumptions must be made. Some of the assumptions introduced: Lambertian world, natural image statistics, constant reflectance image … Assuming a changing reflectance gives better result, but relies on additional assumptions Survey ML Algorithm ML enhancement Summary P-frames are not discussed here, since a different encoder may also generate a different motion vector – causing the P frames to look completely different (so the double-quantization is undetectable)

Using JPEG Characteristics References JPEG Format Quant. Table A. Prati, R. Cucchiara, I. Mikic, M.M. Trivedi, "Analysis and detection of shadows in video streams: a comparative evaluation" in Proceedings of Computer Vision and Pattern Recognition conference (CVPR 2001), vol. 2, pp. 571-576, 2001 Y. Weiss. “Deriving intrinsic images from image sequences”. In ICCV01, pages II: 68-75. IEEE, 2001 Y. Matsushita and K. Nishino, “Illumination normalization with time-dependent intrinsic images for video surveillance,” IEEE Trans. Pattern Anal. Mach. Intell., 26, pp. 1336-1348 (2004) Athinodoros S. Georghiades Peter N. Belhumeur, “From Few to Many: Illumination Cone Models for Face Recognition Under Variable Lighting and Pose”, IEEE Trans. on Pattern Analysis and Machine Intelligence MPEG Format GOP correlation Double Quantization Summary P-frames are not discussed here, since a different encoder may also generate a different motion vector – causing the P frames to look completely different (so the double-quantization is undetectable)