Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss.

Slides:



Advertisements
Similar presentations
Land’s Retinex algorithm
Advertisements

Robust statistical method for background extraction in image segmentation Doug Keen March 29, 2001.
Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros.
Computer vision: models, learning and inference Chapter 13 Image preprocessing and feature extraction.
DREAM PLAN IDEA IMPLEMENTATION Introduction to Image Processing Dr. Kourosh Kiani
The perception of Shading and Reflectance E.H. Adelson, A.P. Pentland Presenter: Stefan Zickler.
Approaches for Retinex and Their Relations Yu Du March 14, 2002.
Proportion Priors for Image Sequence Segmentation Claudia Nieuwenhuis, etc. ICCV 2013 Oral.
Authers : Yael Pritch Alex Rav-Acha Shmual Peleg. Presenting by Yossi Maimon.
December 5, 2013Computer Vision Lecture 20: Hidden Markov Models/Depth 1 Stereo Vision Due to the limited resolution of images, increasing the baseline.
ECCV 2002 Removing Shadows From Images G. D. Finlayson 1, S.D. Hordley 1 & M.S. Drew 2 1 School of Information Systems, University of East Anglia, UK 2.
Bayesian Image Super-resolution, Continued Lyndsey C. Pickup, David P. Capel, Stephen J. Roberts and Andrew Zisserman, Robotics Research Group, University.
Shadow removal algorithms Shadow removal seminar Pavel Knur.
PSY 5018H: Math Models Hum Behavior, Prof. Paul Schrater, Spring 2004 Vision as Optimal Inference The problem of visual processing can be thought of as.
Recovering Intrinsic Images from a Single Image 28/12/05 Dagan Aviv Shadows Removal Seminar.
Image Morphing : Rendering and Image Processing Alexei Efros.
Background Subtraction : Rendering and Image Processing Alexei Efros.
CSSE463: Image Recognition Day 30 Due Friday – Project plan Due Friday – Project plan Evidence that you’ve tried something and what specifically you hope.
Deriving Intrinsic Images from Image Sequences Mohit Gupta Yair Weiss.
Illumination Normalization with Time-Dependent Intrinsic Images for Video Surveillance Yasuyuki Matsushita, Member, IEEE, Ko Nishino, Member, IEEE, Katsushi.
COMP 290 Computer Vision - Spring Motion II - Estimation of Motion field / 3-D construction from motion Yongjik Kim.
Deriving intrinsic images from image sequences. Yair Weiss, Presentation by Leonid Taycher.
Introduction of the intrinsic image. Intrinsic Images The method of Finlayson & Hordley ( 2001 ) Two assumptions 1. the camera ’ s sensors are sufficiently.
Photometric Stereo & Shape from Shading
Jacinto C. Nascimento, Member, IEEE, and Jorge S. Marques
Scale-Invariant Feature Transform (SIFT) Jinxiang Chai.
Advanced Image Processing Image Relaxation – Restoration and Feature Extraction 02/02/10.
Tricolor Attenuation Model for Shadow Detection. INTRODUCTION Shadows may cause some undesirable problems in many computer vision and image analysis tasks,
CSSE463: Image Recognition Day 30 This week This week Today: motion vectors and tracking Today: motion vectors and tracking Friday: Project workday. First.
Perception Introduction Pattern Recognition Image Formation
Computer Vision, Robert Pless Lecture 11 our goal is to understand the process of multi-camera vision. Last time, we studies the “Essential” and “Fundamental”
December 4, 2014Computer Vision Lecture 22: Depth 1 Stereo Vision Comparing the similar triangles PMC l and p l LC l, we get: Similarly, for PNC r and.
Chap. 9: Image Segmentation Jen-Chang Liu, Motivation Segmentation subdivides an image into its constituent regions or objects Example: 生物細胞在影像序列中的追蹤.
Intelligent Scissors for Image Composition Anthony Dotterer 01/17/2006.
Image Processing Jitendra Malik. Different kinds of images Radiance images, where a pixel value corresponds to the radiance from some point in the scene.
December 9, 2014Computer Vision Lecture 23: Motion Analysis 1 Now we will talk about… Motion Analysis.
Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.
1 Computational Vision CSCI 363, Fall 2012 Lecture 21 Motion II.
Motion Analysis using Optical flow CIS750 Presentation Student: Wan Wang Prof: Longin Jan Latecki Spring 2003 CIS Dept of Temple.
3D Imaging Motion.
Digital Image Processing Lecture 16: Segmentation: Detection of Discontinuities Prof. Charlene Tsai.
1 Motion Analysis using Optical flow CIS601 Longin Jan Latecki Fall 2003 CIS Dept of Temple University.
Segmentation of Vehicles in Traffic Video Tun-Yu Chiang Wilson Lau.
Multiple Light Source Optical Flow Multiple Light Source Optical Flow Robert J. Woodham ICCV’90.
Colour and Texture. Extract 3-D information Using Vision Extract 3-D information for performing certain tasks such as manipulation, navigation, and recognition.
Lecture 9 Feature Extraction and Motion Estimation Slides by: Michael Black Clark F. Olson Jean Ponce.
CSSE463: Image Recognition Day 29 This week This week Today: Surveillance and finding motion vectors Today: Surveillance and finding motion vectors Tomorrow:
Digital Image Processing Lecture 16: Segmentation: Detection of Discontinuities May 2, 2005 Prof. Charlene Tsai.
Thresholding Foundation:. Thresholding In A: light objects in dark background To extract the objects: –Select a T that separates the objects from the.
CS332 Visual Processing Department of Computer Science Wellesley College CS 332 Visual Processing in Computer and Biological Vision Systems Analysis of.
EECS 274 Computer Vision Sources, Shadows, and Shading.
Motion Estimation of Moving Foreground Objects Pierre Ponce ee392j Winter March 10, 2004.
Announcements Project 3a due today Project 3b due next Friday.
From local motion estimates to global ones - physiology:
Motion and optical flow
Environmental Remote Sensing GEOG 2021
CSSE463: Image Recognition Day 29
Intrinsic images and shape refinement
Image Blending and Compositing
Motion and Optical Flow
Image Segmentation – Edge Detection
COMP 9517 Computer Vision Motion 7/21/2018 COMP 9517 S2, 2012.
Range Imaging Through Triangulation
CSSE463: Image Recognition Day 29
Shadow Detection and Removal
Patric Perez, Michel Gangnet, and Andrew Black
CSSE463: Image Recognition Day 29
CSSE463: Image Recognition Day 29
CSSE463: Image Recognition Day 29
Shape from Shading and Texture
Presentation transcript:

Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss

Intrinsic Scene Characteristics Introduced by Barrow and Tanenbaum, 1978 Motivation: Early visual system decomposes image into ‘intrinsic’ properties Input ImageReflectanceOrientationIlluminationDistance

Intrinsic Images Input = Reflectance x Illumination Mid-Level description of scenes Information about intrinsic scene properties Falls short of a full 3D description

Motivation Information about scene properties: prior for visual inference tasks Segmentation: Invariant to illumination OriginalIllumination Reflectance

Motivation Information about scene properties: prior for visual inference tasks Shape from Shading: Invariant to reflectance Original Illumination Reflectance

Problem Definition Given I, solve for L and R such that I(x,y) = L(x,y) * R(x,y) I = Input Image L = Illumination Image R = Reflectance Image

Problem Definition Given I, solve for L and R such that I(x,y) = L(x,y) * R(x,y) (disturbed ) This is preposterous!! You can’t possibly solve this !! Dr. Math Classical Ill Posed Problem: # Unknowns = 2 * # Equations

Problem Definition Given I, solve for L and R such that I(x,y) = L(x,y) * R(x,y) (disturbed ) This is preposterous!! You can’t possibly solve this !! Dr. Math Classical Ill Posed Problem: # Unknowns = 2 * # Equations Hey doc, Don’t PANIC These pixels ‘hang out together’ a lot Mohit Exploit ‘structure’ in the images to reduce the no. of unknowns !

Previous Work Retinex Algorithm [Land and McCann] Reflectance image piecewise constant Illumination is attached shadows (photometric sterero) L(x,y,t) = N(x,y). S(t) Illumination images related by a scalar L(x,y,t) =  (t) * L(x,y)

Previous Work Retinex Algorithm [Land and McCann] Reflectance image piecewise constant Illumination is attached shadows (photometric sterero) L(x,y,t) = N(x,y) * S(t) Illumination images related by a scalar L(x,y,t) =  (t) * L(x,y) All exploit temporal or spatial structure in the images to reduce the no. of unknowns !

Cut to the present… R(x,y,t) = R(x,y) Motivation Lot of web-cam images Stationary camera, reflectance doesn’t change This paper relies on temporal structure

Cut to the present… R(x,y,t) = R(x,y) Motivation Lot of web-cam images Stationary camera, reflectance doesn’t change This paper relies on temporal structure I(x,y,t) = R(x,y) * L(x,y,t) T equations, T+1 unknowns Still an Ill-Posed Problem !!

Slight Detour: Background Extraction Problem: Given a sequence of images I(x,y,t), extract the stationary component, or the ‘background’ from them Images: Alyosha Efros

Image Stack t time We can look at the set of images as a spatio-temporal volume Each line through time corresponds to a single pixel in space If camera is stationary, we can decompose the image as: image static background dynamic foreground i(x,y,t) = b(x,y) + f(x,y,t) Images: Alyosha Efros

Power of Median Image image static background dynamic foreground i(x,y,t) = b(x,y) + f(x,y,t) Key Observation: If for each pixel (x,y), f(x,y,t) = 0 ‘most of the times’ then b(x,y) = median t i(x,y,t) Example: b(x,y) = 42; f(x,y,t) = [0, 2, 3, 0, 0]; i(x,y,t) = [42, 44, 45, 42, 42] b(x,y) = median( [42,44,45,42,42]) = 42 !

Power of Median Image

Median Image = Background !

Background Extraction & Intrinsic Images I(x,y,t) = L(x,y,t) * R(x,y) i(x,y,t) = l(x,y,t) + r(x,y) (log) Compare to i(x,y,t) = f(x,y,t) + b(x,y) Static Background = Reflection Image Moving Foregrounds = Illumination Images (shadows) Intrinsic Image Equation

Trouble! Illumination Images, l(x,y,t) sparse: Not a safe assumption Median Image “Shady” Result

Key Idea: Lets look at gradient images… Gradients of shadows are sparse, even though the shadows aren’t ! Rationale: Smoothness of shadows

Key Idea: Lets look at gradient images… Gradients of shadows are sparse, even though the shadows aren’t ! Rationale: Smoothness of shadows i(x,y,t) = l(x,y,t) + r(x,y) gradient i f (x,y,t) = l f (x,y,t) + r f (x,y)

Key Idea: Lets look at gradient images… Gradients of shadows are sparse, even though the shadows aren’t ! Rationale: Smoothness of shadows i(x,y,t) = l(x,y,t) + r(x,y) gradient i f (x,y,t) = l f (x,y,t) + r f (x,y) l f (x,y,t) is sparse r f (x,y) = median t i f (x,y,t)

Median Gradient Image Filtered Reflectance image r f (x,y) = median t i f (x,y,t) Recovered Reflectance image

Median Gradient Image Filtered Reflectance imageRecovered Reflectance image

Median Gradient Image Filtered Reflectance imageRecovered Reflectance image I(x,y,t) = R(x,y) * L(x,y,t) T equations, T+1 unknowns Still an Ill-Posed Problem ? No, sparsity of gradient illumination images imposes additional constraints!

Recovering image from Gradient Images f(x,y) Horizontal filtered image (v 1 ) Vertical filtered image (v 2 ) f = v  f =. v (del operator) Poisson Equation: f = g (from gradient images: g =.v) Along with the boundary condition v = (v 1,v 2 )

Recovering image from Gradient Images f(x,y) Horizontal filtered image (v 1 ) Vertical filtered image (v 2 ) f = v  f =. v (del operator) Poisson Equation: f = g (from gradient images: g =.v) Along with the boundary coundition v = (v 1,v 2 ) Interpretation of solving the Poisson equation: Computes the function (f) whose gradient is the closest to the guidance vector field (v), under given boundary conditions.

Recovering image from Gradient Images f(x,y) Horizontal filtered image (v 1 ) Vertical filtered image (v 2 ) f = v  f =. v (del operator) Poisson Equation: f = g (from gradient images: g =.v) v = (v 1,v 2 ) Boundary can be from mean of input images – hope that edges are mostly shadow-free +

Poisson Image Editing (Perez, Gangnet, Blake, SIGGRAPH ’03) Source Destination CloningPoisson Blending Want to find a new function f, which ‘looks like’ g in the interior and like f* near the boundary  Use g as guiding vector field with f* providing the boundary condition

Poisson Image Editing (Perez, Gangnet, Blake, SIGGRAPH ’03)

Spatial Structure: Impose Priors Additional constraints required to make the problem tractable Idea: Statistics of natural images  sparse derivative filter outputs Input ImageDerivative filter outputsLaplacian distribution P(x) ~ e -  |x|  MLE value of x is 0

Spatial Structure: Impose Priors Additional constraints required to make the problem tractable Idea: Statistics of natural images  sparse derivative filter outputs Input ImageDerivative filter outputsLaplacian distribution P(x) ~ e -  |x|  MLE value of x is 0 Spatial Structure Imposes more constraints to make the problem tractable !!

The Solution N Filters {f n }, filter outputs o n = i * f n Claim: r n (x,y) = median t o n (x,y,t) Proof Sketch: l n (x,y,t) = i n (x,y,t) – r n (x,y) ( log images) …  t | l n (x,y,t)| = 0 ( MLE estimate of laplacian distributed variable)   t | i n (x,y,t) – r n (x,y) | = 0  r n (x,y) = median t o n (x,y,t)

The Algorithm 1. Filter outputs for input image (o n ) are calculated 2. Filtered reflectance image (r n ) is computed as r n (x,y) = median t o n (x,y,t) 3. Reflectance image r is recovered from r n 4. Illumination images are recovered using the relation: l(x,y,t) = i(x,y,t) – r(x,y)

Question: Why filter? Basic Assumption: l n (x,y,t) follows a laplacian distribution ( is sparse) Reasonable for natural images Taking median of unfiltered images requires l(x,y,t) (unfiltered illumination images) to be sparse This assumption is not valid for most of the cases Unfiltered images  P,Q have different reflectance Filtered images  Same reflectance

Results : Synthetic frame i frame j ML illumination (frame i) ML reflectance ** Note that the pixels surrounding the diamond are always in shadow, yet their estimated reflectance is the same as that of pixels that were always in light.

Results : Real World

Some fun … Original ImageLogo belnded with Image Logo blended with reflectance image, and rendered with corresponding illumination image

Limitations Requires multiple images of a static scene in different lighting Highly sensitive to input - scene content and sequence length (basically a shadow detector !) Can't remove static shadows High complexity - filtering the images and finding median are high cost functions.

Conclusions Fully automatic algorithm to derive intrinsic images from a sequence of images Simplification by making constant reflectance assumption Use sparsity of gradient images to derive a simple solution Paper has a rather complex statistical derivation for the same result ! Doesn’t tackle the original problem of recovering intrinsic images from a single image ( next presentation)