Detection and Removal of Rain from Videos Department of Computer Science Columbia University Kshitiz Garg and Shree K. Nayar IEEE CVPR Conference June 2004, Washington DC, USA Sponsors: NSF, DARPA

Outdoor Vision and Weather Fog Mist [Nayar and Narasimhan,99] [Narasimhan and Nayar, 00,01,02,03] RainSnow

Steady versus Dynamic Weather Steady Weather (Fog, Mist)Dynamic Weather (Rain, Snow) Pixel Aggregate scattering Fog-droplet (1-10) Individual scattering Raindrop (0.5-10) Steady Weather Objects Dynamic Weather

Previous Work on Rain Atmospheric Sciences –Physical Properties of Rain Communication –Signal Transmission through Rain Graphics –Particle Systems –Heuristic Based Methods [Beard and Chung 1987, Wang 1975, Marshall and Palmer 1948, Mason 1975] [ Ting 1982, Manning 1993, Robert 1992 ] [ Reeves 1983, Sims 1997 ] [ Starik and Werman 2003 ]

Image Processing Does Not Suffice Median Filtering Difference (Original and Median Filtered) Rain with Scene MotionMedian Filtered Video Courtesy © New Line Productions, Inc.

Modeling the Visual Effects of Rain Rain Drop Rain Streak Dynamics Rain Photometry

Physical Properties of Rain Drops N (1/m) Drop density Beard and Chung 1987 Size Distribution of Rain Drop Drop radius (mm) a Number Density Marshall and Palmer % of Rain Drops are Spherical Shapes of Rain Drop Falling Direction mm

Visual Appearance of a Rain Drop Refraction Internal Reflection Specular Reflection N Beard and Chung 1987 Marshall and Palmer 1948

Brightness Properties of a Rain Drop Distance from the center of the drop Refraction Specular Internal Reflection Rain Drops are Brighter than Background FOV=165 0 Background Brightness is Independent of Background 94%

The World in a Drop Perspective Views of the World A Drop World in a Drop

Rain and Background Intensities Normal Shutter Speed ( Exposure Time = 1/30 s ) High Shutter Speed ( Exposure Time = 1/1000 s )

Motion Blur Model for Rain Drops Pixel Irradiance Background Time a drop stays over a pixel : Time Background Stationary Rain Drop Exposure Time Rain produces a “Delta Signal” in Time Pixel Frame Change in Intensity :

Motion Blurred Intensities of Rain Streaks Change in Intensity is Linearly related to Background Constants Intensity of the Background : Intensity at a Pixel on a Rain Streak : Change in Intensity due to a Rain Streak : Irradiance Time Background Stationary Rain Drop Exposure Time

Dynamics of Rain Pin Hole Continuous Space Time Domain Discrete Space Time Volume Frames X-axis Time Y-axis Rain Produces Strong Directional Correlations in Videos Background

Rain Detection and Removal Algorithm Estimated Rain FieldCorrelation MapRain Segmented Original Video Candidate Rain Pixels Estimated Rain Pixels Select Delta in time Linear Photometric Constraint Dynamic Constraint Compute direction and strength Neighboring Frames

Results: A clip from Magnolia Original Clip Rain Detection Courtesy © New Line Productions, Inc.

Results: A clip from Magnolia Derained VideoOriginal Video Rain Component

Results: Rain vs Ripples Rain Segmentation Original Video

Results: Rain vs Ripples Original VideoDerained VideoRain Component

Results: Person Walking in Rain Original Video Rain Detection

Results: Person Walking in Rain Original Video Derained Video Rain Component

Summary Visual Appearance of Rain Rain Rain Detection and Removal Future Work: Snow Photometry Dynamics Photometry +=