1. Detection and Removal of Rain from Videos Department of Computer Science Columbia University Kshitiz Garg and Shree K. Nayar IEEE CVPR Conference June 2004, Washington DC, USA Sponsors: NSF, DARPA

2. Outdoor Vision and Weather Fog Mist [Nayar and Narasimhan,99] [Narasimhan and Nayar, 00,01,02,03] RainSnow

8. Steady versus Dynamic Weather Steady Weather (Fog, Mist)Dynamic Weather (Rain, Snow) Pixel Aggregate scattering Fog-droplet (1-10) Individual scattering Raindrop (0.5-10) Steady Weather Objects Dynamic Weather

9. Previous Work on Rain Atmospheric Sciences –Physical Properties of Rain Communication –Signal Transmission through Rain Graphics –Particle Systems –Heuristic Based Methods [Beard and Chung 1987, Wang 1975, Marshall and Palmer 1948, Mason 1975] [ Ting 1982, Manning 1993, Robert 1992 ] [ Reeves 1983, Sims 1997 ] [ Starik and Werman 2003 ]

10. Image Processing Does Not Suffice Median Filtering Difference (Original and Median Filtered) Rain with Scene MotionMedian Filtered Video Courtesy © New Line Productions, Inc.

11. Modeling the Visual Effects of Rain Rain Drop Rain Streak Dynamics Rain Photometry

12. Physical Properties of Rain Drops N (1/m) Drop density Beard and Chung 1987 Size Distribution of Rain Drop Drop radius (mm) a Number Density 12 3 4 200 400 600 800 Marshall and Palmer 1948 95% of Rain Drops are Spherical Shapes of Rain Drop Falling Direction 0.5 1 1.5 2 2.5 mm

13. Visual Appearance of a Rain Drop Refraction Internal Reflection Specular Reflection N Beard and Chung 1987 Marshall and Palmer 1948

14. Brightness Properties of a Rain Drop 0.2 0.4 0.6 0.8 1 0.20.40.60.810 Distance from the center of the drop Refraction Specular Internal Reflection Rain Drops are Brighter than Background FOV=165 0 Background Brightness is Independent of Background 94%

15. The World in a Drop Perspective Views of the World A Drop World in a Drop

16. Rain and Background Intensities Normal Shutter Speed ( Exposure Time = 1/30 s ) High Shutter Speed ( Exposure Time = 1/1000 s )

17. Motion Blur Model for Rain Drops Pixel Irradiance Background Time a drop stays over a pixel : Time Background Stationary Rain Drop Exposure Time Rain produces a “Delta Signal” in Time Pixel Frame Change in Intensity :

18. Motion Blurred Intensities of Rain Streaks Change in Intensity is Linearly related to Background Constants Intensity of the Background : Intensity at a Pixel on a Rain Streak : Change in Intensity due to a Rain Streak : Irradiance Time Background Stationary Rain Drop Exposure Time

19. Dynamics of Rain Pin Hole Continuous Space Time Domain Discrete Space Time Volume Frames X-axis Time Y-axis Rain Produces Strong Directional Correlations in Videos Background

20. Rain Detection and Removal Algorithm Estimated Rain FieldCorrelation MapRain Segmented Original Video Candidate Rain Pixels Estimated Rain Pixels Select Delta in time Linear Photometric Constraint Dynamic Constraint Compute direction and strength Neighboring Frames

21. Results: A clip from Magnolia Original Clip Rain Detection Courtesy © New Line Productions, Inc.

22. Results: A clip from Magnolia Derained VideoOriginal Video Rain Component

23. Results: Rain vs Ripples Rain Segmentation Original Video

24. Results: Rain vs Ripples Original VideoDerained VideoRain Component

25. Results: Person Walking in Rain Original Video Rain Detection

26. Results: Person Walking in Rain Original Video Derained Video Rain Component

27. Summary Visual Appearance of Rain Rain Rain Detection and Removal Future Work: Snow Photometry Dynamics Photometry +=