1. Spatio-Temporal Frequency Analysis for Removing Rain and Snow from Videos Carnegie Mellon University June 16, 2007 Peter Barnum Takeo Kanade Srinivasa Narasimhan

2. Bad weather in outdoor videos Rain Snow

3. Previous Work Garg and Nayar CVPR ‘04 Garg and Nayar ICCV ‘05 Image-based blurring Streak detection Hase et al. ICIP ’98 Starik and Werman IWTAS ‘03 Zhang et al. ICME ‘06 Camera-based blurring Pixel Intensity Time Spikes due to rain

4. Groups of streaks

5. Imaging a falling particle Raindrops Gaussians Snowflakes Breadth Length

6. Gaussian model of streak appearance A Gaussian Including streak orientation (just a coordinate space rotation) A blurred Gaussian streak Camera parameters are constant

7. Where are the streaks?

8. Fourier transform of the streaks

9. Building a complete model Blurred Gaussian For all depths that are in-focus For a given precipitation intensity For all common drop sizes

10. Model accuracy Original image 2D Fourier Transform Model Model with 50% randomly set to zero

11. Finding the precipitation rate Rain and snow have two useful properties Large frame-to-frame difference Distributed evenly in frequency space Mailbox Building Snow 0% 100%

12. Frame-to-frame differences t=1 t=2 t=3 w=-1 w=0 w=+1

13. Frame-to-frame differences t=1 t=2 t=3 w=-1 w=0 w=+1

14. Finding the precipitation rate For most objects But for rain and snow Because of these properties

15. Estimating the model parameters The precipitation rate is approximately: Precipitation rate Streak orientation The orientation is found by: Estimating the streak orientation requires a spatial consistent estimate

16. Original image 2D FT Model Frequency space examples

17. Computing per-frequency estimates At a given frequency: =

18. Computing per-pixel estimates

19. Refining the single frame estimate Exactly the same model, constant in w t=1 t=2 t=3

20. Computing per-pixel estimates

24. “Into each life some rain must fall.” -Henry Wadsworth Longfellow Conclusions and future work Refining global estimates with local features A global frequency method for rain and snow removal

25. Extra slides

26. Imaging a falling particle