1. Photorealistic Rendering of Rain Streaks Department of Computer Science Columbia University Kshitiz Garg Shree K. Nayar SIGGRAPH Conference July 2006, Boston, USA Sponsors: NSF, ONR

2. A Rain Scene from “The Matrix Revolutions”

3. Frame 1Frame 2Frame 3

4. Rendering Streaks using Simple Brightness Model Commercial SoftwareHeuristic-based Hybrid-Spectral Method ATI ToyShop Demo [Starik and Werman 03] [Langer et al. 04] [N. Tataturschuk et al. 05]

5. Original Image Courtesy Andy Lomas Hand Drawn Textures Original Image Courtesy Warner Bros. Frame from “The Matrix Revolutions” Rendering Streaks using Hand Drawn Textures

6. Study of Visual Appearance of Rain Streaks Rain Streak Appearance Model Rain Streak Texture Database Image-Based Rain Rendering Algorithm

7. Oscillations in a Rain Drop A Falling Rain Drop [Beard and Kubesh 1989 Andsager et al. 1992] Turbulent Flow of air Turbulent Flow of air

8. Oscillations in a Rain Drop Oblate-Prolate ModeTransverse Mode Oscillations in a Rain Drop [Beard and Kubesh 1989 Andsager et al. 1992]

9. Appearance of a Rain Drop Refraction Internal Reflection Specular Reflection N

10. Appearance of a Non-Oscillating Rain Drop N Rain Streak Instant Time Shots Falling Drop Non-Oscillating Rain Drop

11. N Rain Streak Instant Time Shots Falling Drop Appearance of an Oscillating Rain Drop Oscillating Rain Drop

12. Rain Streak Appearance Model Oblate-Prolate Mode Transverse Mode ?? Shape of Rain Drop: Fall Direction Not Rotationally Symmetric [Beard and Kubesh 1989 Andsager et al. 1992] f drop = (0 0 – 360 0 )

13. Measurement of Rain Streak Appearances Drop generator Height: 15 m -z x x drop y,y drop f drop f lig ht q vie w q lig ht q light = (50 0, 90 0, 130 0 ) f light = (10 0, 30 0,…, 170 0 ) q view = (70 0, 90 0, 110 0 ) (A 2,0 A 3,1 f drop ) =(10 images per light & view) Lighting and Viewing Configuration Experimental Setup Oscillation Parameters

14. Estimation Result 90 70 130 50 130 90 50 130 90 50 f lig ht q vie w q lig ht 110 90 1301070301015030101105017030170901105013030 Estimating the Oscillation Amplitudes Measured Oscillation Amplitudes:

15. Comparison of Actual vs. Rendered Rain Streaks 90 70 130 50 130 90 50 130 90 50 f lig ht q vie w q lig ht 110 90 1301070301015030101105017030170901105013030 Rendered Measured

16. Precomputed Database of Streak Textures Lighting and Viewing Direction Oscillations Parameters Database contains 6300 HDR Rain Streak Textures (400 MB) Rain Drop Size Source and Camera Distance Camera Parameter Factors Affecting the Appearance ( Publicly available: Please send email to rainstreaks@cs.columbia.edu )rainstreaks@cs.columbia.edu Render Online (Algorithm) Render Offline (Database)

17. User Input Scene Image/Video Camera and Rain Parameters Coarse Depth Map Light Source Properties Scene Image/Video With Rain Raindrop Distribution and Trajectories Using Particle System Render Streak Appearance Scale Rotate Perspective Effects Camera Effects Exposure Crop/ Blend Defocus Blur Rain Dynamics Rain Rendering Pipeline Render Novel Streak Textures Rain Streak Database Streak texture (Source 1) Streak texture (Source n) + + Streak texture (Ambient)

18. Rendered Rain with Varying Light Direction -150 o 150 o Light Source Direction

19. Constant Brightness Streak Model

20. -150 o 150 o Light Source Direction Comparison with Constant Brightness Model Streak Appearance Model Constant Brightness Model

21. Night Scene with Isotropic Source Coarse Depth Map Image of the Scene

22. Night Scene with Isotropic Source Rendered Rain

23. Night Scene with Isotropic Source Defocus Effects

24. Scene with Multiple Sources Scene with Colored and Anisotropic Sources Coarse Depth Map

25. Scene with Colored and Anisotropic Sources Rendered Rain

26. Scene with Colored and Anisotropic Sources Streak Appearance Model Constant Brightness Model

27. Scene with Changing Sky Illumination Input Scene Video

28. Scene with Changing Sky Illumination Rendered Rain

29. A Scene with Moving Light Sources Input Scene Video

30. Rendered Video A Scene with Moving Light Sources

31. Future Extensions Real-time Implementation of Rendering Algorithm Incorporating the Effects of Wind and Turbulence

32. Summary Image-Based Rain Rendering Algorithm Rain Streak Texture Database ( Publicly available: rainstreaks@cs.columbia.edu )rainstreaks@cs.columbia.edu Rain Streak Appearance Model

33. Complex Appearances of Rain Streaks Real Images of Rain Streaks