1. Precomputed Wave Simulation for Real-Time Sound Propagation of Dynamic Sources in Complex Scenes Nikunj Raghuvanshi †‡, John Snyder †, Ravish Mehra ‡, Ming C. Lin ‡, and Naga K. Govindaraju † † Microsoft Research ‡ University of North Carolina at Chapel Hill

2. Sound Propagation Essential for immersion Physically complex, perceivable effects Positive pressure Negative pressure

3. State-of-the-art: Games Hand-tuned filters: difficult & tedious Simulated propagation → automatic, detailed, scene-dependent acoustics Direct path muffled Reflections muffled Direct path muffled Reflections clear Direct path clear Reflections muffled Images: Dmitri Gait © 2003

4. Previous Work Beam-tracing [Funkhouser et. al. 2004] Frustum-tracing [Chandak et. al., 2008] Image-source method [Tsingos, 2009] Geometric methods: diffraction/scattering, high-order reflections challenging

5. Our Approach Wave simulation –diffraction: attenuation behind obstructions, low-pass filtering –scattering from complex surfaces –dense reverberation Precompute & render –similar to [James et al. 06] –but propagation, not radiation

6. Comparison with Half Life 2 TM “Train Station” scene Our engine vs. game’s default sound engine

9. Precompute & Render Impulse Response Time Pressure Precompute direct reflected Render ? + = Pressure Time * Convolution

10. Moving Source & Listener Time Pressure

11. Moving Source & Listener Time Pressure

12. Moving Source & Listener Time Pressure

13. Moving Source & Listener Time Pressure

14. Moving Source & Listener Time Pressure

15. Main Challenge Impulse Responses vary spatially 7D sampling space –source (3D) x listener (3D) x time (1D) Can reduce to 6D –Restrict listener to 2.5D surface Brute-force sampling still infeasible

16. Brute-force sampling Scene: “Citadel” (28m x 60m x 32m) Simulation grid (12 cm) –200,000 gigabytes Sub-sampled (~1m) –60 gigabytes –Interpolation non-trivial

17. Main Contributions Compact representation for impulse responses –100x reduction in memory usage –works with (band-limited) wave simulations –plausible high-frequency extrapolation Spatial interpolation allowing coarse (~1m) grid Real-time wave effects –automatic correspondence to scene geometry

18. Our approach: Overview Source locations Listener locations ~ 1-2 meters

19. Wave Simulation Response (Listener at source location) Encode this compactly Simulator: ARD [Raghuvanshi et. al., 2009] Fast and memory-efficient Band-limited (~1kiloHertz) – time/memory constraints during precomputation Source (Gaussian pulse) Positive pressure Negative pressure

20. Psycho-acoustics Time Pressure Early Reflections Late Reverberation Direct Sound: sense of direction Early Reflections: loudness, timbre; spatial variation; 50-100 ms Late Reverberation: decay envelope; no spatial variation; few seconds Reference: “Room Acoustics” by Heinrich Kuttruff

21. Demo Perceptual effect –Only Direct sound –Direct + Early Reflections –Direct + Early Reflections + Late Reverberation

23. Time Pressure Record response Probe Source (Gaussian) Listener Technique: Wave Simulation

24. Technique: Peak Detection search for local extrema extract peak delays and amplitudes wide-band information

25. Early Reflections (ER) Late Reverberation (LR) Technique: ER/LR Decomposition separation based on echo density (500 peaks/sec) single LR filter per room 10x reduction in memory & precomputation time ER varies spatially Store one per room

26. Early Reflections (ER) Late Reverberation (LR) Time Frequency Technique: Frequency Trend Extraction Band-limited FFT

27. Technique: Frequency Trend Extraction peak data does not capture downward trend (diffraction) Early Reflections (ER) Late Reverberation (LR) Time Frequency FFT

28. Divide Frequency trend Extrapolated Technique: Frequency Trend Extraction Early Reflections (ER) Late Reverberation (LR) Time Frequency FFT

29. Divide Store, [ ] Peak times and amplitudes Frequency trend Extrapolated Technique: ER Representation Early Reflections (ER) Late Reverberation (LR) Time Frequency FFT

30. impulse response interpolation Runtime: Spatial Interpolation listener source

31. Runtime: Spatial Interpolation Time Pressure P1 P2 Time Pressure P1 Ours Time Pressure Linear Time Pressure peak aliasing no aliasing ?P1P2

32. Runtime: Render Fast frequency-domain convolutions –bottleneck: FFT –Performed using Intel MKL (single-threaded)

33. Results: Citadel Walkthrough Game environment from Half Life 2 TM Size: 60m x 28m x 22m

35. Results: Walkway Size: 19m x 19m x 8m Realistic attenuation behind walls

37. Results: Walkway Sound focusing –Loudness increases beneath concave reflector

39. Results: Living Room Empty vs. furnished living room Scattering off furnishings changes acoustics

42. Integration with Half Life 2 TM Train Station scene: 36m x 83m x 32m

44. Cost Run-time: ~5% of a core per source Memory: few hundred MBs Precompute: few hours per scene Machine: 2.8GHz Quad-core Intel Xeon, 2.5GB RAM

45. Conclusion First real-time wave-based propagation engine –diffraction, occlusion/obstruction, late reverberation, focusing, complex 3D scenes Moving sources and listener Compact IR encoding – 100x compression Fast, high-quality IR interpolation on coarse (~1m) grid

46. Future Work Need further 10-100x reduction in memory –4x: 32-bit peak/frequency trend data → 8-bit on decibel scale –temporal IR compression: loudness, comb-filtering etc. –spatial compression + adaptive sampling Scalability: cells and portals for sound –efficient boundary integral for portal transfers Limited dynamic geometry –Precomputed frequency-dependent occlusion factors Runtime: Use GPU for convolution (FFT and IFFT)

47. Thank you!! Go to: http://research.microsoft.com/ en-us/um/people/nikunjr/siggraph2010/FastWave.avi http://research.microsoft.com/ en-us/um/people/nikunjr/siggraph2010/FastWave.avi Hear the video over headphones

48. Runtime Processing: Rendering Per Ear. FFT IFFT Output Short FFT Pre-baked Input ER LR Frequency Trend. + + Source 2 Source n : element-wise multiplication. : element-wise sum +

50. Overview State of the art Our approach Results Conclusion

51. Overview State of the art Our approach Results Conclusion

52. Overview State of the art Our approach Results Conclusion

53. Overview State of the art Our approach Results Conclusion

54. Overview State of the art Our approach Results Conclusion

55. State-of-the-art: Games Image © Dmitry Gait 2003 Occlusion Obstruction Exclusion Hand-tuned filters: tedious Simulated propagation → automatic, detailed scene-dependent acoustics