1. Real-time wave acoustics for games Nikunj Raghuvanshi Microsoft Research

2. Immersion Image taken from: incrysis.com

3. Sound in games today Manually tuned roll-off/occlusion/reverb filters Realism requires tedious labor Reduces time for creativity 3

4. Physically-based sound Physics Perception automation control Use auditory perception  compact, parametric models realistic-sounding starting point artistic control quality-efficiency tradeoff More time for creative work “Physically-based”

5. Sound Propagation (Acoustics) Sound waves propagate from source, complex interactions with boundary Diffraction, high-order reflection, scattering Listener Source

6. Sound Propagation Essential for immersion in acoustic spaces Physically complex, perceivable effects Positive pressure Negative pressure

7. Diffraction Sound does NOT propagate in rays like light, it bends So, you often hear what you don’t see Low-frequencies bend more: occluders act as low-pass filters Light Sound

8. Scattering Light Sound Geometric details cause scattering: A furnished room sounds very different from an empty one

9. Games today Hand-tuned filters: difficult & tedious Simulated propagation → automatic, detailed, scene-dependent acoustics Direct path muffled Reflections muffled Direct path muffled Reflections clear Direct path clear Reflections muffled Images: Dmitri Gait © 2003

10. Ray-based methods Beam-tracing [Funkhouser et. al. 2004] Ray frustum-tracing [Chandak et. al., 2008] Image-source method [Tsingos, 2009] Problems: diffraction/scattering, high-order reflections challenging

11. Our Approach Pre-computed wave simulation attenuation behind obstructions, low-pass filtering scattering from complex surfaces realistic, scene-dependent reverb works with “triangle-soup” scenes Pre-compute & render scene geometry not needed at runtime

12. Comparison with Half Life 2 TM “Train Station” scene Our engine vs. game’s default sound engine

15. Pre-compute & Render Impulse Response Time Pressure Precompute direct reflected Render ? + = Pressure Time * Convolution

16. Moving Source & Listener Time Pressure

17. Moving Source & Listener Time Pressure

18. Moving Source & Listener Time Pressure

19. Moving Source & Listener Time Pressure

20. Moving Source & Listener Time Pressure

21. Main Challenge Acoustic responses vary spatially 7D sampling space source (3D) x listener (3D) x time (1D) Can reduce to 6D Restrict listener to 2.5D surface Brute-force sampling still infeasible

22. Brute-force sampling Scene: Half-Life 2’s Citadel (28m x 60m x 32m) Simulation grid (12 cm) Direct storage: 200,000 GB

23. Main ideas Compact representation for acoustic impulse responses works well with wave simulations Spatial interpolation allowing coarse (~1m) grid Real-time wave effects automatic correspondence to scene geometry

24. Our approach: Overview Source locations Listener locations ~ 1-2 meters

25. Wave Simulation Response (Listener at source location) Encode this compactly Simulator: ARD [Raghuvanshi et. al., 2009] Fast and memory-efficient Input: scene geometry (triangle mesh) acoustic material properties Source pulse Positive pressure Negative pressure

26. Psycho-acoustics Time Pressure Early Reflections Late Reverberation Direct Sound: sense of direction Early Reflections: loudness, timbre; spatial variation; ~100 ms Late Reverberation: decay envelope; no spatial variation; few seconds Reference: “Room Acoustics” by Heinrich Kuttruff

27. Demo Perceptual effect Only Direct sound Direct + Early Reflections Direct + Early Reflections + Late Reverberation

29. Time Pressure Record response Probe Source Listener Technique: Wave Simulation

30. Early Reflections (ER) Late Reverberation (LR) Early Reflections / Late Reverb Automatic separation: echo density (500 peaks/sec) single late reverb filter per room: room-dependent reverb early reflection sampled on a grid within room Store one per room Peak Detection Time Pressure

31. Store, [ Peak times and amplitudes Frequency trend ER Representation Early Reflections (ER) Late Reverberation (LR) captures reflection/diffraction strengths and delays captures spectral effects (low-pass filtering etc.) Time Pressure ]

32. Runtime: Spatial Interpolation Time Pressure P1 P2 Time Pressure P1 Ours Time Pressure Linear Time Pressure peak aliasing no aliasing ?P1P2

33. Runtime: Audio Engine Game Engine Audio Engine source listener Source position Listener position Impulse Response Interpolated Early reflection Room Late reverb + Source sound * Convolve Output

34. Results: Citadel Walkthrough Game environment from Half Life 2 TM Size: 60m x 28m x 22m

36. Results: Walkway Size: 19m x 19m x 8m Realistic attenuation behind walls

38. Results: Walkway Sound focusing Loudness increases beneath concave reflector

40. Results: Living Room Empty vs. furnished living room Scattering off furnishings changes acoustics

43. Integration with Half Life 2 TM Train Station scene: 36m x 83m x 32m

45. Cost Run-time: ~5% of a core per source Reduce further: partitioned convolution Memory: <100 MB – current work to reduce to <10 MB Pre-compute: few minutes per source location, few hours per scene Reference processor: 2.8GHz Quad-core Intel Xeon

46. Summary First real-time wave-based sound propagation engine Automatic diffraction, occlusion/obstruction, late reverberation, focusing, in complex 3D scenes Moving sources and listener

47. Future Work Need further 10x reduction in memory: ~5 MB footprint spatial compression adaptive sampling Dynamic geometry could be added Pre-computed frequency-dependent occlusion factors Integration with existing audio middleware

48. Conclusion Wave acoustics for games automatic scene-dependent roll-off, obstructions, focusing, reverb etc., depending on source/listener location Physically-based techniques are a powerful way to deal with game sounds automation for the tedious part of audio design

49. Acknowledgements Collaborators: Ming C. Lin, UNC Chapel Hill John Snyder, Microsoft Research Naga K. Govindaraju, Microsoft Ravish Mehra, UNC Chapel Hill Kristofor Mellroth, Microsoft Game Studios

50. Thank you!! Questions? http://research.microsoft.com/people/nikunjr/ Papers, demos etc. nikunjr@microsoft.com

52. State-of-the-art: Games Image © Dmitry Gait 2003 Occlusion Obstruction Exclusion Hand-tuned filters: tedious Simulated propagation → automatic, detailed scene-dependent acoustics

53. Technique: Peak Detection search for local extrema extract peak delays and amplitudes wide-band information

54. Early Reflections (ER) Late Reverberation (LR) Time Frequency Technique: Frequency Trend Extraction Band-limited FFT

55. Technique: Frequency Trend Extraction peak data does not capture downward trend (diffraction) Early Reflections (ER) Late Reverberation (LR) Time Frequency FFT

56. Divide Frequency trend Extrapolated Technique: Frequency Trend Extraction Early Reflections (ER) Late Reverberation (LR) Time Frequency FFT

57. Divide Store, [ ] Peak times and amplitudes Frequency trend Extrapolated Technique: ER Representation Early Reflections (ER) Late Reverberation (LR) Time Frequency FFT

58. Peak Detection search for local extrema extract peak delays and amplitudes – captures reflection/diffraction strength and delay

59. impulse response interpolation Runtime: Spatial Interpolation listener source

60. Applying the acoustic response Convolve source sound with response Need fast convolution Fast frequency-domain convolutions bottleneck: FFT need a frequency-domain audio pipeline no need to access scene geometry at runtime

61. Propagation Engine Per Ear. FFT IFFT Output Short FFT Pre- baked Input ER LR Frequency Trend. + + Source 2 Source n : element-wise multiplication. : element-wise sum +