1. Simulation of High Quality Sound Fields for Interactive Graphics Applications Nicolas TSINGOS iMAGIS - GRAVIR / IMAG - INRIA UMR CNRS C5527

2. Introduction Explosion of "3D sound" techniques
consumer products
Multi-modal experiences
computer animation, video games
simulators, teleconferencing
Acoustic simulations
room and environmental acoustics

3. Overview Context Previous approaches
Interactive treatment of sound occlusion
Integrated sound and graphics rendering system
Adaptive simulation technique

4. Underlying context Modeling and propagation simulation
wave theory
geometrical acoustics
statistical acoustics
Human hearing and restitution systems
sound perceived in 3D
need to reproduce spatial audio
Signal processing

5. Basics of sound rendering*
source
receiver
Variations of the pressure
propagation delay (sound speed  340 m/s)
All info can be represented by a digital filter
impulse response
Rendering Convolution

6. Overview Context Previous approaches
Interactive treatment of sound occlusion
Integrated sound and graphics rendering system
Adaptive simulation technique

7. Previous work: acoustic simulations
Finite element approaches [Jean98, Hothersall+91,Kopuz+95,Kludsuweit91,Wright95]
solve the wave equation
discretize space and time
treat all propagation phenomena
high computational cost (2D and steady state)
Geometrical acoustics
sound rays
valid for high frequencies

8. Previous work: image sources
[Allen+79,Borish84, Foster+91, Strauss+95]
Specular reflections
Straightforward but exponential cost

9. Previous work: Ray/beam tracing
[Martin+93,Dalenbäck96, Funkhouser+98, Monks+96]
More general
Faster but less flexible updates

10. Previous work: Radiant exchanges
[Kuttruff71, Lewers93, Goral+84, Cohen+85, Nishita+85]
Diffuse reflections [Hodgson91]

11. Previous work: interactive acoustics
Artificial reverberators [Schroeder62,Moorer79,Jot+92]
limited control
Audio and video integration [Takala+92,Hahn+95]
post-processing "timbre-trees"
Multi-media libraries [SGI Cosmo 3D, Intel RSX, Microsoft Direct Sound]
limited propagation effects

12. Overview Context Previous approaches
Interactive treatment of sound occlusion
Integrated sound and graphics rendering system
Adaptive simulation technique

13. Sound waves occlusion Obstacles cause diffraction
not a 0/1 "visibility"
frequency dependent
Simulating diffraction
finite elements
diffracted rays [Keller62,Kouyoumjian+74]
difficult and costly
from Isaac Newton's Principia (1686)

14. An interactive geometrical approach
Use a 3D polygonal model
identify the diffracting objects
use graphics hardware
Keep the frequency dependent aspect
"Ray-tracing"
use "thick" rays
defined by first Fresnel ellipsoids
Extended visibility term between 0 and 1

15. - Fresnel ellipsoids + + M R S
k =1 M S + R - +
Alternate constructive and destructive contributions
Twice the unoccluded energy in the first ellipsoid

16. General algorithm occluders Receiver occlusion map Source and receiver
positions
Receiver
For each frequency
Render occlusion map
Compute attenuation
occlusion map
Source
Build filter

17. Computing the occlusion term
Render the objects in the first Fresnel ellipsoids
parallel projection
Occlusion factor: (occluded area) (area of the largest Fresnel zone)
400 Hz
4000 Hz

18. Results [GI97]
Results
Source v t
Receiver t v
Receiver
Source

19. The Fresnel-Kirchoff diffraction theory
A wave is a sum of "wavelets"
Kirchoff integral theorem
Contribution of unoccluded "wavelets"
Secondary
wavelets R s
Primary wave

20. Computing the diffraction integral
Compute a depth map of the obstacles
read the Z-buffer
For each occluded pixel
evaluate occluded contribution
subtract
obstacle R S

21. Results: diffraction patterns [AES98]
sampling a receiving plane
200*200 pixels
0.02 sec. / point avg.
(180 MHz SGI O2 workstation)

22. Diffraction patterns (II) [AES98]
Square apertures
wide aperture
narrow aperture
close-up
Fresnel diffraction
Fraunhofer diffraction

23. Summary Two methods to compute sound occlusion Fresnel ellipsoids
generic and fast
use graphics hardware
Fresnel ellipsoids
Fresnel-Kirchoff integration

24. Overview Context Previous approaches
Interactive treatment of sound occlusion
Integrated sound and graphics rendering system
Adaptive simulation technique

25. Integrating sound with 3D graphics
Fabule platform
Sound path
Image-source model
Doppler shifting
Occlusions
Source, receiver and surface characteristics
Télémédia project (CNET)

26. Overview Context Previous approaches
Interactive treatment of sound occlusion
Integrated sound and graphics rendering system
Adaptive simulation technique

27. Goal Treat both diffuse and specular reflections
source
receiver
Treat both diffuse and specular reflections
Listener independent solution
Radiant exchanges between patches

28. Hierarchical radiosity
Designed for lighting simulations [Hanrahan+91,Cohen+88]

29. Extension to temporal phenomena
Echoes
frequency band
intensity (I)
arrival time (T)
duration
Temporal radiosity
list of echoes stored on patches (echograms)
duration I t T

30. Temporal transport spreading duration duration+spreading I FF I.FF t t
Tmin
Tmax
T+ Tmin
Tmax Pj Pi
Tmin

31. Hierarchical simulation
Efficient hierarchical representation
Refinement
based on energy
based on echo spreading

32. Adding specular reflections
Non diffuse radiosity [Sillion+89/91, Immel+86, Aupperle+93,Christensen+96]
Image-sources model
Hierarchical specular exchanges
use patches centroids
shooter
reflector
gatherer

33. Echo merging Merge echoes within a given temporal threshold
Control the time-complexity
Take interferences into account
use difference in arrival times

34. Results [Siggraph97(sketch)]
Impulse responses
listener-independent
Visualization
wavefront propagation
energy mappings
Acoustic predictions
Collaboration with CSTB

35. Conclusion Two fast sound occlusion models
use graphics hardware
Integrated sound rendering system
computer animation and virtual reality
A model for temporal radiant exchanges
hierarchical
both diffuse and specular reflections
listener-independent solution
tunable time/accuracy tradeoff

36. Extensions Validation tests in progress Treat high reflection orders
adaptive echo bucketing
statistical approaches [Monks+93,Martin+93]
Clustering [Sillion+95,Paquette+98]
Dynamic environments
fast update of energy transfers [Drettakis+97]
Application to radio waves
cellular phones and wireless networks

37. Push-pull Push Pull copy echoes in sons' echograms preserve intensity
reduce width
Pull
copy echoes in father's echogram
multiply intensity by area ratio
do not combine echoes

38. Reconstructing an impulse response
Energetic response
wide echoes (specular+diffuse paths)
dirac impulses (pure specular paths)
Reconstruct wide echoes
render the echoes
energetic enveloppe
use white noise to get the missing phase
Band pass filter and add up

39. Updating the sound path information
Evaluate the delay at each time-step
iterative approach [Noser+95]
simple interpolation
Combination of 4 filters
source, receiver, environment, reflection
short Finite Impulse Response (128 pts/32 kHz)
Calculating the sound occlusion
use a mirrored scene for image-sources
use the previous semi-quantitative approach