Virtualized Audio as a Distributed Interactive Application Peter A. Dinda Northwestern University Access Grid Retreat, 1/30/01

2 Overview Audio systems are pathetic and stagnant We can do better: Virtualized Audio (VA) VA can exploit distributed environments VA demands interactive response What I believe Why I care

3 Performer Microphones Performance Room Mixer Amp Listening Room Listener Sound Field 1 Sound Field 2 Loudspeakers Headphones Traditional Audio (TA) System

4 TA Mixing And Filtering Performer Performance Room Filter Mixing (reduction) Amp Filter Loudspeaker Filter Microphone Sampling Listening Room Filter Listener’s Location and HRTF Headphones Perception of Loudspeaker Reproduced Sound Listener’s Location and HRTF Perception of Real Sound Perception of Headphone Reproduced Sound

5 Virtualized Audio (VA) System

6 VA: Filtering, Separation, and Auralization VA Forward Problem VA Reverse Problem

7 The Reverse Problem - Source Separation Human SpaceMicrophones Recovery Algorithms microphone signals microphone positions “Reverse Problem” sound source positions room geometry and properties sound source signals other inputs Microphone signals are a result of sound source signals, positions, microphone positions, and the geometry and material properties of the room. We seek to recover these underlying producers of the microphone signals.

8 The Reverse Problem Blind source separation and deconvolution Statistical estimation problem Can “unblind” problem in various ways –Large number of microphones –Tracking of performers –Separate out room deconvolution from source location –Directional microphones –Phased arrays Potential to trade off computational requirements and specialized equipment Much existing research to be exploited

9 Transducer Beaming Transducer Wave L  L  L  L  L  L

10 Phased Arrays of Transducers Phased Array Physical Equivalent

11 The Forward Problem - Auralization Auralization Algorithms sound source positions room geometry/properties sound source signals Listener positions Listener signals Listener wearing Headphones (or HSS scheme) In general, all inputs are a function of time Auralization must proceed in real-time

12 Ray-based Approaches To Auralization For each sound source, cast some number of rays, then collect rays that intersect listener positions –Geometrical simplification for rectangular spaces and specular reflections Problems –Non-specular reflections requires exponential growth in number of rays to simulate –Most interesting spaces are not rectangular

13 Wave Propagation Approach Captures all properties except absorption absorption adds 1st partial terms  2 p/  2 t =  2 p/  2 x +  2 p/  2 y +  2 p/  2 z

14 Method of Finite Differences Replace differentials with differences Solve on a regular grid Simple stencil computation (2D Ex. in Fx) Do it really fast pdo i=2,Y-1 pdo j=2,X-1 workarray(m0,j,i) = (.99) * ( $ R*temparray(j+1,i) $ + 2.0*(1-2.0*R)*temparray(j,i) $ + R*temparray(j-1,i) $ + R*temparray(j,i+1) $ + R*temparray(j,i-1) $ - workarray(m1,j,i) ) endpdo

15 How Fast is Really Fast? O(xyz(kf) 4 / c 3 ) stencil operations per second are necessary –f=maximum frequency to be resolved –x,y,z=dimensions of simulated space –k=grid points per wavelength (2..10 typical) –c=speed of sound in medium for air, k=2, f=20 KHz, x=y=z=4m, need to perform 4.1 x stencil operations per second (~30 FP operations each)

16 LTI Simplification Consider the system as LTI - Linear and Time-Invariant We can characterize an LTI system by its impulse response h(t) In particular, for this system there is an impulse response from each sound source i to each listener j: h(i,j,t) Then for sound sources s i (t), the output m j (t) listener j hears is m j (t) =  i  h(i,j,t) * s i (t), where * is the convolution operator

17 LTI Complications Note that h(i,j) must be recomputed whenever space properties or signal source positions change The system is not really LTI –Moving sound source - no Doppler effect Provided sound source and listener movements, and space property changes are slow, approximation should be close, though. Possible “virtual source” extension

18 Where do h(i,j,t)’s come from? Instead of using input signals as boundary conditions to wave propagation simulation, use impulses (Dirac deltas) Only run simulation when an h(i,j,t) needs to be recomputed due to movement or change in space properties.

19 Exploiting a Remote Supercomputer or the Grid

20 Interactivity in the Forward Problem Auralization Algorithms Listener positions Listener signals Listener wearing headphones sound source positions room geometry/properties sound source signals

21 Full Example of Virtualized Audio Human SpaceMicrophones Recovery Algorithms microphone signals microphone positions “Reverse Problem” sound source positions room geometry and properties sound source signals other inputs Human SpaceMicrophones Recovery Algorithms microphone signals microphone positions “Reverse Problem” sound source positions room geometry and properties sound source signals other inputs Human SpaceMicrophones Recovery Algorithms microphone signals microphone positions “Reverse Problem” sound source positions room geometry and properties sound source signals other inputs Combine Auralization Algorithms room geometry/properties sound source signals sound source positions

22 VA as a Distributed Interactive Application Disparate resource requirements –Low latency audio input/output –Massive computation requirements Low latency control loop with human in the loop Response time must be bounded Adaptation mechanisms –Choice between full simulation and LTI simplification number of listeners –Frequency limiting versus delay –Truncation of impulse responses –Spatial resolution of impulse response functions

23 Conclusion We can and should do better than the current state of audio Lots of existing research to exploit –The basis of virtualized audio Trade off computation and specialized hardware VA is a distributed interactive application VA forward problem currently being implemented at Northwestern