Improved Headphone Sound Stage Background to ENCM515 Labs on DSO Audio Channel Modeling Based on concepts in Bessinger Thesis Various copyrights M. Smith, E. Bessinger and L.E.Turner M. Smith, Electrical and Computer Engineering, University of Calgary, Canada ucalgary.ca This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation In Slide Show, click on the right mouse button Select “Meeting Minder” Select the “Action Items” tab Type in action items as they come up Click OK to dismiss this box This will automatically create an Action Item slide at the end of your presentation with your points entered.

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 2 / 33 To be tackled today Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab Implementation in “C” (with and without pointers) Lab Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Sound colouration through FIR filters Lab In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling Lab Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project?

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 3 / 33 Main reference material E. C. Bessinger, Localization of Sound Using Headphones M. Sc. thesis, Electrical and Computer Engineering, January Supervisor Dr. Turner Translated for DSP processors by M. Smith

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 4 / 33 Improved Head Sound Stage Actual Mono Sound Source Perceived Repositioned Sound Source

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 5 / 33 To be tackled today Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab Implementation in “C” (with and without pointers) Lab Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Sound colouration through FIR filters Lab In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling Lab Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project?

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 6 / 33 Concept behind Labs. OFF-LINE initial development using VisualDSP++ simulator Sound Source -- fixed input data array Process Sound -- KNOWN ALGORITHM Sound Sink -- fixed output data array Testing -- plotting and comparing to theory Possible algorithmic time constraints investigated ON-LINE (REAL-TIME) testing using SHARC board Sound Source -- background task (or CODEC) Process Sound -- SAME KNOWN ALGORITHM Sound Sink -- CODEC background task Testing -- using aurally using GOOD earphones and quantified using known sound source

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 7 / 33 Labs. 2 and 3 -- delay line -- Concept No relative delay modelled into the audio channel -- then sound perceived in centre of head Modelling a relative delay into the right ear audio channel. Sound arrival will shift sound to left as “sound” seems to get to left ear first DELAY 0 DELAY_LEFT DELAY_RIGHT

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 8 / 33 Delay Necessary -- Calculation Distance to Sound Sound Time Delay = Speed of Sound Algorithm Delay = Time Delay * Sampling Rate Relative Distance = 0.3 m Speed of sound = 300 m/s Sampling Rate = 44 kHz Delay needed in array = ???? Sound resolution = ???? Reasonable or NOT?

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 9 / 33 GLOSSARY FIFO First In -- First Out stack Delay Line-- Implementation using FIFO LIFOLast In -- First Out stack C-stack-- Implementation using LIFO FIR Finite duration Impulse Response IIRInfinite duration Impulse Response FT (CFT)continuous Fourier transform DFTdiscrete Fourier transform FFT-- fast Fourier transform which is an implementation of DFT

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 10 / 33 To be tackled today Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab Implementation in “C” (with and without pointers) Lab Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Sound colouration through FIR filters Lab In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling Lab Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project?

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 11 / 33 Real-time N-pt Delay code concept void ProcessSound(int channel_one, int channel_two, int *left_channel, int *right_channel) { if ((sound_source & LEFT_DELAY) == LEFT_DELAY) { if ((sound_source & MOVEDELAY) == MOVEDELAY) MemoryMove_LeftDelay(process_var1, &channel_one); void MemoryMove_LeftDelay(int N, int *channel_one) { float left_delayline[MAXDELAY + 1] = {0}; // Insert new value into the back of the FIFO delay line left_delayline[0 + N] = (float) *channel_one; // N delays implimented // Grab delayed value from the front of the FIFO delay line *channel_one = (int) left_delayline[0]; // Update the FIFO delay line using inefficient mass memory moves for (count = 0; count < N; count++)<--- Or N + 1 or N - 1? left_delayline[count] = left_delayline[count - 1]; } WHAT’S WRONG AND MISSING? WHAT’S WRONG

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 12 / 33 Basic Algorithm -- Handle Delay Requires 2 * (LD + RD - 2) memory operations Sample into audio channel Sample from channel

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 13 / 33 To be tackled today Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab Implementation in “C” (with and without pointers) Lab Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Sound colouration through FIR filters Lab In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling Lab Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project?

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 14 / 33 Labs delay line -- Concept Get “ambience” by taking into account constructive and destructive interference around the face. This implies knowing characteristics of “audio channel” and modelling using an FIR filter -- 2 FIR filters per speaker -- processing requirement increasing FILTER 330 LEFT FILTER 30 LEFT FILTER 30 RIGHT FILTER 330 RIGHT

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 15 / 33 Delay is not only thing to take into account Four audio channels from 2 sources in front

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 16 / 33 Lab Transfer function h30(f)

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 17 / 33 Lab FIR transfer -- h330(f)

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 18 / 33 Why different Audio Channels? Delay is not only positioning element -- take into account actual direction Transfer function from loudspeaker to ear modelled by FIR filterS Enough time to perform calculation between samples?

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 19 / 33 FIR schematic diagram

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 20 / 33 FIR = Complex Delay Line

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 21 / 33 Real-time FIR Filter void ProcessSound(int channel_one, int channel_two, int *left_channel, int *right_channel{ if ((sound_source & FIRFilter) FIRFilter(&channel_one, &channel2); float fircoeffs_30[], fircoeffs[330]; void FIRFilter(int *channel_one, int *channel_two) { // Insert new value into FIFO delay line left_delayline[0 + N] = (float) *channel_one; right_delayline[0 + N] = (float) *channel_two; channel_one_30 = channel_one 330 = 0; Need equivalent of following loop for EACH sound source for (count = 0, count < FIRlength - 1, count++) { channel_one_30 = channel_one_30 + arrayleft[count] * fir_30(count); channel_one_330 = channel_one_330 + arrayright[count]* fir_330[count]; } *channel_one = (int) channel_one_30 + scale_factor * channel_one_30 ditto 2 Update Left Channel delay line; Update Right Channel Delay line }

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 22 / 33 To be tackled today Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab Implementation in “C” (with and without pointers) Lab Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Sound colouration through FIR filters Lab In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling Lab Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project – Try on other processor architectures – try with video?

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 23 / 33 Lab Room Effects Need to have multiple sources more FIR operations Need to have reflections off the wall -- possibilities IIR filters, Comb, FFT (frequency domain)

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 24 / 33 Standard IIR filter example -- biquad Normally in multi-stages Make parallel BETWEEN rather than IN stages -- see Smith (Micro Magazine article, 1992) SMALL DELAYS

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 25 / 33 Real Time standard IIR code finalleftear = h0 * lefttemp1 + h1 * lefttemp2 + h3 * lefttemp3; finalrightear = h0 * rightemp1 + h1 * righttemp2 + h3 * righttemp3; newlefttemp1 = gain * leftear + h4 * lefttemp2 + h5 * lefttemp3; newrighttemp1 = gain * rightear + h4 * rightemp2 + h5 * righttemp3; lefttemp3 = lefttemp2;// Parallel ear calculations lefttemp2 = lefttemp1;// Parallel stages lefttemp1 = newlefttemp1; righttemp3 = righttemp2; // Optimization -- final exam question? righttemp2 = righttemp1;// Optimization righttemp1 = newrighttemp1; // Need other delay and FIR filters for audio channel characteristics

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 26 / 33 Lab IIR -- comb filter -- different Long delay line

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 27 / 33 Full Lab 5 -- FIR and IIR Time available if multi-task?

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 28 / 33 Lab. 5 requires 7 comb filters BIG DELAYS (Use PACK instructions or decimation?)

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 29 / 33 Lab. 5 Possibilities -- FFT? Implement filters in frequency domain rather than in time domain Frequency domain algorithms can be considerably faster than time domain algorithms especially when performing several filtering operations on one set of data Require more memory to process BLOCKS OF DATA

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 30 / 33 Real Time FIR Convolution Filter via multiplication in frequency domain FFT is faster that FIR and IIR if memory available (2N log N + 4N) rather than 3 * N * N operations for 512 points // Signal Conditioning -- Windowing wleftear[n] = leftear[n] * window[n]; FFT(wleftear, leftfrequency, forward FFT) for 512 points // Signal convolution with FIR frequency domain response new_leftfrequency1[n] = leftfrequency[n] * filter1[n] + leftfrequency[n] * filter2[n] + leftfrequency[n] * filter3[n]; FFT(new_leftfrequency, newleftear, inverse FFT)

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 31 / 33 DSP 4 -- FFT -- Decimation Require special architecture for speed

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 32 / 33 Major FFT implementation Reference Major reference on signal conditional and filter implementation in frequency domain FFT -- friscy Fourier Transforms? Smith (Microprocessors and Microsystems)

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 33 / 33 DIF -- FFT code -- in Fortran pseudo code DO 11 I = 1, allows FFT size up to 2^15 M = I; N2 = 2 **I; IF (N.EQ.N2) go to CONTINUE RETURN 21N2 = N DO 30 I = 1, M N1 = N2; N2 = N2/2; I1 = 1; I2 = N/N1; -- indexes DO 30 J = 1, N2 --- Note that N2 gets smaller C= cosarray(I1); S = sinarray(I1); I1 = I1 + I2-- Note indexing format DO 30 K = J, N step N1 L = K + N2 tempr = xreal(K) - xreal(L); xreal(K) = xreal(K) + xreal(L) tempi = ximag(K) - ximag(L); ximag(K) = ximag(K) + ximag(L) xreal(L) = C * tempr + S * tempi; ximag(L) = C * tempi + S * tempr; 30 CONTINUE BitReverseArrayManipulation(N, xreal, ximag)

6/3/2015 ENCM Background to Audio Channel Modelling Copyright 34 / 33 To be tackled today Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab Implementation in “C” (with and without pointers) Lab Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Sound colouration through FIR filters Lab In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling Lab Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project – Try on other processor architectures – try with video?