Noise reduction and binaural cue preservation of multi- microphone algorithms Simon Doclo, Tim van den Bogaert, Marc Moonen, Jan Wouters Dept. of Electrical Engineering (ESAT-SCD), KU Leuven, Belgium Dept. of Neurosciences (ExpORL), KU Leuven, Belgium Oldenburg, June

2 Overview Problem statement oImprove speech intelligibility + preserve spatial awareness oBilateral vs. binaural processing Binaural signal processing using multi-channel Wiener filter oMWF: noise reduction and preservation of speech cues, noise cues are distorted oExtension of MWF to preserve binaural cues of all components: – MWFv: partial estimation of noise component – MWF-ITF: extension with Interaural Transfer Function oPhysical and perceptual evaluation Reduce bandwidth requirements of wireless link oDistributed binaural MWF

3 Problem statement Many hearing impaired are fitted with a hearing aid at both ears oSignal processing to selectively enhance useful speech signal and improve speech intelligibility oSignal processing to preserve directional hearing and spatial awareness oMultiple microphone available: spectral + spatial processing Binaural auditory cues: oInteraural Time Difference (ITD) – Interaural Level Difference (ILD) oBinaural cues, in addition to spectral and temporal cues, play an important role in binaural noise reduction and sound localisation oITD: f 2000Hz IPD/ITD ILD  Problem statement -bilateral/binaural  Binaural processing  Bandwidth reduction  Conclusions

4 Bilateral vs. Binaural Bilateral system Independent left/right processing: Preservation of binaural cues for localisation ? Binaural system More microphones:  better performance ?  preservation of binaural cues ? Need of binaural link  Problem statement -bilateral/binaural  Binaural processing  Bandwidth reduction  Conclusions

5 Bilateral system: oIndependent processing of left and right hearing aid oLocalisation cues are distorted RMS error per loudspeaker when accumulating all responses of the different test conditions (NH = normal hearing, NO = hearing impaired without hearing aids, O = omnidirectional configuration, A = adaptive directional configuration) [Van den Bogaert et al., 2006] Bilateral vs. Binaural  Problem statement -bilateral/binaural  Binaural processing  Bandwidth reduction  Conclusions  also effect on intelligibility through binaural hearing advantage

6 Bilateral system: oIndependent processing of left and right hearing aid oLocalisation cues are distorted [Van den Bogaert et al., 2006] [Bronkhorst and Plomp, 1988][Beutelmann and Brand, 2006]  also effect on intelligibility through binaural hearing advantage Bilateral vs. Binaural  Problem statement -bilateral/binaural  Binaural processing  Bandwidth reduction  Conclusions

7 Bilateral system: oIndependent processing of left and right hearing aid oLocalisation cues are distorted Binaural system: oCooperation between left and right hearing aid (e.g. wireless link) oAssumption: all microphone signals are available at the same time Objectives/requirements for binaural algorithm: 1.SNR improvement: noise reduction, limit speech distortion 2.Preservation of binaural cues (speech/noise) to exploit binaural hearing advantage 3.No assumption about position of speech source and microphones [Van den Bogaert et al., 2006] Bilateral vs. Binaural  Problem statement -bilateral/binaural  Binaural processing  Bandwidth reduction  Conclusions

8 Binaural noise reduction Fixed beamforming oLow complexity oLimited performance, only speech cues may be preserved (in ideal situations) [Desloge 1997, Lotter 2004] CASA-based techniques oComputation and application of (real-valued) binaural mask based on binaural and temporal/spectral cues oTypically good preservation of binaural speech and noise cues oMostly for 2 microphones, “spectral subtraction”-like artefacts [Kollmeier, Peissig, Wittkop, Dong, Haykin] Adaptive beamforming oBased on GSC-structure + passing low-pass portion unaltered to preserve ITD cues oPreserves part of the binaural cues oSubstantial drop in noise reduction performance [Welker, 1997]  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

9 Binaural noise reduction Binaural multi-channel Wiener filter oMMSE estimate of speech component in microphone signal at both ears [Doclo, Klasen, Wouters, Moonen] speech cues are preserved, no a-priori assumptions about position of speech source and microphones noise cues may be distorted Extension of MWF : preservation of binaural speech and noise cues without substantially compromising noise reduction performance  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

10 Configuration and signals Configuration: microphone array with M microphones at left and right hearing aid, communication between hearing aids noise component speech component Use all microphone signals to compute output signal at both ears  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

11 Overview of cost functions Multi-channel Wiener filter (MWF): MMSE estimate of speech component in microphone signal at both ears trade-off noise reduction and speech distortion Speech-distortion weighted multi-channel Wiener filter (SDW-MWF) [Doclo 2002, Spriet 2004] binaural cue preservation of speech + noise Partial estimation of noise component (MWFv) [Klasen 2005] Extension with ITD-ILD or Interaural Transfer Function (ITF) [Doclo 2005, Klasen 2006, Van den Bogaert 2007]  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

12 Binaural SDW-MWF: estimate of speech component in microphone signal at both ears (usually front microphone) + trade-off between noise reduction and speech distortion Binaural multi-channel Wiener filter speech component in front microphones noise reductionspeech distortion  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions estimate oDepends on second-order statistics of speech and noise oEstimate R y during speech-dominated time-frequency segments, estimate R v during noise-dominated segments, requiring robust voice activity detection (VAD) mechanism oNo assumptions about positions of microphones and sources oAdaptive (LMS-based) algorithm available [Spriet 2004, Doclo 2007]

13 Binaural multi-channel Wiener filter Interpretation for single speech source: oSpectral and spatial filtering operation with  (spatial) coherence matrix and P (spectral) power oEquivalent to superdirective beamformer (diffuse noise field) or delay-and-sum beamformer (spatially white noise field) + single-channel WF-based postfilter (spectral subraction) Spatial separation between speech and noise sources SNR Binaural cues (ITD-ILD) : Perfectly preserves binaural cues of speech component Binaural cues of noise component  speech component !!  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

14 Partial estimation of noise component oEstimate of sum of speech component and scaled noise component oRelationship with SDW-MWF: mix with reference microphone signals reduction of noise reduction performance works for multiple noise sources Partial noise estimation (MWFv)  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

15 Interaural Wiener filter (MWF-ITF) Extension of SDW-MWF with binaural cues oAdd term related to binaural cues of noise (and speech) component oPossible cues: ITD, ILD, Interaural Transfer Function (ITF) e.g. ITF preservation speechITF preservation noise oClosed form expression! olarge  changes direction of speech  increase weight  oImplicit assumption of single noise source  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

16 Overview of batch algorithm Left input signals Right input signals FFT Left outputRight output IFFT Frequency-domain filtering Off-line computation of statistics VAD Calculate binaural input cues and filter  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

17 Simulation setup Identification of HRTFs: oBinaural recordings on CORTEX MK2 artificial head o2 omni-directional microphones on each hearing aid (d=1cm) oLS = -90  :15  :90 , 90  :30  :270 , 1m from head oRoom reverberation: T 60 =140 ms (and T 60 =510 ms)  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

18 Experimental results Simulations: oS x N y, SNR = 0 dB on left front microphone (broadband) of s = kHz MWF algorithmic parameters: obatch procedure, perfect VAD oL=96,  =5 oMWFv for different , MWF-ITF for different ,  Physical evaluation: oSpeech = HINT, noise = babble noise oSpeech intelligibility:  SNR oLocalisation:  ITD /  ILD Perceptual evaluation: oPreliminary study with NH subjects oSpeech intelligibility: SRT oLocalisation: localise S and N  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

19 Physical evaluation Performance measures: oIntelligibility weighted SNR improvement (left/right) oILD error (speech/noise component)  power ratio oITD error (speech/noise component)  phase of cross-correlation importance of i-th frequency for speech intelligibility low-pass filter 1500 Hz  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

Physical evaluation: MWFv S 0 N 60

Physical evaluation: MWF-ITF S 0 N 60 SNR improvement only slightly degraded

Physical evaluation: MWF-ITF S 0 N 60 Compromise possible between speech and noise localisation error

23 Procedure: oheadphone experiments, using measured HRTFs oFilters are calculated off-line on VU speech-weighted noise as S and multitalker babble noise as N oAll stimuli presented at comfort level, 5 NH subjects (ongoing) Perceptual evaluation Headphones HRTF x HRTF v speech noise G Binaural filter  MicL R  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions Speech intelligibility: oAdaptive procedure to find 50% Speech Reception Threshold (SRT) Localisation: oS and N components (telephone) are sent separately through filter oLocalise S and N in room where HRTFs were measured oLevel roving 6 dB, 3 repetitions per condition for each subject

24 Perceptual evaluation: MWFv Algorithms: unprocessed, state-of-the art bilateral, MWF, MWFv (  =0.2) Conditions: S 0 N 60, S 45 N 315 and S 90 N 270 (T 60 =510 ms)  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

25 Perceptual evaluation: MWFv With state-of-the-art systems: preservation of binaural cues only within central angle of frontal hemisphere. Binaural MWF: opreserves localization cues for speech source opreserves localization cues for noise source(s) with small mixing  oRecent SRT experiments (N=2) show no substantial SRT difference between  =0 and  =0.2 Ongoing research: oPerceptual evaluation (SRT and localisation) for MWF-ITF  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions

26 Perceptual evaluation: MWF-ITF Condition: oS 0 N 60, reference condition = no processing oDutch sentences, stationary speech weighted noise, T 60 =140 ms Results: oaverage SRT without processing = -9.2 dB oSRT improvements in the range dB  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions  =0,  =1 Binaural speech intelligibility advantage due to spatial separation does not seem to compensate for loss in SNR improvement

27 Condition S x N 0 : Speech arrives from angle x, with x from -90° to +90° in steps of 30°, noise arrives from 0° Sum of localisation errors S x and N 0 Perceptual evaluation: MWF-ITF  Problem statement  Binaural processing -MWF -Cue preservation -Physical evaluation -Perceptual eval  Bandwidth reduction  Conclusions Parameters can be tuned to achieve better overall localization performance at the cost of some noise reduction Good correlation between physical and perceptual evaluation

28 Bandwidth constraints Binaural MWF: o2M microphone signals are transmitted over wireless link Reduce bandwidth requirement of wireless link: oTransmit one signal from contralateral ear  Problem statement  Binaural processing  Bandwidth reduction  Conclusions – Front contralateral microphone signal – Output of contralateral fixed (e.g. superdirective) beamformer – Output of MWF using only M contralateral microphone signals – Iterative distributed binaural MWF scheme

29 Physical evaluation  Problem statement  Binaural processing  Bandwidth reduction  Conclusions Performance of dB-MWF close to full binaural MWF !

30 Contralateral directivity pattern T 60 =140 ms S 0 N 120 Left HA  SNR=14.6dB B-MWFMWF-front  SNR=10.5dB MWF-contra  SNR=14.2dB dB-MWF  SNR=14.2dB

31 Conclusions State-of-the art signal processing in (bilateral) HAs: preservation of binaural cues only within central angle of frontal hemisphere Binaural MWF: oSubstantial noise reduction (MWF 4  3 > 2) oPreservation of binaural speech cues oDistortion of binaural noise cues oNo assumptions about positions and microphones  VAD Compromise between noise reduction and binaural cue preservation can be achieved with extensions of MWF oMixing with microphone signals oInteraural Transfer Function Reduction of bandwidth using distributed MWF  Problem statement  Binaural processing  Bandwidth reduction  Conclusions

32 S. Doclo, M. Moonen, “GSVD-based optimal filtering for single and multi-microphone speech enhancement,” IEEE Trans. Signal Processing, vol. 50, no. 9, pp. 2230–2244, Sept A. Spriet, M. Moonen, J. Wouters, “Spatially pre-processed speech distortion weighted multi-channel Wiener filtering for noise reduction,” Signal Processing, vol. 84, no. 12, pp. 2367–2387, Dec S. Doclo, A. Spriet, J. Wouters and M. Moonen, “Frequency-Domain Criterion for the Speech Distortion Weighted Multichannel Wiener Filter for Robust Noise Reduction,” Speech Communication, special issue on Speech Enhancement, in press, T. Van den Bogaert, T.J. Klasen, M. Moonen, L. Van Deun, J. Wouters, “Horizontal localization with bilateral hearing aids: Without is better than with”, J. Acoust. Soc. Am., vol. 119, no. 1, pp , Jan T.J. Klasen, T. Van den Bogaert, M. Moonen, J. Wouters, “Binaural noise reduction algorithms for hearing aids that preserve interaural time delay cues,” IEEE Trans. Signal Processing, vol. 55, no. 4, Apr. 2007, pp S. Doclo, R. Dong, T.J. Klasen, J. Wouters, S. Haykin, M. Moonen, “Extension of the multi-channel Wiener filter with localisation cues for noise reduction in binaural hearing aids,” in Proc. IWAENC, Eindhoven, The Netherlands, Sep. 2005, pp T.J. Klasen, S. Doclo, T. van den Bogaert, M. Moonen and J. Wouters, “Binaural multi-channel Wiener filtering for hearing aids: preserving interaural time and level differences”, in Proc. ICASSP, Toulouse, France, May 2006, pp. V T. van den Bogaert, S. Doclo, J. Wouters, M. Moonen, “Binaural cue preservation for hearing aids using an interaural transfer function multichannel Wiener filter,” in Proc. ICASSP, Honolulu HI, USA, Apr. 2007, pp Online papers available at and