EE Dept., IIT Bombay NCC 2015, Mumbai, 27 Feb.- 1 Mar. 2015, Paper No (28 th Feb., Sat., Session SI, 10:05 – 11:15, Paper I) ============================================================================ Speech Enhancement Using Noise Estimation Based on Dynamic Quantile Tracking for Hearing Impaired Listeners Nitya Tiwari & Prem C. Pandey {nitya, ee.iitb.ac.in IIT Bombay

EE Dept., IIT Bombay 2/25 Overview 1.Introduction 2.Signal Processing for Speech Enhancement 3.Implementation for Real-time Processing 4.Test Results 5.Summary & Conclusion

EE Dept., IIT Bombay /25 1. Introduction Conductive loss Causes: abnormalities in the middle ear Characteristics: frequency-dependent elevation of hearing thresholds without change in the dynamic range of hearing, or change in temporal & spectral resolution Sensorineural hearing loss Causes: abnormalities in the cochlear hair cells or the auditory nerve Characteristics ◦ Increase in hearing thresholds (due to loss of inner hair cells) ◦ Loudness recruitment (abnormal loudness growth) & decrease in dynamic range (due to loss of outer hair cells) ◦ Increased spectral & temporal masking >> Poor intelligibility and degraded speech perception, particularly in noisy environments

EE Dept., IIT Bombay /25 Signal processing techniques currently used in hearing aids Frequency-selective amplification to compensate for frequency dependent elevation of hearing thresholds Amplitude compression to compensate for decreased dynamic range Processing for reducing the effects of increased spectral & temporal masking in sensorineural loss Binaural dichotic presentation (Lunner et al. 1993, Kulkarni et al. 2012) Spectral contrast enhancement (Yang et al. 2003) Multiband frequency compression (Arai et al. 2004, Kulkarni et al. 2012) Consonant-vowel ratio enhancement (Jayan et al. 2014)

EE Dept., IIT Bombay /25 Suppression of background noise for speech enhancement For improving speech quality and intelligibility As a pre-processing stage for signal processing techniques to improve speech perception by hearing impaired listeners Single-input speech enhancement using spectral subtraction (Boll 1979, Berouti et al.1979, Martin 1994, Loizou 2007) Short-time spectral analysis Dynamic estimation of stationary or non-stationary noise spectrum using voice activity detection during non-speech segments, or continuously using statistical techniques Estimation of noise-free speech spectrum using spectral noise subtraction, or multiplication by noise suppression function Speech resynthesis using enhanced magnitude and noisy phase

EE Dept., IIT Bombay /25 Noise spectrum estimation Dynamic estimation of stationary & non-stationary noise spectrum. Under-estimation results in residual noise & over-estimation results in distortion leading to degraded quality with reduced intelligibility. Techniques for noise spectrum estimation Noise estimation during silence intervals using voice activity detector: Detection may not be satisfactory under low-SNR conditions & the method may not correctly track the noise spectrum during long speech segments. Minimum statistics based estimation of noise spectrum: Needs estimation of SNR-dependent subtraction factor, may underestimate noise & remove parts of speech signal during weaker segments. Quantile-based noise estimation (QBNE): Based on observation that signal energy in a particular frequency bin is low in most of the frames & high only in 10-20% frames corresponding to voiced speech segments. Reported to be useable without voice activity detection & SNR estimation.

EE Dept., IIT Bombay /25 Spectral subtraction using QBNE techniques Quantiles obtained by ordering the spectral samples or from dynamically generated histograms ◦Stahl et al. (2000): 0.55-quantile used to estimate the noise spectrum. 26% decrease in word error in speech recognition. ◦Evans & Mason (2002): Time-frequency quantile based noise estimation. 35% improvement in word recognition accuracy. ◦Bai & Wan (2003): A two-pass quantile based noise estimation. SNR estimation for each time-frequency point using a fixed quantile & using it for a new quantile for each sub-band. Not suited for use in hearing aids due to large memory space required for storing the spectral samples & high computational complexity. Cascaded-median based noise estimation (Waddi et al., 2013) ◦Median (0.5-quantile) to reduce computation requirement. ◦Cascaded-median as approximation to median for real-time implementation. Different improvements for different noises indicated need for using frequency dependent quantiles for noise estimation.

EE Dept., IIT Bombay /25 Research objective Technique for noise spectrum estimation for speech enhancement using spectral subtraction & its real-time implementation for possible use in hearing aids. Proposed technique Dynamic quantile tracking as an approximation to the sample quantile obtained by sorting without use of voice activity detector, without involving storage and sorting of past samples, low computational complexity & memory requirement, low signal delay (algorithmic + computational).

EE Dept., IIT Bombay /25 2. Signal Processing for Speech Enhancement Dynamic quantile tracking for noise estimation Speech enhancement using spectral subtraction Implementation for offline processing Dynamic quantile tracking for noise estimation Quantile estimation Range estimation Quantile tracking Quantile estimation The noise spectral sample at each frame estimated as a quantile of the past input spectral samples by applying an increment or a decrement on the previous estimate where D n (k) = k th spectral sample of estimated noise magnitude spectrum for analysis frame positioned at sample n ; S = Number of samples for shift between successive analysis frames; |X n (k)| = Input magnitude spectral sample; d n (k) = Change in D n (k).

EE Dept., IIT Bombay /25 Δ + (k) & Δ − (k) selected to be fractions of the range such that D n (k) approaches sample quantile & sum of changes approaches zero: where p(k) = p th quantile of the magnitude spectrum; M = Number of frames; R = Range (difference between max. & min. of the spectral samples); λ = Factor controlling the step size. → → Ripple in estimation → & No. of steps between initial estimate D i (k) & final estimate D f (k) (|D f (k) − D i (k)|) max = R → Convergence factor λ to be selected for tradeoff between δ & s max. Method not suited for very low or high p. & →

EE Dept., IIT Bombay /25 Range estimation Range updated dynamically as the underlying distribution is unknown. Peak & valley detection using first-order recursive relations. where τ = factor controlling rise time ; σ : factor controlling fall time. Small τ for fast detector response & relatively large σ to avoid ripples. R n (k) = P n (k) − V n (k) Quantile tracking of noise spectral sample

EE Dept., IIT Bombay /25 Block diagram of dynamic quantile tracking

EE Dept., IIT Bombay /25 Speech enhancement using spectral subtraction Short-time spectral analysis using FFT Dynamic estimation of non-stationary noise spectrum Enhanced magnitude spectrum calculation by generalized spectral subtraction Enhanced complex spectrum calculation without explicit phase estimation Re-synthesis using IFFT with overlap- add

EE Dept., IIT Bombay /25 Generalized spectral subtraction |Y n (k)| = β 1/γ D n (k), if |X n (k)| < (α + β) 1/γ D n (k) [ |X n (k)| γ – α(D n (k)) γ ] 1/γ otherwise where γ = exponent factor ( 2: power subtraction, 1: magnitude subtraction), α = o ver-subtraction factor (for limiting effect of short-term variations in noise spectrum), β = floor factor to mask the musical noise due to over-subtraction, X n (k) = Windowed speech spectrum, D n (k) = Estimated noise mag. spectrum. Enhanced complex spectrum calculation with phase of input spectrum without explicit phase calculation Y n (k) = |Y n (k)| X n (k) / |X n (k)|

EE Dept., IIT Bombay /25 Implementation for offline processing Implementation using Matlab 7.10 for evaluating the performance of the proposed technique and the effect of processing parameters. Processing parameters ◦ f s = 10 kHz ◦ Frame length = 25.6 ms ( L = 256 ) ◦ Overlap = 75% ( S = 64 ) ◦ FFT size N = 512 Implementation using γ = 1 (magnitude subtraction) showed higher tolerance to variation in α, β values Offline processing to get optimal combination of α and β for spectral subtraction Offline processing to get optimal combination of τ and σ for noise estimation

EE Dept., IIT Bombay /25 3. Implementation for Real-time Processing DSP board with 16-bit fixed-point processor with on-chip FFT hardware DSP chip: TI/TMS320C5515 ◦16 MB memory space ( 320 KB on-chip RAM with 64 KB dual access data memory) ◦ Three 32 -bit programmable timers ◦4 DMA controllers each with 4 channels ◦ FFT hardware accelerator ( up to point FFT) ◦ Max. clock speed: 120 MHz DSP Board: eZdsp ◦ 4 MB on-board NOR flash for user program ◦ Stereo codec TLV320AIC3204: 16/20/24/32-bit ADC & DAC, 8 – 192 kHz sampling Software development: C using TI's 'CCStudio ver. 4. 0

EE Dept., IIT Bombay /25 Implementation DMA based I/O with cyclic buffers ADC and DAC with 16 -bit quantization (left channel of stereo codec ) f s = 10 kHz, L = 256, S = 64, N = 512 (as for offline processing) Data representation (input, spectral values, output) : 16 -bit real & 16 -bit imaginary

EE Dept., IIT Bombay /25 Data transfer & buffering operations ( S = L/4 ) DMA cyclic buffers 5 -block S - sample input buffer 2 -block S - sample output buffer Pointers Current input block Just-filled input block Current output block Write-to output block (incremented cyclically on DMA interrupt) Delay Algorithmic: 1 frame (25.6 ms) Computational: ≤ frame shift ( 6.4 ms)

EE Dept., IIT Bombay /25 4. Test Results Test material Speech: Recording with three isolated vowels, a Hindi sentence, an English sentence (-/a/-/i/-/u/– “aayiye aap kaa naam kyaa hai?” – “Where were you a year ago?”) from a male speaker. Noise: white, street, babble, car, and train noises (AURORA ). SNR: ∞, 15, 12, 9, 6, 3, 0, –3, –6, –9, and –12 dB. Evaluation methods Informal listening Objective evaluation using PESQ measure (0 – 4.5)

EE Dept., IIT Bombay /25 Results: Offline processing Investigations for most suitable values of processing parameters Processing with noise estimation carried out using sample quantile (SQ) values & the following processing parameters: β = 0, α = 0.4 – 6 τ = 0.1, σ = (0.9) 1/1024 (rise time = 1 frame shift, fall time = 1024 frame shift) p = 0.1, 0.25, 0.5, 0.75, 0.9 M = 32, 64, 128, 256, & 512 M = 128 resulted in highest PESQ scores (for fixed SNR, α, & p ). Noise estimation with p = 0.25 resulted in nearly the best scores for different types of noises at all SNRs PESQ scores obtained for processing with noise estimation using dynamic quantile tracking with λ = 1/256 nearly equal to the PESQ scores obtained using SQ with M = 128.

EE Dept., IIT Bombay /25 Processing examples & PESQ scores PESQ scores of the unprocessed (Unpr.) noisy speech with babble (a non-stationary noise) and processed (Pr.) signals with noise estimation by sample quantile (SQ) with M = 128 and dynamic quantile tracking (DQT) with λ = 1/256. SNR (dB) PESQ Score Unpr. Pr., α=1,β=0Pr., α=2,β=0Pr., α=3, β=0 SQDQTSQDQTSQDQT PESQ scores obtained using 0.25-quantile not sensitive to changes in α Combination of λ = 1/256, p = 0.25, & α = 2 used for more detailed examination of scores

EE Dept., IIT Bombay /25 PESQ score vs SNR: noisy & enhanced speech Increase in scores: 0.24 – 0.46 for white noise, 0.08 – 0.32 for babble noise. SNR advantage: ≈ 6 dB for white noise, ≈ 3 dB for babble noise. Informal listening: β = reduced the musical noise without degrading speech quality.

EE Dept., IIT Bombay /25 Results: Real-time processing Testing of real-time processing using white, babble, car, street, and train noises at different SNRs ◦ Listening: Real-time processed output perceptually similar to the offline processed output ◦ Objective verification: High PESQ scores (> 3.5) for output of real-time processing with output of offline processing as the reference Signal delay: 36 ms Processing capacity required: ≈ 41% (System clock needed for satisfactory processing = 50 MHz, highest system clock = 120 MHz)

EE Dept., IIT Bombay /25 (a) Clean speech (b) Noisy speech (c) Offline processed (d) Real-time processed Example: -/a/-/i/-/u/– “aayiye aap kaa naam kyaa hai?” – “Where were you a year ago?”), white noise, input SNR = 3 dB. More examples :

EE Dept., IIT Bombay /25 5. Summary & Conclusions Proposed technique: Suppression of stationary & non-stationary background noise by estimation of noise spectrum using dynamic quantile tracking without voice activity detection or storage & sorting of past samples. Speech enhancement: SNR advantage (at PESQ score = 2) of 3 – 6 dB for different stationary & non-stationary noises. Implementation for real-time operation using 16-bit fixed-point processor TI/TMS320C5515: signal delay ≈36 ms, processing capacity required ≈41%. Technique permits use of frequency-dependent quantile for noise estimation without introducing processing overheads. Further work – Combination of noise suppression with other processing techniques in sensory aids – Implementation using other processors

EE Dept., IIT Bombay

EE Dept., IIT Bombay References [1]H. Levitt, J. M. Pickett, and R. A. Houde (eds.), Senosry Aids for the Hearing Impaired. New York: IEEE Press, [2]J. M. Pickett, The Acoustics of Speech Communication: Fundamentals, Speech Perception Theory, and Technology. Boston, Mass.: Allyn Bacon, 1999, pp. 289–323. [3]H. Dillon, Hearing Aids. New York: Thieme Medical, [4]T. Lunner, S. Arlinger, and J. Hellgren, “8-channel digital filter bank for hearing aid use: preliminary results in monaural, diotic, and dichotic modes,” Scand. Audiol. Suppl., vol. 38, pp. 75–81, [5]P. N. Kulkarni, P. C. Pandey, and D. S. Jangamashetti, “Binaural dichotic presentation to reduce the effects of spectral masking in moderate bilateral sensorineural hearing loss,” Int. J. Audiol., vol. 51, no. 4, pp. 334–344, [6]J. Yang, F. Luo, and A. Nehorai, “Spectral contrast enhancement: Algorithms and comparisons,” Speech Commun., vol. 39, no. 1–2, pp. 33–46, [7]T. Arai, K. Yasu, and N. Hodoshima, “Effective speech processing for various impaired listeners,” in Proc. 18th Int. Cong. Acoust., 2004, Kyoto, Japan, pp. 1389–1392. [8]P. N. Kulkarni, P. C. Pandey, and D. S. Jangamashetti, “Multi-band frequency compression for improving speech perception by listeners with moderate sensorineural hearing loss,” Speech Commun., vol. 54, no. 3 pp. 341–350, [9]A. R. Jayan and P. C. Pandey, “Automated modification of consonant-vowel ratio of stops for improving speech intelligibility,” Int. J. Speech Technol., 2014, [online] DOI /s [10]M. Berouti, R. Schwartz, and J. Makhoul, “Enhancement of speech corrupted by acoustic noise,” in Proc. IEEE ICASSP 1979, Washington, D.C., pp

EE Dept., IIT Bombay [11]S. F. Boll, “Suppression of acoustic noise in speech using spectral subtraction,” IEEE Trans. Acoust., Speech, Signal Process., vol. 27, no. 2, pp , [12]P. C. Loizou, Speech Enhancement: Theory and Practice. New York: CRC, [13]Y. Lu and P. C. Loizou, “A geometric approach to spectral subtraction,” Speech Commun., vol. 50, no. 6, pp , [14]K. Paliwal, K. Wójcicki, and B. Schwerin, “Single-channel speech enhancement using spectral subtraction in the short-time modulation domain,” Speech Commun., vol. 52, no. 5, pp. 450–475, [15]R. Martin, “Spectral subtraction based on minimum statistics,” in Proc. 6th Eur. Signal Process. Conf. (EUSIPCO 1994), Edinburgh, U.K., 1994, pp [16]I. Cohen, “Noise spectrum estimation in adverse environments: improved minima controlled recursive averaging,” IEEE Trans. Speech Audio Process., vol. 11, no. 5, pp , [17]G. Doblinger, “Computationally efficient speech enhancement by spectral minima tracking in subbands,” in Proc. EUROSPEECH 1995, Madrid, Spain, pp [18]V. Stahl, A. Fisher, and R. Bipus, “Quantile based noise estimation for spectral subtraction and Wiener filtering,” in Proc. IEEE ICASSP 2000, Istanbul, Turkey, pp [19]N. W. Evans and J. S. Mason, "Time-frequency quantile-based noise estimation," in Proc. 11th Eur. Signal Process. Conf. (EUSIPCO 2002), Toulouse, France, 2002, pp [20]H. Bai and E. A. Wan, "Two-pass quantile based noise spectrum estimation," Center of spoken language understanding, OGI School of Science and Engineering at OHSU (2003), [online] Available: [21]S. K. Waddi, P. C. Pandey, and N. Tiwari, “Speech enhancement using spectral subtraction and cascaded-median based noise estimation for hearing impaired listeners,” in Proc. 19th Nat. Conf. Commun. (NCC 2013), Delhi, India, 2013, paper no [22]Texas Instruments, Inc., “TMS320C5515 Fixed-Point Digital Signal Processor,” 2011, [online] Available: focus.ti.com/lit/ds/symlink/ tms320c5515.pdf.

EE Dept., IIT Bombay [23]Spectrum Digital, Inc., “TMS320C5515 eZdsp USB Stick Technical Reference,” 2010, [online] Available: support.spectrumdigital.com/ boards/usbstk5515/reva/files/usbstk5515_TechRef_RevA.pdf [24]Texas Instruments, Inc., “TLV320AIC3204 Ultra Low Power Stereo Audio Codec,” 2008, [online] Available: focus.ti.com/lit/ds/ symlink/tlv320aic3204.pdf. [25]ITU, “Perceptual evaluation of speech quality (PESQ): an objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs,” ITU-T Rec., P.862, [26] N. Tiwari, “Speech enhancement using noise estimation based on dynamic quantile tracking for hearing impaired listeners: Processing results”, 2015, [online] Available: /nitya/ncc2015.