1. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay IEEE Workshop on Intelligent Computing, IIIT Allahabad, 13-15 Oct. 2014 Signal processing for improving speech perception by persons with sensorineural hearing loss: Challenges and some solutions P. C. Pandey IIT Bombay

2. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 2/15 Outline A. Speech & Hearing B. Sliding-band Dynamic Range Compression (N. Tiwari & P. C. Pandey, NCC 2014) C. Automated modification of consonant-vowel ratio of stops (A. R. Jayan & P. C Pandey, Int. J. Speech Technology, 2014) Outline A. Speech & Hearing B. Sliding-band Dynamic Range Compression (N. Tiwari & P. C. Pandey, NCC 2014) C. Automated modification of consonant-vowel ratio of stops (A. R. Jayan & P. C Pandey, Int. J. Speech Technology, 2014)

3. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 3/15 Part A Speech & Hearing P. C. Pandey, "Signal processing for improving speech perception by persons with sensorineural hearing loss: Challenges and some solutions", IEEE Workshop on Intelligent Computing, IIIT Allahabad, 13-15 Oct. 2014

4. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 4/15 Speech Production Mechanism Excitation source & filter model Excitation: voiced/unvoiced glottal, frication Filtering: vocal tract filter

5. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 5/15 Speech segments Words Syllables Phonemes Sub-phonemic segments Phonemes: basic speech units Vowels: Pure vowels, Diphthongs Consonants: Semivowels, Stops, Fricatives, Affricates, Nasals /aba/ /apa/ /aga/ /ada/

6. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 6/15 Phonemic features Modes of excitation Glottal Unvoiced (constriction at the glottis), Voiced (glottal vibration) Frication Unvoiced (constriction in vocal tract), Voiced (constriction in v.t., glottal vibration) Movement of articulators Continuant (steady-state v.t. configuration): vowels, nasal stops, fricatives Non-continuant (changing v.t.): diphthongs, semivowels, oral stops (plosives) Place of articulation (place of maximum constriction in v.t.) Bilabial, Labio-dental, Linguo-dental, Alveolar, Palatal, Velar, Gluttoral Changes in voicing frequency (Fo) Supra-segmental features: Intonation, Rhythm

7. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 7/15 Hearing Mechanism Peripheral auditory system External ear: sound collection ○ Pinna ○ Auditory canal Middle ear: impedance matching ○ Ear drum ○ Middle ear bones Inner ear (cochlea): analysis & transduction Auditory nerve: transmission of neural impulses Central auditory system Information processing & interpretation

8. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 8/15 Tonotopic map of cochlea Auditory system

9. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 9/15 Hearing impairment Types of hearing losses Conductive Sensorineural Central Functional Sensorineural hearing loss: abnormalities in the cochlear hair cells or the auditory nerve Aging Excessive exposure to noise Infection Adverse effect of medicines Congenital Hearing impairment Types of hearing losses Conductive Sensorineural Central Functional Sensorineural hearing loss: abnormalities in the cochlear hair cells or the auditory nerve Aging Excessive exposure to noise Infection Adverse effect of medicines Congenital

10. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 10/15 Effects of sensorineural hearing loss Elevated hearing thresholds: inaudibility of low-level sounds Reduced dynamic range & loudness recruitment (abnormal loudness growth): distortion of loudness relationship among speech components Increased temporal masking: poor detection of acoustic landmarks Increased spectral masking (widening of auditory filters): reduced ability to sense spectral shapes >> Poor intelligibility and degraded perception of speech, particularly in noisy environment. Effects of sensorineural hearing loss Elevated hearing thresholds: inaudibility of low-level sounds Reduced dynamic range & loudness recruitment (abnormal loudness growth): distortion of loudness relationship among speech components Increased temporal masking: poor detection of acoustic landmarks Increased spectral masking (widening of auditory filters): reduced ability to sense spectral shapes >> Poor intelligibility and degraded perception of speech, particularly in noisy environment.

11. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 11/15 Signal processing in hearing aids Currently available techniques Frequency selective amplification: improves audibility but not necessarily intelligibility Automatic volume control: not effective in improving intelligibility Multichannel dynamic range compression (with settable attack & release times, compression ratios): effectiveness reduced due to processing artifacts Signal processing in hearing aids Currently available techniques Frequency selective amplification: improves audibility but not necessarily intelligibility Automatic volume control: not effective in improving intelligibility Multichannel dynamic range compression (with settable attack & release times, compression ratios): effectiveness reduced due to processing artifacts

12. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 12/15 Techniques under development Noise suppression Distortion-free dynamic range compression Techniques for reducing the effects of increased spectral masking o Binaural dichotic presentation o Spectral contrast enhancement o Multi-band frequency compression Improvement of consonant-to-vowel ratio (CVR): for reducing the effects of increased temporal masking Techniques under development Noise suppression Distortion-free dynamic range compression Techniques for reducing the effects of increased spectral masking o Binaural dichotic presentation o Spectral contrast enhancement o Multi-band frequency compression Improvement of consonant-to-vowel ratio (CVR): for reducing the effects of increased temporal masking

13. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 13/15 Analog Hearing Aids Pre-amp → AVC → Selectable Freq. Response → Amp. Digital Hearing Aids Pre-amp & AVC → ADC → Multi-band Amplitude Compr. & Freq. Resp. → DAC & Amp. Existing Problems Noisy environment & reverberation Distortions due to multiband amplitude compression Poor speech perception due to increased spectral & temporal masking Visit to audiologist for change of settings Analog Hearing Aids Pre-amp → AVC → Selectable Freq. Response → Amp. Digital Hearing Aids Pre-amp & AVC → ADC → Multi-band Amplitude Compr. & Freq. Resp. → DAC & Amp. Existing Problems Noisy environment & reverberation Distortions due to multiband amplitude compression Poor speech perception due to increased spectral & temporal masking Visit to audiologist for change of settings

14. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 14/15 Proposed Hearing Aids Distortion-free dynamic range compression & adjustable frequency response Noise suppression & de-reverberation Processing for reducing the effects of increased spectral masking Processing for reducing the effects of increased temporal masking User selectable settings Implementation using a low-power DSP chip with acceptable signal delay (< 60 ms) Proposed Hearing Aids Distortion-free dynamic range compression & adjustable frequency response Noise suppression & de-reverberation Processing for reducing the effects of increased spectral masking Processing for reducing the effects of increased temporal masking User selectable settings Implementation using a low-power DSP chip with acceptable signal delay (< 60 ms)

15. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 15/15 Some Solutions for improving speech perception by listeners with moderate-to- severe sensorineural loss Sliding-band dynamic range compression as a solution to the problem posed by loudness recruitment Automated modification of consonant-vowel ratio of stop consonants as a solution to the problem posed by increased intraspeech spectral and temporal masking. Implementation using a 16-bit fixed-point DSP processor & testing for satisfactory operation. Some Solutions for improving speech perception by listeners with moderate-to- severe sensorineural loss Sliding-band dynamic range compression as a solution to the problem posed by loudness recruitment Automated modification of consonant-vowel ratio of stop consonants as a solution to the problem posed by increased intraspeech spectral and temporal masking. Implementation using a 16-bit fixed-point DSP processor & testing for satisfactory operation.

16. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay To be continued to Part B.

17. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay Workshop: IEEE Workshop on Intelligent Computing, Allahabad, October 13-15, 2014, organized jointly by CSIR-CEERI Pilani and IIIT Allahabad. Speaker: Prof. P. C. Pandey, EE Dept, IIT Bombay Topic: Signal processing for improving speech perception by persons with sensorineural hearing loss: Challenges and some solutions Abstract Sensorineural hearing loss is caused by abnormalities in the cochlear hair cells or the auditory nerve. It occurs due to aging, excessive exposure to noise, infection, or abnormalities at the time of birth. It is generally associated with elevated hearing thresholds, reduced dynamic range, and increased temporal and spectral masking, leading to degraded perception of speech, particular in noisy environment. Several signal processing techniques have been investigated and reported to address these problems. However, most of these are not suited for use in hearing aids due to distortions caused by processing related artifacts or due to constraints of size, power, and acceptable signal delay. As some of the possible solutions, two signal processing techniques have been investigated: (i) a sliding-band dynamic range compression as a solution to the problem posed by loudness recruitment, and (ii) automated modification of consonant-vowel ratio of stop consonants as a solution to the problem posed by increased intraspeech spectral and temporal masking. Both techniques have been implemented using a 16-bit fixed-point DSP processor and tested for satisfactory operation. Persons with sensorineural loss generally have a highly reduced dynamic range of hearing, with a significant frequency-dependent elevation of hearing threshold levels without corresponding increase in the upper comfortable listening levels. Signal processing for dynamic range compression is used to present the sounds comfortably within the limited dynamic range of the listener. Analog hearing aids generally use single- band compression with the gain being dependent on the time-varying signal level. As the power is mostly contributed by the low-frequency components, the amplification of the high-frequency components depends on the energy in the low-frequency components. Thus the high frequency components may become inaudible and distortions in temporal envelope may get introduced. In multiband compression available in most digital hearing aids, the spectral components of the input signal are divided in multiple bands and the gain for each band is calculated on the basis of signal power in that band. This type of processing can introduce spurious spectral distortions and use of a large number of bands reduces spectral contrasts and the modulation depth of speech, resulting in an adverse effect on the perception of certain speech cues. Further, the frequency response of a multiband compression system has a time-varying magnitude response without corresponding variation in the phase response, which can cause audible distortions, particularly for non-speech audio. These distortions may partly offset the advantages of dynamic range compression for the hearing-impaired listener. In order to significantly reduce the temporal and spectral distortions associated with the currently used single-band and multiband compressions in hearing aids, a "sliding-band compression" has been developed. It involves calculating a frequency-dependent gain function, in which the gain for each spectral sample is determined by the short-time power in an auditory critical band centered at it. The gain calculation takes into account the specified hearing thresholds, compression ratios, and attack and release times. Unlike single-band compression, it does not result in any significant temporal distortions because the effect of short-time energy of a spectral component on other spectral components is limited to those located within a critical bandwidth. Due to use of sliding critical bands for calculating the power spectrum, formant transitions do not result in discontinuities in the processed output. The technique is realized using an FFT-based analysis-synthesis method which masks phase related discontinuities and can be integrated with other FFT-based signal processing in hearing aids. The technique is implemented and tested for satisfactory real-time operation on a 16-bit fixed-point DSP processor.

18. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay Increasing the level of the consonant segments relative to the nearby vowel segments, known as consonant-vowel ratio (CVR) modification, is reported to be effective in improving speech intelligibility for listeners in noisy backgrounds and for hearing impaired listeners. A technique for real-time CVR modification of stops using the rate of change of spectral centroid for detection of spectral transitions is presented. Its effectiveness in improving the recognition of consonants in the presence of speech spectrum shaped noise is evaluated by conducting listening tests on normal-hearing subjects. At lower values of SNR, there was an increase of 7 - 21% in recognition scores and an equivalent SNR advantage of 3 dB. The technique is implemented on a DSP board based on a 16-bit fixed point processor with on-chip FFT hardware and tested for satisfactory real-time operation. References [1] N. Tiwari and P. C. Pandey, A sliding-band dynamic range compression for use in hearing aids, Proc. National Conference on Communications 2014 (NCC 2014), Kanpur, Feb. 28 - Mar. 2, 2014, paper no. 1569847357. [2] A. R. Jayan & P. C. Pandey, "Automated modification of consonant-vowel ratio of stops for improving speech intelligibility", Int. J. Speech Technology, 2014 (accepted for publication). Dr. Prem C. Pandey Dr. Pandey is a Professor in Electrical Engineering at IIT Bombay. He is currently also the Associate Dean of Academic Programmes. He received B.Tech. in electronics engineering from Banaras Hindu University in 1979, M.Tech. in electrical engineering from IIT Kanpur in 1981, and Ph.D. in electrical & biomedical engineering from the University of Toronto (Canada) in 1987. In 1987, he joined the University of Wyoming (USA) as an assistant professor and later joined IIT Bombay in 1989. His research interests include speech & signal processing; biomedical signal processing & instrumentation; electronic instrumentation & embedded system design. The focus of his R&D efforts has been in the areas of impedance cardiography and aids for persons with speech and hearing impairment.