Speech & Audio Processing - Part–II Digital Audio Signal Processing Marc Moonen Dept. E.E./ESAT-STADIUS, KU Leuven marc.moonen@esat.kuleuven.be homes.esat.kuleuven.be/~moonen/

Speech & Audio Processing Part-I (H. Van hamme) speech recognition speech coding (+audio coding) speech synthesis (TTS) Part-II (M. Moonen): Digital Audio Signal Processing microphone array processing noise cancellation acoustic echo cancellation acoustic feedback- cancellation active noise control 3D audio PS: selection of topics

Digital Audio Signal Processing Aims/scope Case study: Hearing instruments Overview Prerequisites Lectures/course material/literature Exercise sessions/project Exam

Aims/Scope Aim is 2-fold : Speech & audio per se S & A industry in Belgium/Europe/… Basic signal processing theory/principles : Optimal filters Adaptive filter algorithms (APA, Filtered-X LMS,..) Kalman filters (linear/nonlinear) etc...

Case Study: Hearing Instruments 1/14 Outer ear/middle ear/inner ear Tonotopy of inner ear: spatial arrangement of where sounds of different frequency are processed Low-freq tone © www.cm.be = Cochlea Neural activity for low-freq tone High-freq tone Neural activitity for high-freq tone 5

Case Study: Hearing Instruments 2/14 Hearing loss types: conductive sensorineural mixed One in six adults (Europe) …and still increasing Typical causes: aging exposure to loud sounds … [Source: Lapperre]

Case Study: Hearing Instruments 3/14 Hearing impairment : Dynamic range & audibility Normal hearing Hearing impaired subjects subjects Level 100dB 0dB

Case Study: Hearing Instruments 4/14 Hearing impairment : Dynamic range & audibility Dynamic range compression (DRC) (…rather than `amplification’) Level 100dB 100dB Output Level (dB) 0dB 0dB 100dB 0dB Input Level (dB) Design: multiband DRC, attack time, release time, …

Case Study: Hearing Instruments 5/14 Hearing impairment : Audibility vs speech intelligibility Audibility does not imply intelligibility Hearing impaired subjects need 5..10dB larger signal-to-noise ratio (SNR) for speech understanding in noisy environments Need for noise reduction (=speech enhancement) algorithms: State-of-the-art: monaural 2-microphone adaptive noise reduction Near future: binaural noise reduction (see below) Not-so-near future: multi-node noise reduction (see below) SNR 20dB 0dB 30 50 70 90 Hearing loss (dB, 3-freq-average)

Case Study: Hearing Instruments 6/14 Hearing Aids (HAs) Audio input/audio output (`microphone-processing-loudspeaker’) ‘Amplifier’, but so much more than an amplifier!! History: Horns/trumpets/… `Desktop’ HAs (1900) Wearable HAs (1930) Digital HAs (1980) State-of-the-art: MHz’s clock speed Millions of arithmetic operations/sec, … Multiple microphones 1921 2007 (Oticon)

Case Study: Hearing Instruments 7/14 Cochlear Implants (CIs) Audio input/electrode stimulation output Stimulation strategy + preprocessing similar to HAs History: Volta’s experiment… First implants (1960) Commercial CIs (1970-1980) Digital CIs (1980) State-of-the-art: MHz’s clock speed, Mops/sec, … Multiple microphones Other: Bone anchored HAs, middle ear implants, … Alessandro Volta 1745-1827 Intra-cochlear electrode © Cochlear Ltd Electrical stimulation for low frequency Electrical stimulation for high frequency

Case Study: Hearing Instruments 8/14 External Processor Digital/analog-conversion (cfr infra) Digital processing & filterbank Etc.. Coil Inductive/magnetic coupling Implant Electrode array PS: number of CI-implantees worldwide approx. 200.000 PS: 1 CI is approx. 25kEURO, plus surgery, revalidation,.. PS: 3 companies (Cochlear LtD, Med-El, Advanced Bionics) © Cochlear Ltd 12/32 12

Case Study: Hearing Instruments 9/14 HA technology requirements Small form factor (cfr. user acceptance) Low power: 1…5mW (cfr. battery lifetime ≈ 1 week) Low processing delay: 10msec (cfr. synchronization with lip reading) DSP challenges in hearing instruments Dynamic range compression (cfr supra) Dereverberation: undo filtering (`echo-ing’) by room acoustics Feedback cancellation Noise reduction

Case Study: Hearing Instruments 10/14 DSP Challenges: Feedback Cancellation Problem statement: Loudspeaker signal is fed back into microphone, then amplified and played back again Closed loop system may become unstable (howling) Similar to feedback problem in public address systems (for the musicians amongst you) Model F - Similar to echo cancellation in GSM handsets, Skype,… but more difficult due to signal correlation

Case Study: Hearing Instruments 11/14 DSP Challenges: Noise reduction Multimicrophone ‘beamforming’, typically with 2 microphones, e.g. ‘directional’ front microphone and ‘omnidirectional’ back microphone “filter-and-sum” the microphone signals

Case Study: Hearing Instruments 12/14 Binaural hearing: Binaural auditory cues ITD (interaural time difference) ILD (interaural level difference) Binaural cues (ITD: f < 1500Hz, ILD: f > 2000Hz) used for Sound localization Noise reduction =`Binaural unmasking’ (‘cocktail party’ effect) 0-5dB signal ILD ITD

Case Study: Hearing Instruments 13/14 Binaural hearing aids Two hearing aids (L&R) with wireless link & cooperation Opportunities: More signals (e.g. 2*2 microphones) Better sensor spacing (17cm i.o. 1cm) Constraints: power/bandwith/delay of wireless link ..10kBit/s: coordinate program settings, parameters,… ..300kBits/s: exchange 1 or more (compressed) audio signals Challenges: Improved localization through cue preservation Improved noise reduction + benefit from binaural unmasking Signal selection/filtering, audio coding, synchronisation, …

Case Study: Hearing Instruments 14/14 Future: Multi-node noise reduction – sensor networks

Overview General speech communication set-up : - background ‘noise’  noise suppression, source separation - far-end echoes  acoustic echo cancellation - reverberation  de-reverberation/deconvolution Applications : teleconferencing/teleclassing hands-free telephony hearing aids, etc..

Overview : Lecture-2&3 Microphone Array Processing Spatial filtering - Beamforming Fixed (Lecture 2) vs. adaptive (Lecture3) Example filter-and-sum beamformer : Application: hearing aids

Overview : Lecture-4 Noise Reduction `microphone_signal[k] = speech[k] + noise[k]’ Single-microphone noise reduction Spectral Subtraction Methods (spectral filtering) Iterative methods based on speech modeling (Wiener & Kalman Filters) Multi-microphone noise reduction Beamforming revisited Optimal filtering approach : spectral+spatial filtering

Overview : Lecture-5 Acoustic Echo Cancellation Adaptive filtering problem: non-stationary/wideband/… speech signals non-stationary/long/… acoustic channels Adaptive filtering algorithms AEC Control AEC Post-processing Stereo AEC

Overview : Lecture-6 Acoustic Feedback Cancellation amplifier Ex: Hearing aids Ex: PA systems correlation between filter input (`x ’) and near-end signal (‘ n ’) fixes : noise injection, pitch shifting, notch filtering, ... amplifier

Overview : Lecture-7 Active Noise Control Solution based on `filtered-X LMS’ Application : active headsets/ear defenders

Overview : Lecture-7bis & 8 3D Audio & Loudspeaker Arrays Binaural synthesis …with headphones head related transfer functions (HRTF) …with 2+ loudspeakers (`sweet spot’) crosstalk cancellation

Aims/Scope (revisited) Aim is 2-fold : Speech & audio per se Basic signal processing theory/principles : Optimal filtering / Kalman filters (linear/nonlinear) here : speech enhancement other : automatic control, spectral estimation, ... Advanced adaptive filter algorithms here : acoustic echo cancellation other : digital communications, ... Filtered-X LMS here : 3D audio other : active noise/vibration control

Lectures Lectures: 7*2hrs (Lectures 1-7) + 1*1hr (Lectures 8) PS: Time budget = (15hrs)*4 = 60 hrs Course Material: Slides Use version 2013-2014 ! Download from DASP webpage http://homes.esat.kuleuven.be/~dspuser/dasp/

Prerequisites H197 Signals & Systems (JVDW) HJ09 Digital Signal Processing (I) (PW) signal transforms, sampling, multi-rate, DFT, … HC63 DSP-CIS (MM) filter design, filter banks, optimal & adaptive filters

Literature Literature (General) (available in DSP-CIS library) Simon Haykin `Adaptive Filter Theory’ (Prentice Hall 1996) P.P. Vaidyanathan `Multirate Systems and Filter Banks’ (Prentice Hall 1993) Literature (specialized) (some available in DSP-CIS library) S.L. Gay & J. Benesty `Acoustic Signal Processing for Telecommunication’ (Kluwer 2000) M. Kahrs & K. Brandenburg (Eds) `Applications of Digital Signal Processing to Audio and Acoustics’ (Kluwer1998) B. Gold & N. Morgan `Speech and Audio Signal Processing’ (Wiley 2000)

Exercise Sessions/Project Acoustic source localization Direction-of-arrival estimation Noise reduction Echo cancellation Simulated set-up Direction-of-arrival θ

Acoustic Source Localization Project Runs over 4 weeks (non-consecutive) Each week 1 PC/Matlab session (supervised, 2.5hrs) 2 ‘Homework’ sesions (unsupervised, 2*2.5hrs) PS: Time budget = 4*(2.5hrs+5hrs) = 30 hrs ‘Deliverables’ after week 2 & 4 Grading: based on deliverables, evaluated during sessions TAs: guiliano.bernardi@esat (English+Italian) PS: groups of 2

Acoustic Source Localization Project Work Plan Week 1: Design Matlab simulation set-up Week 2: Direction-of-arrival (DoA) estimation *deliverable* Week 3: DoA estimation + noise reduction Week 4: DoA estimation + echo cancellation ..be there !

Exam Oral exam, with preparation time Open book Grading 7 for question-1 7 for question-2 +6 for project ___ = 20

September Retake Exam Oral exam, with preparation time Open book Grading 7 for question-1 7 for question-2 +6 for question-3 (related to project work) ___ = 20

Website TOLEDO http://homes.esat.kuleuven.be/~dspuser/dasp/ Contact: guiliano.bernardi@esat Slides (use `version 2014-2015’ !!) Schedule DSP-library FAQs (send questions to marc.moonen@esat)

Questions? Ask teaching assistant (during exercises sessions) E-mail questions to teaching assistant or marc.moonen@esat 3) Make appointment marc.moonen@esat ESAT Room 01.69