The Development of the AMI System for the Transcription of Speech in Meetings Thomas Hain, Lukas Burget, John Dines, Iain McCowan, Giulia Garau, Martin Karafiat, Mike Lincoln, Darren Moore, Vincent Wan, Roeland Ordelman, Steve Renals July 12, 2005 MLMI Edinburgh

Outline Multi-site development Development strategy Resources Modelling System integration Results Conclusions MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

AMI ASR around the globe MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Multi-site development Large vocabulary ASR is complex and requires considerable resources Split development effort across multiple sites DICT LM CORE ADAPT AUDIO-PREPROC Central storage and compute resources Communication: frequent telephone conferences internet chat, “working phone calls” (VoIP), multiple workshops, WIKI MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Development paradigm Resource building Resource driven Dictionary LM Acoustic data Resource driven Bootstrap from conversational telephone speech (CTS) Generic technology selection Pick generic techniques with maximum gain VTLN, HLDA, MPE, CN Task specific components Front-ends Language models MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Resources Meeting resources “sparse” Language model data Dictionary Corpora: ICSI, ISL, NIST (LDC,VT) The AMI corpus (initial parts) 100 hours of meeting data Language model data Broadcast News (220MW) Web-data - CTS/AMI/Meeting (600MW) Meetings (ICSI/ISL/NIST/AMI) CTS (Swbd/Fisher) … Dictionary Edinburgh UNISYN MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Dictionary Baseline dictionary based upon UNISYN (Fitt, 2000) with 114,876 words Semi-automatic generation of pronunciations Part-word pronunciations initially automatically guessed from the existing pronunciations Automatic CART based letter-to-sound conversion trained from UNISYN Hand correction/checking of all automatic hypotheses Words were all converted to British spellings An additional 11,595 words were added using a combination of automatic and manual generation: Pronunciation probabilities (estimated from alignment of the training data) MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Vocabulary ICSI NIST ISL AMI 0.01 0.47 0.58 0.57 0.43 0.09 0.59 0.66 Source ICSI NIST ISL AMI 0.01 0.47 0.58 0.57 0.43 0.09 0.59 0.66 0.41 0.37 0.03 0.53 0.30 ALL 0.16 0.42 0.55 Test data Out of Vocabulary rates (OOV) with padding to 50k words from general Broadcast News data. No need for specific vocabulary ! MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Language modelling ICSI NIST ISL AMI ALL 68.2 74.6 73.8 77.1 68.0 Source   ICSI NIST ISL AMI ALL 68.2 74.6 73.8 77.1 68.0 105.9 100.9 102.0 106.0 101.3 104.7 99.5 98.5 106.4 102.9 115.6 114.3 114.4 88.9 94.1 107.5 105.7 90.6 92.7 Test data Interpolated trigram language models on meeting data optimised for each domain (on independent dev data) Perplexity results Meeting resource specific outperform general models Translates into 0.5% abs Word Error Rate improvement MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Acoustic modelling Standard HMM based framework Decision tree state clustered triphones Hidden Markov model toolkit (HTK) Maximum likelihood training Approx. 70k Gaussian/Model set MAP Adaptation from CTS models Bandwidth problem: CTS is narrowband data (4kHz), meetings are recorded at 8kHz bandwith Developed MLLR/MAP Front-end feature transform SHLDA = Smoothed Heteroscedastic Linear Discriminant Analysis Typically 1.5% WER improvement MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Speaker/channel adaptation CMN/CVN (channel) Vocal tract length normalisation (VTLN) maximum likelihood training & test Typically 3-4% WER gain MLLR Mean and variance Transforms for speech and silence Typically 1-2% improvement Histograms Warp factors female/male MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Front-ends Meeting recordings with a variety of source types Microphone locations Close talking: head-mounted/lapel Distant: “arbitrary location”, various array configuration Requires: speech activity detection, speaker “grouping”, speaker and location tracking. Objective: Achieve “close-talking” performance with distant microphones Enhancement type approach for simplicity MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

IHM front-end processing Signal enhancement (cross-talk suppression) LMS echo cancellation Speech activity detection (SAD) Using Multi-Layer Perceptron (MLP) Cross-talk suppression Feature extraction x (IHM channel) x’ (enhanced signal) Smoothing parameters (insertion penalty, minimum duration) x’ (36 dim feature vector) Yk (remaining IHM channels) MLP classification Viterbi decoder MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

IHM cross-talk suppression Multiple-reference LMS adaptive filtering with 256 tap FIR filter Adaptation is frozen during period of speech activity Automatic correction for channel timing misalignment MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Multiple distant microphones Gain Calibration Simple gain calibration is performed in which the maximum amplitude of each audio channel is normalised. Gain Calibration Noise Removal Noise spectrum of each input channel is estimated A Wiener filter is applied to each channel to remove stationary noise. Noise removal Delay Estimation Computed per frame Scale factors: ratio of energy Delay: peak finding in cross-correlation Delay Estimation Beamformer Beamformer filters using superdirective technique using a noise correlation matrix estimated above Beamformer MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Towards a model set Model initialisation (WER on ICSI only) Training data bandwidth adapt WER CTS NB - 33.3 ICSI 27.1 WB 25.3 MAP 25.8 MLLR + MAP 24.6 More training data Training data TOT ISL ICSI LDC NIST AMI-TOT UEDIN IDIAP ICS,NIST 50.4 56.2 24.1 61.1 36.9 59.1 60.2 58.4 ICSI,NIST,ISL 50.6 22.9 61.8 37.2 58.6 60.0 57.6 ICSI,NIST,ISL,AMI 50.3 54.5 27.4 61.3 36.2 57.3 59.0 MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

First pass recognition System architecture Front-end (IHM/MDM) Modified Audio, Segments, Speaker Info First pass recognition First recognition result Adaptation Lattice generation Word lattices LM Rescoring Final word level result MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Results on rt05seval CTS-adapted ML models, unadapted, trigram LM (first pass)   TOT Sub Del Ins AMI ISL ICSI NIST VT IHM 41.1 21.1 14.7 5.3 42.3 36.3 37.1 49.1 IHMREF 34.9 23.0 7.1 4.8 34.5 34.0 26.6 42.2 37.9 MDM 53.6 32.1 17.3 4.1 46.5 50.2 48.2 63.0 MDMREF 50.6  34.1 11.8 4.6 43.0 49.4 46.4 49.9 59.5 MDM segmentation provided by ICSI/SRI REF denotes reference segmentation and speaker labels The performance of the full system on the above AMI subset is 30.9% for IHM and 35.1% for MDM. BUT: difference on REF remains MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005

Conclusions Multi-site development! Competitive ASR system in 10-11 months Meeting domains inhomogeneous ? Good improvements with VTLN/SHLDA/MLLR Pre-processing needs to be sorted ! Reasonable performance on Seminar data MLMI – The AMI ASR Meeting System Thomas Hain / July 12, 2005