LREC 2010 Malta, May 20, 2010  ELDA 1 Evaluation Protocol and Tools for Question-Answering on Speech Transcripts N. Moreau, O. Hamon, D. Mostefa ELDA/ELRA, Paris, France S. Rosset, O. Galibert, L. LamelLIMSI, Paris, France J. Turmo, P. R. ComasUPC, Barcelona, Spain P. Rosso, D. BuscaldiUPV, Valencia, Spain Contact:

LREC 2010 Malta, May 20, 2010  ELDA Outline -What is QAST? -QAST evaluations -Evaluation data and tasks -QASTLE evaluation interface -Overview of main results -Conclusions and perspectives 2

LREC 2010 Malta, May 20, 2010  ELDA 3 What is QAST? -QAST stands for Question-Answering on Speech Transcripts -4 QAST evaluation campaigns (in 2006, 2007, 2008, 2009) -Organized by UPC, UPV, LIMSI and ELDA. -Goals: -Development of robust QA for speech -Measure loss due to ASR inaccuracies -Measure loss at different ASR word error rates -Measure loss when using oral spontaneous questions (in 2009)

LREC 2010 Malta, May 20, 2010  ELDA QAST evaluations CHIL(EN) Manual transc 1 ASR output CHIL (EN) AMI (EN) Manual transc 1 ASR output +Words graph CHIL(EN) AMI (EN) ESTER (FR) EPPS (EN) EPPS (ES) Manual trans 3 ASR ESTER (FR) EPPS (EN) EPPS (ES) Manual trans 3 ASR outputs Written questionsOral questions 4

LREC 2010 Malta, May 20, 2010  ELDA 5 QAST Data Sets CorpusLanguageDescriptionSpeech TranscriptsWERCampaigns CHILEnglish 25 lectures (~25h) Manual-2006,2007, 2008 ASR20%2006, 2007, 2008 AMIEnglish 168 meetings (~100h) Manual-2007, 2008 ASR38%2007, 2008 ESTERFrench 18 BN shows (~10h) Manual-2008, 2009 ASR11.9%2008, 2009 ASR23.9%2008, 2009 ASR35.4%2008, 2009 EPPSEnglish 6 sessions (~3h) Manual-2008, 2009 ASR10.6%2008, 2009 ASR14.0%2008, 2009 ASR24.1%2008, 2009 EPPSSpanish 6 sessions (~3h) Manual-2008, 2009 ASR11.5%2008, 2009 ASR12.7%2008, 2009 ASR13.7%2008, 2009

LREC 2010 Malta, May 20, 2010  ELDA 6 Questions and Evaluation Tasks Different evaluation tasks: –QA in manual transcriptions –QA in automatic transcriptions (ASR) –QA using written questions –QA using transcription of oral questions Question sets created each year for each dataset: –100 questions for training + 50 questions for tests –Question type: Factual + Definitional –New in 2009: spontaneous oral questions

LREC 2010 Malta, May 20, 2010  ELDA 7 Creation of oral questions People were presented short text excerpts (taken from the corpus) After reading each excerpt they had to ask a few ‘spontaneous’ questions Oral questions were recorded Oral questions were manually transcribed (including speech disfluencies) A canonical “written” version was created for each question Example:Oral:When did the bombing of Fallujah t() take euh took place? Written:When did the bombing of Fallujah take place?

LREC 2010 Malta, May 20, 2010  ELDA 8 Up to 5 ranked answers per question Answers for ‘manual transcriptions’ tasks: Answer_string + Doc_ID Answers for ‘automatic transcriptions’ tasks: Answer_string + Doc_ID + Time_start + Time_end Submissions Time slot of the answer

LREC 2010 Malta, May 20, 2010  ELDA Four possible judgments : Correct / Incorrect / Inexact / Unsupported QA on manual transcriptions: Manual assessment with the QASTLE interface QA on automatic (ASR) transcriptions: Automatic assessment (script) + manual check with QASTLE 2 metrics: –Mean Reciprocal Rank (MRR) measures how well right answers are ranked on average –Accuracy fraction of correct answers ranked in the first position Assessments 9

LREC 2010 Malta, May 20, 2010  ELDA 10 QASTLE interface

LREC 2010 Malta, May 20, 2010  ELDA Automatic script to assess QA on ASR transcriptions The script compares of time slot boundaries of: –Reference time slot (created beforehand) –Hypothesis time slot (submitted answer) The overlap is compared to a predefined threshold: –overlap > threshold => Answer is CORRECT –overlap Answer is INEXACT –no overlap=> Answer is INCORRECT 2nd pass: Manual check with QASTLE Semi-automatic assessments 11

LREC 2010 Malta, May 20, 2010  ELDA 12 Best results (Accuracy %) CorpusTranscr Written Q Oral Q. CHIL Manual ASR (20.0%) AMI Manual ASR (38.0%) ESTER Manual ASR (11.9%) ASR (23.9%) ASR (35.4%) EPPS-EN Manual ASR (10.6%) ASR (14.0%) ASR (24.1%) EPPS-ES Manual ASR (11.5%) ASR (12.7%) ASR (13.7%)

LREC 2010 Malta, May 20, 2010  ELDA 13 We presented evaluation campaigns of QA on speech data Evaluations were done for several languages and on different data (seminars, meetings, BN, parliament speeches) New methodology for semi-automatic evaluation of QA in ASR transcriptions QASTLE interface free for download Conclusion & perspectives (1/2)

LREC 2010 Malta, May 20, 2010  ELDA Future evaluation campaigns: –Multilingual / cross lingual QA –Oral questions with ASR transcription of the questions QAST evaluation package soon available through the ELRA Catalog of language resources Conclusion & perspectives

LREC 2010 Malta, May 20, 2010  ELDA 15 Thank you for your attention... QAST : QASTLE : ELRA Catalogue of Language Resources: