CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 1 Overview of QAST Question Answering on Speech Transcriptions - J. Turmo, P. R. Comas,TALP Research Centre (UPC), Barcelona, Spain S. Rosset, O. Galibert, LIMSI, Paris, France N. Moreau, D. MostefaELDA/ELRA, Paris, France P. Rosso, D. BuscaldiNLE Lab. - ELiRF Research Group (UPV), Spain QAST Website :

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 2 Objectives of QAST #3 (after 2007 & 2008)‏ -Development of robust QA for speech transcripts -Measure loss due to ASR inaccuracies Manual Transcriptions / ASR Transcriptions -Measure loss at different ASR word error rates -New in 2009: test with oral questions Written Questions / Spontaneous Oral Questions

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 3 Evaluation Data & Tasks 1 manual 3 ASR: WER= 11.9%, 23.9%, 35.4% T3(a): Written Questions10h (18 Broadcast News Shows)‏ FrenchESTER T3(b): Oral Questions T2(b): Oral Questions T2(a): Written Questions T1(b): Oral Questions T1(a): Written Questions Tasks 1 manual 3 ASR: WER= 11.5%, 12.7%, 13.7% 3h (6 European Parliament Sessions)‏ SpanishEPPS-ES 1 manual 3 ASR: WER= 10.6%, 14.0%, 24.1% 3h (6 European Parliament Sessions)‏ EnglishEPPS-EN TranscriptionsDescriptionLang.Corpus For each task, 4 different transcriptions: 1 manual transcription 3 ASR transcriptions (  WER)‏

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 4 Oral Questions Procedure to create spontaneous oral questions: Random selection of passages in the collections Humans read passages and ask a few questions Exact manual transcriptions of spontaneous questions Question filtering (remove invalid ones)‏ Creation of written versions of oral questions Example:ORAL: When did the bombing of Fallujah t() take took place? WRITTEN: When did the bombing of Fallujah take place? #speakers #quest. recorded #valid 7.1T3 (ESTER, French)‏ 7.7T2 (EPPS, Spanish)‏ 9.1T1 (EPPS, English)‏ avg. #wordsTask

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 5 68% 55% 75% %Fact. 32% 45% 25% %Def. 21% 23% 18% %NIL 10050T3 (ESTER, French)‏ 10050T2 (EPPS, Spanish)‏ 10050T1 (EPPS, English)‏ # test questions # dev questions Task Final Question Sets Final Selection: Factual questions: 5 types: Person, Location, Organization, Measure, Time Definition questions: 3 types of answers: Person, Organization, Other ‘NIL’ questions

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 6 Participants could submit up to: –2 submissions per task and transcript => max. 48 –Up to 5 ranked answers per question Answers for ‘manual transcriptions’ tasks: Answer_string + Doc_ID Answers for ‘automatic transcriptions’ tasks: Answer_string + Doc_ID + Time_start + Time_end Submissions

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 7 Four possible judgments (as in Correct / Incorrect / Inexact / Unsupported ‘Manual transcriptions’ tasks: Manual assessment with the QASTLE interface ‘Automatic’ transcriptions tasks Automatic assessment (script) + manual check 2 metrics: –Mean Reciprocal Rank (MRR)‏ measures how well right answers are ranked on average –Accuracy fraction of correct answers ranked in the first position Assessments

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 8 86 submissions from 4 participants: Participants T3 French T2 Spanish T1 English T3bT3aT2bT2aT1bT1a LIMSI (France)‏ TOTAL: UPC (Spain)‏ TOK (Japan)‏ INAOE (Mexico)‏

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 9 Best results for ASR transcriptions Manual 10.6% T1 14.0% 24.1% 11.5% T2 12.7% 13.7% 11.9% T3 23.9% 35.4% TaskWER 29.0% % % % % % % % % % % %0.39 Acc(%)‏MRR (a) Written – All 28.0% % % % % % % % % % % %0.37 Acc(%)‏MRR (b) Oral – All

CLEF 2009 Workshop Corfu, September 30, 2009  ELDA 10 4 participants (5 in 2007 and 2008)‏ New methodology for creating “spontaneous” questions Loss in accuracy compared to 2008 Even harder evaluation, but closer to real applications QAST 2010 ??? –Difficult task, but promising –Find more participants –Find new data (manual + ASR transcriptions)… Conclusion