Presentation is loading. Please wait.

Presentation is loading. Please wait.

CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 1 Overview of QAST 2008 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1), L.

Published byRodger Harvey Modified over 8 years ago

Similar presentations

Presentation on theme: "CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 1 Overview of QAST 2008 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1), L."— Presentation transcript:

1 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 1 Overview of QAST 2008 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1), L. Lamel, S. Rosset (2), N. Moreau, D. Mostefa (3) (1) UPC, Spain (2) LIMSI, France (3) ELDA, France QAST Website : http://www.lsi.upc.edu/~qast/

2 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 2 Outline 1.Objectives 2.Description of the tasks 3.Participants 4.Results 5.Future work

3 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 3 Objectives of QAST 2008 -Development of robust QA for speech transcripts -Measure loss due to ASR inaccuracies manual transcriptions, automatic transcriptions -Measure loss at different ASR word error rates -Test with different kinds of speech spontaneous speech, prepared speech -Development of QA for languages other than English English, French, Spanish

4 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 4 QAST 2008 Organization Task jointly organized by : -UPC, Spain (Coordinator) J. Turmo, P. Comas -ELDA, France N. Moreau, D. Mostefa -LIMSI-CNRS, France S. Rosset, L. Lamel

5 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 5 Evaluation Data CorpusLang.DescriptionTasksWER CHIL QAST 2007 EnglishLectures (~25h)T1(a): Manual transcriptions - T1(b): ASR transcriptions 20% AMI QAST 2007 EnglishMeetings (~100h)T2(a): Manual transcriptions - T2(b): ASR transcriptions 38% ESTERFrenchBroadcast News (~10h) T3(a): Manual transcriptions - T3(b): ASR transcriptions 11.9% / 23.9% / 35.4% EPPS- EN EnglishSessions European Parliament (~3h) T4(a): Manual transcriptions - T4(b): ASR transcriptions 10.6% / 14.0% / 24.1% EPPS- ES SpanishSessions European Parliament (~3h) T5(a): manual transcriptions - T5(b): ASR transcriptions 11.5% / 12.7% / 13.7%

6 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 6 Development setEvaluation set TaskData# questionsData# questions T1 (CHIL, English)10 seminars5015 seminars100 T2 (AMI, English)50 meetings50118 meetings100 T3 (ESTER, French)6 shows5012 shows100 T4 (EPPS, English)3 sessions503 sessions100 T5 (EPPS, Spanish)1 session505 sessions100 Questions Factual questions: ~75% Expected answers = named entities (10 types: person, location, organization, language, system, measure, time, color, shape, material) Definition questions: ~25% 4 types of answers: person, organization, object, other ‘NIL’ questions: ~10%

7 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 7 Participants could submit up to: –2 submissions per task (and per WER) –5 answers per question Answers for ‘manual transcriptions’ tasks: Answer_string + Doc_ID Answers for ‘automatic transcriptions’ tasks: Answer_string + Doc_ID + Time_start + Time_end Submissions

8 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 8 Four possible judgments (as in QA@CLEF): Correct / Incorrect / Inexact / Unsupported ‘Manual transcriptions’ tasks: Manual assessment with the QASTLE interface ‘Automatic’ transcriptions tasks Automatic assessment (script) + manual check 2 metrics: –Mean Reciprocal Rank (MRR) measures how well right answers are ranked on average –Accuracy fraction of correct answers ranked in the first position Assessments

9 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 9 49 submissions from 5 participants: Participants T1aT1bT2aT2bT3aT3bT4aT4bT5aT5b 2-----2--- ------12-- 1111231323 ------13-- 1212--1616 43232361439 Univ. Chemnitz (CUT) INAOE LIMSI Univ. Alicante (UA) UPC TOTAL:

10 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 10 Best results for manual transcriptions Task T1a T2a T3a T4a T5a Factual MRRAcc(%) 0.5347.4 0.4737.8 0.5045.3 0.4440.0 0.3229.3 Definitional MRRAcc(%) 0.1818.2 0.2219.2 0.4744.0 0.1616.0 0.4436.0 All MRRAcc(%) 0.4541.0 0.4033.0 0.4945.0 0.3734.0 0.3531.0

11 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 11 Best results for ASR transcriptions TaskWER T1b20.0% T2b38.0% T3b11.9% 23.9% 35.4% T4b10.6% 14.0% 24.1% T5b11.5% 12.7% 13.7% All MRRAcc(%) 0.3431.0 0.2018.0 0.4541.0 0.3025.0 0.2421.0 0.3330.0 0.2420.0 0.2319.0 0.2624.0 0.2320.0 0.2523.0 All (manual) MRRAcc(%) 0.4541.0 0.4033.0 0.4945.0 0.3734.0 0.3531.0

12 CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 12 5 participants (as in 2007) 4 different countries (vs. 5 in 2007) Germany, Spain, France, Mexico 49 submitted runs (vs. 28 runs in 2007) Loss in accuracy with ASR transcribed speech (performance falls when WER rises) QAST 2009: Written & Oral Questions... Conclusion

Download ppt "CLEF 2008 Workshop Aarhus, September 17, 2008  ELDA 1 Overview of QAST 2008 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1), L."

Similar presentations

Seminar on Language Teaching IKG 743 (2) A LECTURE BY Rida Wahyuningrum ENGLISH DEPARTMENT FACULTY OF LANGUAGE AND SCIENCE SURABAYA WIJAYA KUSUMA UNIVERSITY.

Seminar on Language Teaching IKG 743 (2) A LECTURE BY Rida Wahyuningrum ENGLISH DEPARTMENT FACULTY OF LANGUAGE AND SCIENCE SURABAYA WIJAYA KUSUMA UNIVERSITY.
SINAI-GIR A Multilingual Geographical IR System University of Jaén (Spain) José Manuel Perea Ortega CLEF 2008, 18 September, Aarhus (Denmark) Computer.

SINAI-GIR A Multilingual Geographical IR System University of Jaén (Spain) José Manuel Perea Ortega CLEF 2008, 18 September, Aarhus (Denmark) Computer.
Markpong Jongtaveesataporn † Chai Wutiwiwatchai ‡ Koji Iwano † Sadaoki Furui † † Tokyo Institute of Technology, Japan ‡ NECTEC, Thailand.

Markpong Jongtaveesataporn † Chai Wutiwiwatchai ‡ Koji Iwano † Sadaoki Furui † † Tokyo Institute of Technology, Japan ‡ NECTEC, Thailand.
Overview of the TAC2013 Knowledge Base Population Evaluation: English Slot Filling Mihai Surdeanu with a lot help from: Hoa Dang, Joe Ellis, Heng Ji, and.

Overview of the TAC2013 Knowledge Base Population Evaluation: English Slot Filling Mihai Surdeanu with a lot help from: Hoa Dang, Joe Ellis, Heng Ji, and.
ResPubliQA 2010: QA on European Legislation Anselmo Peñas, UNED, Spain Pamela Forner, CELCT, Italy Richard Sutcliffe, U. Limerick, Ireland Alvaro Rodrigo,

ResPubliQA 2010: QA on European Legislation Anselmo Peñas, UNED, Spain Pamela Forner, CELCT, Italy Richard Sutcliffe, U. Limerick, Ireland Alvaro Rodrigo,
Thomas Mandl: Robust CLEF 2006 - Overview 1 Cross-Language Evaluation Forum (CLEF) Thomas Mandl Information Science Universität Hildesheim

Thomas Mandl: Robust CLEF Overview 1 Cross-Language Evaluation Forum (CLEF) Thomas Mandl Information Science Universität Hildesheim
TrebleCLEF Evaluation package and language resources CLEF Evaluation Packages Nicolas Moreau / Khalid Choukri - ELDA.

TrebleCLEF Evaluation package and language resources CLEF Evaluation Packages Nicolas Moreau / Khalid Choukri - ELDA.
CLEF 2008 Multilingual Question Answering Track UNED Anselmo Peñas Valentín Sama Álvaro Rodrigo CELCT Danilo Giampiccolo Pamela Forner.

CLEF 2008 Multilingual Question Answering Track UNED Anselmo Peñas Valentín Sama Álvaro Rodrigo CELCT Danilo Giampiccolo Pamela Forner.
Combining Prosodic and Text Features for Segmentation of Mandarin Broadcast News Gina-Anne Levow University of Chicago SIGHAN July 25, 2004.

Combining Prosodic and Text Features for Segmentation of Mandarin Broadcast News Gina-Anne Levow University of Chicago SIGHAN July 25, 2004.
3rd Answer Validation Exercise ( AVE 2008) QA subtrack at Cross-Language Evaluation Forum 2008 UNED Anselmo Peñas Álvaro Rodrigo Felisa Verdejo Thanks.

3rd Answer Validation Exercise ( AVE 2008) QA subtrack at Cross-Language Evaluation Forum 2008 UNED Anselmo Peñas Álvaro Rodrigo Felisa Verdejo Thanks.
ASR Evaluation Julia Hirschberg CS 4706. Outline Intrinsic Methods –Transcription Accuracy Word Error Rate Automatic methods, toolkits Limitations –Concept.

ASR Evaluation Julia Hirschberg CS Outline Intrinsic Methods –Transcription Accuracy Word Error Rate Automatic methods, toolkits Limitations –Concept.
1 CLEF 2009, Corfu Question Answering Track Overview J. Turmo P.R. Comas S. Rosset O. Galibert N. Moreau D. Mostefa P. Rosso D. Buscaldi D. Santos L.M.

1 CLEF 2009, Corfu Question Answering Track Overview J. Turmo P.R. Comas S. Rosset O. Galibert N. Moreau D. Mostefa P. Rosso D. Buscaldi D. Santos L.M.
Alicante, September, 22, 20006 2006 Workshop Overview of the Multilingual Question Answering Track Danilo Giampiccolo.

Alicante, September, 22, Workshop Overview of the Multilingual Question Answering Track Danilo Giampiccolo.
CLEF 2007 Multilingual Question Answering Track Danilo Giampiccolo, CELCT Anselmo Peñas, UNED.

CLEF 2007 Multilingual Question Answering Track Danilo Giampiccolo, CELCT Anselmo Peñas, UNED.
Answer Validation Exercise Anselmo Peñas UNED NLP Group 2005 Breakout session.

Answer Validation Exercise Anselmo Peñas UNED NLP Group 2005 Breakout session.
Baccus Enrichment 2014. Start packing! You have won a trip to visit ONE terrific European country of your choice so that you can come back and give a.

Baccus Enrichment Start packing! You have won a trip to visit ONE terrific European country of your choice so that you can come back and give a.
Search is not only about the Web An Overview on Printed Documents Search and Patent Search Walid Magdy Centre for Next Generation Localisation School of.

Search is not only about the Web An Overview on Printed Documents Search and Patent Search Walid Magdy Centre for Next Generation Localisation School of.
Spanish Question Answering Evaluation Anselmo Peñas, Felisa Verdejo and Jesús Herrera UNED NLP Group Distance Learning University of Spain CICLing 2004,

Spanish Question Answering Evaluation Anselmo Peñas, Felisa Verdejo and Jesús Herrera UNED NLP Group Distance Learning University of Spain CICLing 2004,
LREC 2008 1 Combining Multiple Models for Speech Information Retrieval Muath Alzghool and Diana Inkpen University of Ottawa Canada.

LREC Combining Multiple Models for Speech Information Retrieval Muath Alzghool and Diana Inkpen University of Ottawa Canada.
Lightly Supervised and Unsupervised Acoustic Model Training Lori Lamel, Jean-Luc Gauvain and Gilles Adda Spoken Language Processing Group, LIMSI, France.

Lightly Supervised and Unsupervised Acoustic Model Training Lori Lamel, Jean-Luc Gauvain and Gilles Adda Spoken Language Processing Group, LIMSI, France.

Similar presentations

Ads by Google