Presentation is loading. Please wait.

Presentation is loading. Please wait.

CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 1 Overview of QAST 2007 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1),

Published byDayna Nichols Modified over 9 years ago

Similar presentations

Presentation on theme: "CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 1 Overview of QAST 2007 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1),"— Presentation transcript:

1 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 1 Overview of QAST 2007 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1), C. Ayache, D. Mostefa (2), L. Lamel and S. Rosset (3) (1) UPC, Spain (2) ELDA, France (3) LIMSI, France QAST Website : http://www.lsi.upc.edu/~qast/

2 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 2 Outline 1.Task 2.Participants 3.Results 4.Conclusion and future work

3 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 3 Task: QAST 2007 Organization Task jointly organized by : -UPC, Spain (J. Turmo, P. Comas) Coordinator -ELDA, France (C. Ayache, D. Mostefa) -LIMSI-CNRS, France (S. Rosset, L. Lamel)

4 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 4 4 tasks were proposed: –T1 : QA in manual transcriptions of lectures –T2 : QA in automatic transcriptions of lectures –T3 : QA in manual transcriptions of meetings –T4 : QA in automatic transcriptions of meetings 2 data collections: –The CHIL corpus: around 25 hours (1 hour per lecture) Domain of lectures: Speech and language processing –The AMI corpus: around 100 hours (168 meetings) Domain of meetings: Design of television remote control Task: Evaluation Protocol

5 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 5 For each task, 2 sets of questions were provided: Development set (1 February 2007): –Lectures: 10 lectures, 50 questions –Meetings: 50 meetings, 50 questions Evaluation set (18 June 2007): –Lectures: 15 lectures, 100 questions –Meetings: 118 meetings, 100 questions Task: Questions and answer types

6 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 6 Factual questions Who is a guru in speech recognition? Expected answers = named entities. List of NEs: person, location, organization, language, system/method, measure, time, color, shape, material. No definition questions. Task: Questions and answer types

7 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 7 Assessors used QASTLE, an evaluation tool developed in Perl (by ELDA), to evaluate the data. Four possible judgments: –Correct –Incorrect –Inexact (too short or too long) –Unsupported (correct answers but wrong document) Task: Human judgment

8 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 8 Two metrics were used: –Mean Reciprocal Rank (MRR): measures how well ranked is a right answer. –Accuracy: the fraction of correct answers ranked in the first position in the list of 5 possible answers Participants could submit up to 2 submissions per task and 5 answers per question. Task: Scoring

9 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 9 Five teams submitted results for one or more QAST tasks: –CLT, Center for Language Technology, Australia ; –DFKI, Germany ; –LIMSI-CNRS, Laboratoire d’Informatique et de Mécanique des Sciences de l’Ingénieur, France ; –Tokyo Institute of Technology, Japan ; –UPC, Universitat Politècnica de Catalunya, Spain. In total, 28 submission files were evaluated: Participants CHIL Corpus (lectures)AMI Corpus (meetings) T1 (manual)T2 (ASR)T3 (manual)T4 (ASR) 8 submissions9 submissions5 submissions6 submissions

10 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 10 Due to some problems (typos, answer types and also missing time information at word level for some AMI meetings) some questions have been deleted from test set for scoring. Final counts: –T1 and T2: 98 questions –T3: 96 questions –T4: 93 questions Results

11 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 11 QA on CHIL manual transcriptions: Results for T1 System# Questions Returned # Correct Answers MRRAccuracy clt1_t198160.090.06 clt2_t198160.090.05 dfki1_t198190.170.15 limsi1_t198430.370.32 limsi2_t198560.460.39 tokyo1_t198320.190.14 tokyo2_t198340.200.14 upc1_t198540.530.51

12 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 12 QA on CHIL automatic transcriptions: Results for T2 System# Questions Returned # Correct Answers MRRAccuracy clt1_t298130.060.03 clt2_t298120.050.02 dfki1_t29890.09 limsi1_t298280.230.20 limsi2_t298280.240.21 tokyo1_t298170.120.08 tokyo2_t298180.120.08 upc1_t296370.370.36 upc2_t297290.250.24

13 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 13 QA on AMI manual transcriptions: Results for T3 System# Questions Returned # Correct Answers MRRAccuracy clt1_t396310.230.16 clt2_t396290.250.20 limsi1_t396310.280.25 limsi2_t396400.310.25 upc1_t395230.220.20

14 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 14 QA on AMI automatic transcriptions: Results for T4 System# Questions Returned # Correct Answers MRRAccuracy clt1_t493170.100.06 clt2_t493190.130.08 limsi1_t493210.190.18 limsi2_t493210.190.17 upc1_t491220.220.21 upc2_t492170.150.13

15 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 15 5 participants from 5 different countries (France, Germany, Spain, Australia and Japan) => 28 runs Very encouraging results QA technology can be useful to deal with spontaneous speech transcripts. High loss in accuracy with automatically transcribed speech Conclusion and future work

16 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 16 Future work aims at including: Other languages than English Oral questions Other question types: definition, list, etc. Other domains of data collections: European Parliament, broadcast news, etc. Conclusion and future work

17 CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 17 The QAST Website: http://www.lsi.upc.edu/~qast/ For more information

Download ppt "CLEF 2007 Workshop Budapest, September 19, 2007  ELDA 1 Overview of QAST 2007 - Question Answering on Speech Transcriptions - J. Turmo, P. Comas (1),"

Similar presentations

Markpong Jongtaveesataporn † Chai Wutiwiwatchai ‡ Koji Iwano † Sadaoki Furui † † Tokyo Institute of Technology, Japan ‡ NECTEC, Thailand.

Markpong Jongtaveesataporn † Chai Wutiwiwatchai ‡ Koji Iwano † Sadaoki Furui † † Tokyo Institute of Technology, Japan ‡ NECTEC, Thailand.
ResPubliQA 2010: QA on European Legislation Anselmo Peñas, UNED, Spain Pamela Forner, CELCT, Italy Richard Sutcliffe, U. Limerick, Ireland Alvaro Rodrigo,

ResPubliQA 2010: QA on European Legislation Anselmo Peñas, UNED, Spain Pamela Forner, CELCT, Italy Richard Sutcliffe, U. Limerick, Ireland Alvaro Rodrigo,
TrebleCLEF Evaluation package and language resources CLEF Evaluation Packages Nicolas Moreau / Khalid Choukri - ELDA.

TrebleCLEF Evaluation package and language resources CLEF Evaluation Packages Nicolas Moreau / Khalid Choukri - ELDA.
Overview of Collaborative Information Retrieval (CIR) at FIRE 2012 Debasis Ganguly, Johannes Leveling, Gareth Jones School of Computing, CNGL, Dublin City.

Overview of Collaborative Information Retrieval (CIR) at FIRE 2012 Debasis Ganguly, Johannes Leveling, Gareth Jones School of Computing, CNGL, Dublin City.
CLEF 2008 Multilingual Question Answering Track UNED Anselmo Peñas Valentín Sama Álvaro Rodrigo CELCT Danilo Giampiccolo Pamela Forner.

CLEF 2008 Multilingual Question Answering Track UNED Anselmo Peñas Valentín Sama Álvaro Rodrigo CELCT Danilo Giampiccolo Pamela Forner.
Vikas BhardwajColumbia University NLP for the Web – Spring 2010 Improving QA Accuracy by Question Inversion Prager et al. IBM T.J. Watson Res. Ctr. 02/18/2010.

Vikas BhardwajColumbia University NLP for the Web – Spring 2010 Improving QA Accuracy by Question Inversion Prager et al. IBM T.J. Watson Res. Ctr. 02/18/2010.
1 Question Answering in Biomedicine Student: Andreea Tutos Id: 41064739 Supervisor: Diego Molla.

1 Question Answering in Biomedicine Student: Andreea Tutos Id: Supervisor: Diego Molla.
Named Entity Recognition in an Intranet Query Log Richard Sutcliffe 1, Kieran White 1, Udo Kruschwitz 2 1 - University of Limerick, Ireland 2 - University.

Named Entity Recognition in an Intranet Query Log Richard Sutcliffe 1, Kieran White 1, Udo Kruschwitz University of Limerick, Ireland 2 - University.
XML Document Mining Challenge Bridging the gap between Information Retrieval and Machine Learning Ludovic DENOYER – University of Paris 6.

XML Document Mining Challenge Bridging the gap between Information Retrieval and Machine Learning Ludovic DENOYER – University of Paris 6.
CLEF 2007 Multilingual Question Answering Track Danilo Giampiccolo, CELCT Anselmo Peñas, UNED.

CLEF 2007 Multilingual Question Answering Track Danilo Giampiccolo, CELCT Anselmo Peñas, UNED.
1 Spontaneous-Speech Dialogue System In Limited Domains (98 - 01) Development of an oral human-machine interface, by way of dialogue, for a semantically.

1 Spontaneous-Speech Dialogue System In Limited Domains ( ) Development of an oral human-machine interface, by way of dialogue, for a semantically.
© Johan Bos November 2005 Carol Beer (Little Britain)

© Johan Bos November 2005 Carol Beer (Little Britain)
National Institute of Standards and Technology Information Technology Laboratory 2000 TREC-9 Spoken Document Retrieval Track

National Institute of Standards and Technology Information Technology Laboratory 2000 TREC-9 Spoken Document Retrieval Track
Spanish Question Answering Evaluation Anselmo Peñas, Felisa Verdejo and Jesús Herrera UNED NLP Group Distance Learning University of Spain CICLing 2004,

Spanish Question Answering Evaluation Anselmo Peñas, Felisa Verdejo and Jesús Herrera UNED NLP Group Distance Learning University of Spain CICLing 2004,
LREC 2008 1 Combining Multiple Models for Speech Information Retrieval Muath Alzghool and Diana Inkpen University of Ottawa Canada.

LREC Combining Multiple Models for Speech Information Retrieval Muath Alzghool and Diana Inkpen University of Ottawa Canada.
Lightly Supervised and Unsupervised Acoustic Model Training Lori Lamel, Jean-Luc Gauvain and Gilles Adda Spoken Language Processing Group, LIMSI, France.

Lightly Supervised and Unsupervised Acoustic Model Training Lori Lamel, Jean-Luc Gauvain and Gilles Adda Spoken Language Processing Group, LIMSI, France.
CLEF – Cross Language Evaluation Forum Question Answering at CLEF 2003 (http://clef-qa.itc.it) Bridging Languages for Question Answering: DIOGENE at CLEF-2003.

CLEF – Cross Language Evaluation Forum Question Answering at CLEF 2003 ( Bridging Languages for Question Answering: DIOGENE at CLEF-2003.
CLEF 2008 - Ǻrhus Robust – Word Sense Disambiguation exercise UBC: Eneko Agirre, Oier Lopez de Lacalle, Arantxa Otegi, German Rigau UVA & Irion: Piek Vossen.

CLEF Ǻrhus Robust – Word Sense Disambiguation exercise UBC: Eneko Agirre, Oier Lopez de Lacalle, Arantxa Otegi, German Rigau UVA & Irion: Piek Vossen.
NERIL: Named Entity Recognition for Indian FIRE 2013.

NERIL: Named Entity Recognition for Indian FIRE 2013.
CLEF – Cross Language Evaluation Forum Question Answering at CLEF 2003 (http://clef-qa.itc.it) The Multiple Language Question Answering Track at CLEF 2003.

CLEF – Cross Language Evaluation Forum Question Answering at CLEF 2003 ( The Multiple Language Question Answering Track at CLEF 2003.

Similar presentations

Ads by Google