Presentation is loading. Please wait.

Presentation is loading. Please wait.

LREC 2008 1 Combining Multiple Models for Speech Information Retrieval Muath Alzghool and Diana Inkpen University of Ottawa Canada.

Published byMarilyn Richardson Modified over 9 years ago

Similar presentations

Presentation on theme: "LREC 2008 1 Combining Multiple Models for Speech Information Retrieval Muath Alzghool and Diana Inkpen University of Ottawa Canada."— Presentation transcript:

1 LREC 2008 1 Combining Multiple Models for Speech Information Retrieval Muath Alzghool and Diana Inkpen University of Ottawa Canada

2 LREC 2008 2 Presentation Outline Task: Speech Information Retrieval. Data: Mallach collection (Oard et al, 2004). System description. Model fusion. Experiments using model fusion. Results of the cross-language experiments. Results of manual keywords and summaries. Conclusion and future work.

3 LREC 2008 3 The Mallach collection Used in the Cross-Language Speech Retrieval (CLSR) task at Cross-Language Evaluation Forum (CLEF) 2007. 8104 “documents” (segments) from 272 interviews with Holocaust survivors, totaling 589 hours of speech: ASR transcripts with a word error rate of 25-38%. Additional metadata: automatically-assigned keywords, manually-assigned keywords, and a manual 3-sentence summary. A set of 63 training topics and 33 test topics, created in English from actual user requests and translated into Czech, German, French, and Spanish by native speakers. Relevance judgments were generated standard pooling.

4 LREC 2008 4 Segments VHF[IntCode]-[SegId].[SequenceNum] Interviewee name(s) and birthdate Full name of every person mentioned Thesaurus keywords assigned to the segment 3-sentence segment summary ASR transcript produced in 2004 ASR transcript produced in 2006 Thesaurus keywords from a kNN classifier Thesaurus keywords from a second kNN classifier

5 LREC 2008 5 Example topic (English) 1159 Child survivors in Sweden Describe survival mechanisms of children born in 1930- 1933 who spend the war in concentration camps or in hiding and who presently live in Sweden. The relevant material should describe the circumstances and inner resources of the surviving children. The relevant material also describes how the wartime experience affected their post-war adult life.

6 LREC 2008 6 Example topic (French) 1159 Les enfants survivants en Suède Descriptions des mécanismes de survie des enfants nés entre 1930 et 1933 qui ont passé la guerre en camps de concentration ou cachés et qui vivent actuellement en Suède. Les documents recherches devront décrire les circonstances ainsi que les ressources propres aux enfants ayant survécu. Les documents en question devront décrire également l'influence que l'expérience de la guerre a eu sur leur vie d'adulte après la guerre.

7 LREC 2008 7 System Description SMART: Vector Space Model (VSM). Terrier: Divergence from Randomness models (DFR). Two Query Expansion Methods: Based on thesaurus (novel technique). Blind relevance feedback (12 terms from the top 15 documents): based on Bose-Einstein 1 model (Bo1 from Terrier). Model Fusion: sum of normalized weighted similarity scores (novel way to compute weights). Combined output of 7 machine translation tools.

8 LREC 2008 8 Model Fusion Combine the results of different retrieval strategies from SMART (14 runs) and Terrier (1 run). Each technique will retrieve different sets of relevant documents; therefore combining the results could produce a better result than any of the individual techniques.

9 LREC 2008 9 Experiments using Model Fusion Applied the data fusion methods to 14 runs produced by SMART and one run produced by Terrier. % change is given with respect to the run providing better performance in each combination on the training data. Model fusion helps to improve the performance (MAP and Recall score) on the test data. Monolingual (English): 6.5% improvement (not statistically significant). Cross-language experiments (French) : 21.7% improvement (significant).

10 LREC 2008 10 Experiments using Model Fusion (MAP) Weighting scheme Manual En Auto-English Auto- FrenchAuto- Spanish Train.TestTrain.TestTrain.TestTrain.Test 1.nnc.ntc0.25460.22930.0888 0.0819 0.07920.0550.05930.0614... 11.nnn.ntn0.24760.2228 0.09000.0852 0.07990.05030.05990.061 12.ntn.nnn0.27380.2369 0.09330.0795 0.08430.05070.06910.0578 13.ltn.ntn0.28580.245 0.09690.0799 0.09050.05660.07010.0589 14.atn.ntn0.28430.2364 0.06200.0546 0.07220.03470.05860.0355 15.In(exp)C2 0.3177 0.2737 0.08850.0744 0.09080.04870.07470.0614 Fusion 1 0.32080.27610.09690.0855 0.09120.06220.07310.0682 % change1.0%0.9%0.0%6.5%0.4%21.7%-2.2%10.0% Fusion 2 0.31820.27410.09750.0842 0.09420.06020.07520.0619 % change0.2%0.1%0.6%5.1%3.6%19.1%0.7%0.8%

11 LREC 2008 11 Experiments using Model Fusion (Recall) Weighting scheme Manual En Auto-English Auto- FrenchAuto- Spanish Train.TestTrain.TestTrain.TestTrain.Test 1.nnc.ntc23711827 17261306 1687112215621178... 11.nnn.ntn23701823 17401321 1748115816431190 12.ntn.nnn23701823 17401321 1748115816431190 13.ltn.ntn24331876 15821215 1478107014081134 14.atn.ntn24421859 14551101 139097512971037 15.In(exp)C2 2638 1823 16241286 1676106116311172 Fusion 126451832 17451334 1759114716451219 % change0.3%0.5 %0.3%1.0%0.6%-1.0%0.1%2.4% Fusion 226471823 17271337 1736109816311172 % change 0.3%0.0% 0.8%1.2%-0.7% -5.5%-0.7%-1.5%

12 LREC 2008 12 Results of the cross-language experiments The cross-language results for French are very close to Monolingual (English) on training data (the difference is not significant), but not on test data (the difference is significant). The difference is significant between cross-language results for Spanish and Monolingual (English) on training data but not on test data (the difference is not significant). LanguageTrainingTest 1 English0.09690.0855 2 French0.09120.0622 3 Spanish0.07310.0682

13 LREC 2008 13 Results of manual keywords and summaries LanguageTrainingTest 1 Manual English 0.320810.2761 2 Auto-English 0.09690.0855 Experiments on manual keywords and manual summaries showed high improvements comparing to Auto-English. Our results (for manual and automatic runs) are the highest to date on this data collection in CLEF/CLSR.

14 LREC 2008 14 Conclusion and future work Model fusion helps to improve the retrieval significantly for some experiments (Auto-French) and for other not significantly (Auto-English). The idea of using multiple translations proved to be good (based on previous experiments). Future work  we plan to investigate more methods of model fusion.  Removing or correcting some of the speech recognition errors in the ASR content words.

Download ppt "LREC 2008 1 Combining Multiple Models for Speech Information Retrieval Muath Alzghool and Diana Inkpen University of Ottawa Canada."

Similar presentations

Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki 2008-11629.

Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
SINAI-GIR A Multilingual Geographical IR System University of Jaén (Spain) José Manuel Perea Ortega CLEF 2008, 18 September, Aarhus (Denmark) Computer.

SINAI-GIR A Multilingual Geographical IR System University of Jaén (Spain) José Manuel Perea Ortega CLEF 2008, 18 September, Aarhus (Denmark) Computer.
A Vector Space Model for Automatic Indexing

A Vector Space Model for Automatic Indexing
TI: An Efficient Indexing Mechanism for Real-Time Search on Tweets Chun Chen 1, Feng Li 2, Beng Chin Ooi 2, and Sai Wu 2 1 Zhejiang University, 2 National.

TI: An Efficient Indexing Mechanism for Real-Time Search on Tweets Chun Chen 1, Feng Li 2, Beng Chin Ooi 2, and Sai Wu 2 1 Zhejiang University, 2 National.
Explorations in Tag Suggestion and Query Expansion Jian Wang and Brian D. Davison Lehigh University, USA SSM 2008 (Workshop on Search in Social Media)

Explorations in Tag Suggestion and Query Expansion Jian Wang and Brian D. Davison Lehigh University, USA SSM 2008 (Workshop on Search in Social Media)
Modern Information Retrieval

Modern Information Retrieval
Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.

Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.
Modern Information Retrieval Chapter 5 Query Operations.

Modern Information Retrieval Chapter 5 Query Operations.
Learning to Advertise. Introduction Advertising on the Internet = $$$ –Especially search advertising and web page advertising Problem: –Selecting ads.

Learning to Advertise. Introduction Advertising on the Internet = $$$ –Especially search advertising and web page advertising Problem: –Selecting ads.
1 LM Approaches to Filtering Richard Schwartz, BBN LM/IR ARDA 2002 September 11-12, 2002 UMASS.

1 LM Approaches to Filtering Richard Schwartz, BBN LM/IR ARDA 2002 September 11-12, 2002 UMASS.
Query Reformulation: User Relevance Feedback. Introduction Difficulty of formulating user queries –Users have insufficient knowledge of the collection.

Query Reformulation: User Relevance Feedback. Introduction Difficulty of formulating user queries –Users have insufficient knowledge of the collection.
An investigation of query expansion terms Gheorghe Muresan Rutgers University, School of Communication, Information and Library Science 4 Huntington St.,

An investigation of query expansion terms Gheorghe Muresan Rutgers University, School of Communication, Information and Library Science 4 Huntington St.,
Cross-Language Retrieval INST 734 Module 11 Doug Oard.

Cross-Language Retrieval INST 734 Module 11 Doug Oard.
DIVINES – Speech Rec. and Intrinsic Variation W.S.May 20, 2006 Richard Rose DIVINES SRIV Workshop The Influence of Word Detection Variability on IR Performance.

DIVINES – Speech Rec. and Intrinsic Variation W.S.May 20, 2006 Richard Rose DIVINES SRIV Workshop The Influence of Word Detection Variability on IR Performance.
Search is not only about the Web An Overview on Printed Documents Search and Patent Search Walid Magdy Centre for Next Generation Localisation School of.

Search is not only about the Web An Overview on Printed Documents Search and Patent Search Walid Magdy Centre for Next Generation Localisation School of.
A New Approach for Cross- Language Plagiarism Analysis Rafael Corezola Pereira, Viviane P. Moreira, and Renata Galante Universidade Federal do Rio Grande.

A New Approach for Cross- Language Plagiarism Analysis Rafael Corezola Pereira, Viviane P. Moreira, and Renata Galante Universidade Federal do Rio Grande.
Evaluating the Contribution of EuroWordNet and Word Sense Disambiguation to Cross-Language Information Retrieval Paul Clough 1 and Mark Stevenson 2 Department.

Evaluating the Contribution of EuroWordNet and Word Sense Disambiguation to Cross-Language Information Retrieval Paul Clough 1 and Mark Stevenson 2 Department.
August 21, 2002Szechenyi National Library Support for Multilingual Information Access Douglas W. Oard College of Information Studies and Institute for.

August 21, 2002Szechenyi National Library Support for Multilingual Information Access Douglas W. Oard College of Information Studies and Institute for.
CLEF 2008 - Ǻrhus Robust – Word Sense Disambiguation exercise UBC: Eneko Agirre, Oier Lopez de Lacalle, Arantxa Otegi, German Rigau UVA & Irion: Piek Vossen.

CLEF Ǻrhus Robust – Word Sense Disambiguation exercise UBC: Eneko Agirre, Oier Lopez de Lacalle, Arantxa Otegi, German Rigau UVA & Irion: Piek Vossen.
1 The Domain-Specific Track at CLEF 2008 Vivien Petras & Stefan Baerisch GESIS Social Science Information Centre, Bonn, Germany Aarhus, Denmark, September.

1 The Domain-Specific Track at CLEF 2008 Vivien Petras & Stefan Baerisch GESIS Social Science Information Centre, Bonn, Germany Aarhus, Denmark, September.

Similar presentations

Ads by Google