1. Mandarin-English Information (MEI): Investigating Translingual Speech Retrieval Johns Hopkins University Summer Workshop 2000 Presented at the ANLP-NAACL 2000 Embedded Machine Translation Systems Workshop The MEI Team

2. MEI Team Senior Members Students Helen Meng Chinese University of Hong Kong Erika Grams Advanced Analytic Tools Sanjeev Khudanpur Johns Hopkins University Gina-Anne Levow University of Maryland Douglas Oard University of Maryland Patrick Schone US Department of Defense Hsin-Min Wang Academia Sinica, Taiwan Berlin Chen National Taiwan University Wai-Kit Lo Chinese University of Hong Kong Karen Tang Princeton University Jianqiang Wang University of Maryland

3. Outline Audio indexing MEI Project overview Research challenges System architecture Collaboration opportunities

4. Motivation Speech retrieval applications are emerging –e.g., http://speechbot.research.compaq.com source: www.real.com, Feb 2000 Internet-accessible Radio and Television Stations

5. The Big Picture Speech to Speech Translation Translingual Audio Browsing Translingual Audio Search English Query English Audio SelectExamine MEI

6. Related Work TREC Spoken Document Retrieval –Close coupling of recognition and retrieval TREC Cross-Language Retrieval –Close coupling of translation and retrieval TDT-3 Topic Tracking –Coupling recognition, translation and retrieval Using speech recognition transcripts

7. The MEI Project Query by Example English Example Newswire Stories Mandarin Audio Collection Closely couple recognition and translation –For the purpose of retrieval Using English examples, find Mandarin audio

8. Research Challenges Multi-scale audio indexing –Multiple feature sets capture more information Multi-scale translation –Lexicon and pronunciation are complementary Multi-scale retrieval –Combination of evidence can add robustness

9. Multi-scale Mandarin Audio Indexing Initial/Final Preme/Core Final Preme/Toneme /iang/ /ji/ /j/ /ang/ /i//a/ /ng/ /j/

10. Word-scale –Dictionary-based [Levow & Oard 00] –Parallel corpora [Nie 99] –Comparable corpora [Fung 98] Subword-scale [Knight & Graehl 97] –Cross-language phonetic mapping – /bei2 ai4 er3 lan2/ Kosovo (/ke1-sou3-wo4/, /ke1-sou3-fo2/, /ke1-sou3-fu1/, /ke1-sou3-fu2/) Multi-scale Translation

11. Cross-Language Phonetic Mapping Syllabify English spelling –e.g. Jiang Zemin, Shandong Province Map English pronunciation to Mandarin –Convert phonemes to pinyin e.g. /k ow s ax v ow/ to /ke1-suo3-wo4/ –Plan to investigate alternative techniques Rule-based Statistical mapping

12. Multi-scale Retrieval Word-scale exploits lexical knowledge –Enhances precision Subwords can achieve complete coverage –Enhances recall Combination of evidence may be best –If a good merging strategy can be found

13. Multi-scale Retrieval Techniques Subword-scale –Syllable lattice matching [Chen, Wang & Lee 00] –Overlapping syllable n-grams [Meng et al. 99] –Syllable confusion matrix [Meng et al. 99] Word-scale –Structured queries [Pirkola 98] –Structured translation [Sperer & Oard 00]

14. Merging Strategies Loose coupling –Separate retrieval runs –Merge ranked lists [Voorhees 95] Tight coupling [Ng 00] –Unified indexing of words and subwords –Single ranked list

15. Robust Retrieval Multiple causes –Speech recognition errors –Translation ambiguity –Transliteration ambiguity Possible solutions –Weighted n-best indexing [Levow & Oard 00] –Syllable lattice indexing [Chen, Wang & Lee 00] –Syllable confusion expansion [Meng et al. 99] –Structured queries [Pirkola 98] –Document expansion [Levow & Oard 00]

16. System Architecture Overview Word Translation English Example Phonetic Transcription Known Terms Retrieval System Words Syllable n-grams Mandarin Documents Translation Lexicon Corpus Statistics Eval Code Average Precision Relevance Judgments Syllable n-gram Generation

17. The TDT Collections 41 Hours VOA Mandarin Audio 59 Topics 121 Hours Voice of America (VOA) Mandarin Audio APW+NYT English Associated Press (APW) New York Times (NYT) English Newswire 20 Topics Development Test (TDT-2)Evaluation (TDT-3) Mar 98Oct 98Dec 98Jun 98 Jan 98 Four stories per topic in each language –Each reporting on some aspect of one event Story Boundaries Known Condition

18. MEI Project Schedule Six Weeks at Hopkins: DecFebJunAprAug Workshop 00 Planning Meeting First Team Planning Meeting Second Team Planning Meeting

19. Things We Need Ideas –To sharpen our focus Connections –To build a community of interest Resources –To build on what others have done

20. For More Information MEI Project –http://www.glue.umd.edu/~meiweb Translingual Retrieval –http://www.clis.umd.edu/dlrg/clir Speech Retrieval –http://www.clis.umd.edu/dlrg/speech Hopkins Summer Workshop Series –http://www.clsp.jhu.edu/workshops

21. Backup Slides

22. Background: Chinese Many dialects (e.g., Mandarin and Cantonese) –differences in phonetics, vocabularies, syntax… Syllable-based language –~400 base syllables, 4 lexical tones + light tone Syllable structure (CG)V(X) –(CG): onset, optional, consonant+medial glide –V:nuclear vowel –X:coda, glide / alveolar nasal / velar nasal –~ 21 initials, 39 finals

23. Background: Chinese (cont) Characters (written) -> syllables (spoken) Degenerate mapping – /hang2/, /hang4/, /heng2/ or /xing2/ –/fu4 shu4/ (LDC’s CALLHOME lexicon) Tokenization / Segmentation –/zhe4 yi1 wan3 hui4 ru2 chang2 ju3 xing2/

24. Translating the English Example English Example Documents (words) Named entities and unknown words Words found in translation dictionary TranslationTransliteration Mandarin Queries (words and syllable n-grams) Indexed Mandarin Audio (words and syllable n-grams) Information Retrieval Engine Ranked List

25. White-space separated text with named entity tags. Stopping Stemming Phrase Extraction List of translatable words and phrases Terms that have Mandarin translations Named entities terms with no Mandarin translations Detailed Query Processing (1)

26. term translation English-Mandarin translation lexicon Named entity parsing Named entity parsing rules for transliteration phonetic expansion English pronunciation lexicon term translation phonetic expansion xxxxx xxx zzzzzzz ww xx ww z rrrrrr sss ttttt tt ww www eh n t ih t iy t eh k s t Bag of Mandarin terms Bag of English phone sequences Terms that have Mandarin translations Terms with no Mandarin translations Named entities Northern Ireland Detailed Query Processing (2)

27. English phone strings to Mandarin syllables Mandarin syllabification rules ASR Insertion prone words ASR substitution/deletion prone words Retain this term? sa sb sc sd sc se s1 s2 s2 s5 s1 s2 s3 Mandarin pronunciation lexicon Bag of Mandarin terms Syllabic expansion yes xxxxx xxx zzzzzzz xx ww z rrrrrr sss ttttt tt ww www Bag of English phone sequences Two bags of Mandarin syllable sequences Smaller bag of Mandarin terms To trash (or downweight) no Detailed Query Processing (3)

28. Syllable n-gram generation Syllable n-gram generation s0 s1 s1 s2 s2 s3 s3 s4 s3 s5 s5 s6 s5 s7 s7 sa sb sb sc sc sd sc se s1 s2 s2 s3 s2 s5 s5 s1 Syllable n-gram generation xxxxx xxx zzzzzzz xx ww z rrrrrr sss ttttt tt ww www Mandarin pronunciation lexicon syllabic expansion Mandarin syllable sequences from likely recognition errors Mandarin syllable sequences from unknown words Bag of Mandarin lexical terms Three Bags of Mandarin syllable n-grams from different sources Bag of high-confidence Mandarin terms Detailed Query Processing (4)

29. Translating the Mandarin Audio English Example Documents (words) Indexed Mandarin Audio (words) TranslationInformation Retrieval Engine Ranked List Documents in English