Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology 1 Using the Web for Automated Translation Extraction in Cross- Language Information Retrieval Advisor : Dr. Hsu Presenter : Zih-Hui Lin Author :Ying Zhang and Phil Vines

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 2  Motivation  Objective  Previous work  Methodology  Experiments and results  Conclusions Outline

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 3 Motivation  One of the major remaining reasons that CLIR does not perform as well as monolingual retrieval is the presence of out of vocabulary (OOV) terms. ─ it will not be recognized, and segmented into either smaller sequences of characters or individual characters ─ 北野武 →(north limit military)  Previous work has either relied on manual intervention or has only been partially successful in solving this problem.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 4 Objective  We propose a segmentation free method which can be applied to both Chinese-English and English-Chinese CLIR, correctly extracting translations of OOV terms from the Web automatically, and thus is a significant improvement on earlier work

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 5 English translation extraction in Chinese-English CLIR  Chinese OOV term detection ─ 北野武 (north limit military) → P value given by the HMM will be very low if P value < P min → contains OOV terms  web text extraction ─ we extract strings that contain the Chinese query terms and some English text from the Web.  collection of co-occurrence statistics,  translation selection. search for longest Chinese substring C t :search for the English term e t with the highest frequency: 1. |C targets | = max(|C ij |). 2. f(e t, C t ) = max(f(e i,C targets )). 3. Add (C t, e t ) into the translation dictionary. 1.f(e targets ) = max(f(e i )). 2.f(e t’,C t’ ) = max(f(e targets,C ij )). 3. if C t’ ≠ C t and e t’ ≠ e t, add (C t’, e t’ ) into the translation dictionary. 北野武北野武（ Kitano Takeshi ） c 4 c 5 c 6 e 1 導演北野武 （ Kitano Takeshi ） c 2 c 3 c 4 c 5 c 6 e 1

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 6 Chinese translation extraction in English-Chinese CLIR  Extraction of web text ─ use Google to fetch the top100 Chinese documents with the English OOV term e oov as the query.  Collection of co-occurrence statistics ─ ─ accumulate the frequency f oov. ─ considering all substrings in S left and S right, and collecting the frequency f n and the length |s n | of each Chinese substring. ─  Translation selection ─ exclude any substring that already in the translation dictionary doesn’t occur in the document collection 與區域貿易 ……… 兩岸經貿關係（ Canada and Cross Straits Economic Relations ）三組發表九 ……… 英茂、加 S left ( 包含 20 個字 )S right ( 包含 20 個字 ) e oov Longest length Highest frequency

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 7 Experiments and results Chinese-English CLIREnglish-Chinese CLIR 30 10

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 8 Introduction  When translating from Chinese to English, a standard first step is to segment the text into words based on an existing segmentation dictionary.  However where an OOV term occurs, it will not be recognized, and segmented into either smaller sequences of characters or individual characters.  We propose a segmentation free method based on frequency and length analysis and corpus-based disambiguation

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 9 Previous work  Dictionary-based translation schemes need to address three major issues ─ phrase identification and translation ex. non proliferation treaty and cross straits. ─ translation ambiguity using techniques such as term co-occurrence, mutual information or language modeling. ─ out of vocabulary (OOV) terms. ex. Dioxin

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 10 Previous work- Existing approaches to the OOV problem  Depending on the language, it may be possible to deduce appropriate transliterated translations automatically. ─ that they successfully applied in English-Arabic CLIR.  However the issue is more difficult in Chinese as many characters have the same sound, and many English syllables do not have equivalent sounds in Chinese, meaning that selecting the correct characters to represent a transliterated word can be problematic. ─ cross straits( 兩岸 ) 、北野武 (north limit military)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 11 Previous work- Segmentation free translation extraction  It is common to find a small amount of English text in Chinese web documents, but extremely rare to find Chinese text in English web documents.  We therefore rely on Chinese web documents to extract translations in both directions.  The problem is that the Chinese OOV term we are looking for is currently unknown, and thus we have no information about how it should be segmented. ─ In previous work, this problem was overcome by manual intervention to provide appropriate segmentation.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 12 Experiments and results  Chinese-English CLIR ─ retrieving English documents using Chinese queries

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 13 Experiments and results (cont.)  English-Chinese CLIR ─ retrieving Chinese documents using English queries. The aim of our work is to find appropriate Chinese translations of English OOV terms

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 14 Conclusions  We have also described improved ways to extract the translation of OOV terms from the Web in a way that does not rely on prior segmentation.  Although the Web is constantly changing, we were able to find most OOV terms, many of which related to news events up to 10 years ago.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 15 My opinion Advantage: Segmentation free translation extraction Disadvantage: Apply: 線上翻譯 ………..