Phrase Extraction in PB-SMT Ankit K Srivastava NCLT/CNGL Presentation: May 6, 2009

Phrase Extraction | Ankit | 6-May-09 2 About Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote

Phrase Extraction | Ankit | 6-May-09 3 PB-SMT Modeling Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote

Phrase Extraction | Ankit | 6-May-09 4 PB-SMT Process sequence of words as opposed to mere words Segment input, translate input, reorder output Translation model, Language Model, Decoder argmax e p(e|f) = argmax e p(f|e) p(e)

Phrase Extraction | Ankit | 6-May-09 5 Learning Phrase Translations Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote

Phrase Extraction | Ankit | 6-May-09 6 Extraction I Input is sentence-aligned parallel corpora Most approaches use word alignments Extract (learn) phrase pairs Build a phrase translation table

Phrase Extraction | Ankit | 6-May-09 7 Extraction II Get word alignments (src2tgt, tgt2src) Perform grow-diag-final heuristics Extract phrase pairs consistent with the word alignments Non-syntactic phrases :: STR [Koehn et al., ’03]

Phrase Extraction | Ankit | 6-May-09 8 Extraction III Sentence-aligned and word-aligned text Monolingual parsing of both SRC & TGT Align subtrees and extract string pairs Syntactic phrases

Phrase Extraction | Ankit | 6-May-09 9 Extraction IV Parse using constituency parser Phrases are syntactic constituents :: CON [Tinsley et al., ’07] (ROOT (S (NP (NNP Vinken)) (VP (MD will) (VP (VB join) (NP (DT the) (NN board)) (PP (IN as) (NP (DT a) (JJ nonexecutive) (NN director))) (NP (NNP Nov) (CD 29))))))

Phrase Extraction | Ankit | 6-May Extraction V Parse using dependency parser Phrases have head-dependent relationships :: DEP [Hearne et al., ’08] HEADDEPENDENT joinVinken joinwill boardthe joinboard joinas directora directornonexecutive asdirector 29Nov join29

Phrase Extraction | Ankit | 6-May Extraction VI Numerous other phrase extractions Estimate phrase translations directly [Marcu & Wong ’02] Use heuristic other than grow-diag-final Use marker-based chunks [Groves & Way ’05] String-to-String translation models herein

Phrase Extraction | Ankit | 6-May Head Percolation and Phrase Extraction Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote

Phrase Extraction | Ankit | 6-May Percolation I It is straightforward to convert constituency tree to an unlabeled dependency tree [Gaifman ’65] Use head percolation tables to identify head child in a constituency representation[Magerman ’95] Dependency tree is obtained by recursively applying head child and non-head child heuristics [Xia & Palmer ’01]

Phrase Extraction | Ankit | 6-May Percolation II (NP (DT the) (NN board)) NP right NN/NNP/CD/JJ (NP-board (DT the) (NN board)) the is dependent on board

Phrase Extraction | Ankit | 6-May Percolation III (ROOT (S (NP (NNP Vinken)) (VP (MD will) (VP (VB join) (NP (DT the) (NN board)) (PP (IN as) (NP (DT a) (JJ nonexecutive) (NN director))) (NP (NNP Nov) (CD 29)))))) HEADDEPENDENT joinVinken joinwill boardthe joinboard joinas directora directornonexecutive asdirector 29Nov join29 NP right NN / NNP / CD / JJ PP left IN / PP S right VP / S VP left VB / VP INPUT OUTPUT

Phrase Extraction | Ankit | 6-May Percolation IV cf. slide - Extraction III (syntactic phrases) Parse by applying head percolation tables on constituency-annotated trees Align trees, extract surface chunks Phrases have head-dependent relations :: PERC

Phrase Extraction | Ankit | 6-May Tools, Resources, and MT System Performance Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote

Phrase Extraction | Ankit | 6-May System setup I RESOURCE TYPENAMEDETAILS CorporaJOC EUROPARL Chiao et al., ‘06 Koehn, ’05 ParsersBerkeley Parser Syntex Parser Head Percolation Petrov et al., ’06 Bourigault et al.,’05 Xia & Palmer ‘01 Alignment ToolsGIZA++ Phrase Heuristics Tree Aligner Och & Ney ’03 Koehn et al., ‘03 Zhechev ‘09 Lang ModelingSRILM ToolkitStolcke ‘02 DecoderMosesKoehn et al., ‘07 Evaluation ScriptsBLEU NIST METEOR, WER, PER Papineni et al., ’02, Doddington ’02, Banerjee & Lavie ‘05

Phrase Extraction | Ankit | 6-May System setup II All 4 “systems” are run with the same configurations (with MERT tuning) on 2 different datasets They only differ in their phrase tables (# chunks) CORPORATRAINDEVTEST JOC7, EUROPARL100,0001,8892,000 CORPORASTRCONDEPPERC JOC236 K79 K74 K72 K EUROPARL2145 K663 K583K565 K

Phrase Extraction | Ankit | 6-May System setup III SYSTEMBLEUNISTMETEORWERPER On JOC (7K) data On EUROPARL (100K) data STR CON DEP PERC

Phrase Extraction | Ankit | 6-May Analyzing Str, Con, Dep, and Perc Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote Analysis w.r.t. Europarl data only

Phrase Extraction | Ankit | 6-May Analysis I No. of common & unique phrase pairs Maybe we should combine the phrase tables… Phrase Types Common to both Unique in 1 st type Unique in 2 nd type DEP & PERC369K213K195K CON & PERC492K171K72K STR & PERC127K2,018K437K CON & DEP391K271K191K STR & DEP128K2,016K454K STR & CON144K2,000K518K

Phrase Extraction | Ankit | 6-May Analysis II Concatenate phrase tables and re-estimate probabilities 15 different phrase table combinations: ∑4Cr, 1≤r≤4 STR + CON + DEP + PERC UNIBITRIQUAD SSC, SD, SPSCD, SCP, SDPSCDP CCD, CPCDP- DDP-- P---

Phrase Extraction | Ankit | 6-May Analysis III All 15 “systems” are run with the same configurations (with MERT tuning) They only differ in their phrase tables This is combining at “translation model” level

Phrase Extraction | Ankit | 6-May Analysis IV Performance on Europarl

Phrase Extraction | Ankit | 6-May Analysis V REF: Does the commission intend to seek more transparency in this area? S: Will the commission ensure that more than transparency in this respect? C: The commission will the commission ensure greater transparency in this respect? D: The commission will the commission ensure greater transparency in this respect? P: Does the commission intend to ensure greater transparency in this regard? SC: Will the commission ensure that more transparent in this respect? SD: Will the commission ensure that more transparent in this respect? SP: Does the commission intend to take to ensure that more than openness in this regard? CD: The commission will the commission ensure greater transparency in this respect? CP: The commission will the commission ensure greater transparency in this respect? DP: The commission will the commission ensure greater transparency in this respect? SCD: Does the commission intend to take to ensure that more transparent commit? SCP: Does the commission intend to take in this regard to ensure greater transparency? SDP: Does the commission intend to take in this regard to ensure greater transparency? CDP: The commission will the commission ensure greater transparency in this respect? SCDP: Does the commission intend to take to ensure that more transparent suspected?

Phrase Extraction | Ankit | 6-May Analysis VI Which phrases does the decoder use? Decoder trace on S+C+D+P Out of 11,748 phrases: S(5204); C(2441); D(2319); P(2368)

Phrase Extraction | Ankit | 6-May Analysis VII Automatic per-sentence evaluation using TER on testset of 2000 sentences [Snover et al., ’06] C (1120); P (331); D (301); S (248) Manual per-sentence evaluation on a random testset of 100 sentences using pairwise system comparison P=C (27%); P>D (5%); SC>SCP(11%)

Phrase Extraction | Ankit | 6-May Analysis VIII Treat the different phrase table combinations as individual MT systems Perform system combination using MBR-CN framework [Du et al., 2009] This is combining at “system” level SYSTEMBLEUNISTMETEORWERPER STR CON DEP PERC ||MBR|| ||CN||

Phrase Extraction | Ankit | 6-May Analysis IX Using Moses baseline phrases (STR) is essential for coverage. SIZE matters! However, adding any system to STR increases baseline score. Symbiotic! Hence, do not replace STR, but supplement it.

Phrase Extraction | Ankit | 6-May Analysis X CON seems to be the best combination with STR (S+C seems to be the best performing system) Has most common chunks with PERC Does PERC harm a CON system – needs more analysis (bias between CON & PERC)

Phrase Extraction | Ankit | 6-May Analysis XI DEP is different from PERC chunks, despite being equivalent in syntactic representation DEP can be substituted by PERC Difference between knowledge induced from dependency and constituency. A different aligner?

Phrase Extraction | Ankit | 6-May Analysis XII PERC is a unique knowledge source. Is it just a simple case of parser combination? Sometimes, it helps. Needs more work on finding connection with CON / DEP

Phrase Extraction | Ankit | 6-May Customizing Moses for syntax- supplemented phrase tables Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote

Phrase Extraction | Ankit | 6-May Moses customization Incorporating syntax (CON, DEP, PERC) Reordering model Phrase scoring (new features) Decoder Parameters Log-linear combination of T-tables Good phrase translations may be lost by the decoder. How can we ensure they remain intact? M OSES

Phrase Extraction | Ankit | 6-May Work in Progress and Future Plans Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote

Phrase Extraction | Ankit | 6-May Ongoing & future work Scaling (data size) (lang. pair) (lang. dir.) Bias between CON & PERC Combining Phrase pairs Combining Systems Classify performance into sentence types Improve quality of phrase pairs in PBSMT

Phrase Extraction | Ankit | 6-May Endnote… Phrase-based statistical machine translation Methods for phrase extraction Phrase induction via percolated dependencies Experimental setup & evaluation results Other facts & figures Moses customization Ongoing & future work Endnote

Phrase Extraction | Ankit | 6-May Endnote Explored 3 linguistically motivated phrase extractions against Moses phrases Improves baseline. Highest recorded is 10% relative increase in BLEU on 100K Rather than pursuing ONE way, combine options Need more analysis of supplementing phrase table with multiple syntactic T-tables

Phrase Extraction | Ankit | 6-May Thank You!

Phrase Extraction | Ankit | 6-May Phrase Extraction in PB-SMT P hrase-based Statistical Machine Translation (PB-SMT) models – the most widely researched paradigm in MT today – rely heavily on the quality of phrase pairs induced from large amounts of training data. There are numerous methods for extracting these phrase translations from parallel corpora. In this talk I will describe phrase pairs induced from percolated dependencies and contrast them with three pre-existing phrase extractions. I will also present the performance of the individual phrase tables and their combinations in a PB-SMT system. I will then conclude with ongoing experiments and future research directions.

Phrase Extraction | Ankit | 6-May Thanks! Andy Way Patrik Lambert John Tinsley Sylwia Ozdowska Ventisislav Zhechev Sergio Penkale Jinhua Du