1. Introduction to IBM Model 1&2 Alignment
钱炘祺

2. machine translation model
French 𝑓  English 𝑒
𝑝 𝑒 𝑓 = 𝑝 𝑒,𝑓 𝑝 𝑓 = 𝑝 𝑒 𝑝 𝑓 𝑒 𝑒 𝑝 𝑒 𝑝 𝑓 𝑒
𝑒 ∗ = arg max 𝑒 𝑝 𝑒 𝑓 = arg max 𝑒 𝑝 𝑒 𝑝 𝑓 𝑒
𝑝 𝑒 the language model
𝑝 𝑓 𝑒 the translation model (difficult)

3. why alignment e (𝑙 words) = And the program has been implemented
f (𝑚 words)= Le programme a ete mis en application
𝑝 𝑓 𝑒,𝑚 is difficult
with the help of alignment: 𝑝 𝑓 𝑒,𝑚 = 𝑎∈𝒜 𝑝 𝑓,𝑎 𝑒,𝑚

4. alignment e (𝑙 words) = And the program has been implemented
f (𝑚 words)= Le programme a ete mis en application
alignment = {2,3,4,5,6,6,6}
# of all possible alignments = 1 𝑙+1 𝑚

5. alignment e (𝑙 words) = And the program has been implemented
f (𝑚 words) = __ _______ __ __ ___ ___ ______ programme
𝑝 𝑓 𝑒,𝑚 = 𝑎∈𝒜 𝑝 𝑓,𝑎 𝑒,𝑚
𝑝 𝑓,𝑎 𝑒,𝑚 =𝑝 𝑎 𝑒,𝑚 𝑝 𝑓 𝑎,𝑒,𝑚
𝑝 𝑓 𝑒,𝑚 = 𝑎∈𝒜 𝑝 𝑎 𝑒,𝑚 𝑝 𝑓 𝑎,𝑒,𝑚

6. most likely alignment 𝑝 𝑓,𝑎 𝑒,𝑚 =𝑝 𝑎 𝑒,𝑚 𝑝 𝑓 𝑎,𝑒,𝑚
𝑝 𝑓 𝑒,𝑚 = 𝑎∈𝒜 𝑝 𝑎 𝑒,𝑚 𝑝 𝑓 𝑎,𝑒,𝑚
𝑝 𝑎 𝑓,𝑒,𝑚 = 𝑝 𝑓,𝑎 𝑒,𝑚 𝑎∈𝒜 𝑓,𝑎 𝑒,𝑚 = 𝑝 𝑓,𝑎 𝑒,𝑚 𝑝 𝑓 𝑒,𝑚
𝑎 ∗ = arg max 𝑎 𝑝 𝑎 𝑓,𝑒,𝑚

7. alignment example French English Alignment
le conseil a rendu son avis, et nous devons à présent adopter un nouvel avis sur la base de la première position.
English
the council has stated its position, and now, on the basis of the first position, we again have to give our opinion.
Alignment
the/le council/conseil has/à stated/rendu its/son position/avis ,/, and/et now/présent ,/, on/sur the/le basis/base of/de the/la first/première position/position ,/NULL we/nous again/NULL have/devons to/a give/adopter our/nouvel opinion/avis ./.

8. IBM Model 1 e (𝑙 words) = And the program has been implemented
f (𝑚 words) = __ _______ __ __ ___ ___ ______
𝑝 𝑓,𝑎 𝑒,𝑚 =𝑝 𝑎 𝑒,𝑚 𝑝 𝑓 𝑎,𝑒,𝑚
all alignments are equally likely: 𝑝 𝑎 𝑒,𝑚 = 1 𝑙+1 𝑚
there are 𝑙+1 𝑚 possible values for 𝑎

9. IBM Model 1 e (𝑙 words) = And the program has been implemented
f (𝑚 words) = __ _______ __ __ ___ ___ ______ la programme a ete mis en application
𝑝 𝑓,𝑎 𝑒,𝑚 =𝑝 𝑎 𝑒,𝑚 𝑝 𝑓 𝑎,𝑒,𝑚
all alignments are equally likely: 𝑝 𝑓 𝑎,𝑒,𝑚 = 𝑗=1 𝑚 𝑡 𝑓 𝑗 𝑒 𝑎 𝑗
𝑝(each aligned term translation)

10. IBM Model 1
estimate 𝑡 𝑓 𝑒
train from parallel corpus

11. IBM Model 2
probability that 𝑗'th French word ( 𝑓 𝑗 ) align to 𝑖'th English word ( 𝑒 𝑖 ), given length 𝑙 and 𝑚
𝑞 𝑖 𝑗,𝑙,𝑚
𝑝 𝑎 𝑒,𝑚 = 𝑗=1 𝑚 𝑞 𝑎 𝑗 𝑗,𝑙,𝑚 ≠ 1 𝑙+1 𝑚
𝑝 𝑓,𝑎 𝑒,𝑚 =𝑝 𝑎 𝑒,𝑚 𝑝 𝑓 𝑎,𝑒,𝑚 = 𝑗=1 𝑚 𝑞 𝑎 𝑗 𝑗,𝑙,𝑚 𝑡 𝑓 𝑗 𝑒 𝑎 𝑗

12. IBM Model 2 e (𝑙 words) = And the program has been implemented
f (𝑚 words) = __ _______ __ __ ___ ___ ______ programme
𝑝 𝑎 𝑒,𝑚 =𝑝 𝑎 𝑒,7 =𝑞 2|1,6,7 ∗𝑞 3|2,6,7 ∗𝑞 4|3,6, ∗𝑞 5|4,6,7 ∗𝑞 6|5,6,7 ∗𝑞 6|6,6, ∗𝑞 6|7,6,7
𝑝 𝑓,𝑎 𝑒,𝑚 = 𝑗=1 𝑚 𝑞 𝑎 𝑗 𝑗,𝑙,𝑚 𝑡 𝑓 𝑗 𝑒 𝑎 𝑗

13. IBM Model 2 e (𝑙 words) = (NULL) And the program has been implemented
f (𝑚 words)= Le programme a ete mis en application
𝑎 𝑗 = arg max 𝑎∈ 0…𝑙 𝑞 𝑎 𝑗,𝑙,𝑚 ∗𝑡( 𝑓 𝑗 | 𝑒 𝑎 )
NULL: 𝑞 0 3,6,7 ∗𝑡(𝑎|𝑁𝑈𝐿𝐿)
And: 𝑞 1 3,6,7 ∗𝑡(𝑎|𝐴𝑛𝑑)
the: 𝑞 2 3,6,7 ∗𝑡(𝑎|𝑡ℎ𝑒)
program: 𝑞 3 3,6,7 ∗𝑡(𝑎|𝑝𝑟𝑜𝑔𝑟𝑎𝑚)
has: 𝑞 4 3,6,7 ∗𝑡(𝑎|ℎ𝑎𝑠)
been: 𝑞 5 3,6,7 ∗𝑡(𝑎|𝑏𝑒𝑒𝑛)
implemented: 𝑞 6 3,6,7 ∗𝑡(𝑎|𝑖𝑚𝑝𝑙𝑒𝑚𝑒𝑛𝑡𝑒𝑑)

14. EM for IBM Model 2 input output: challenge: a set of bitext 𝑒 𝑘 , 𝑓 𝑘
𝑒 = And the program has been implemented
𝑓 = Le programme a ete mis en application
output:
𝑡 𝑓 𝑒 & 𝑞 𝑗 𝑖,𝑙,𝑚
challenge:
do not have alignments on training data

15. EM for IBM Model 2 if alignments are observed
estimate 𝑡 𝑓 𝑒 & 𝑞 𝑗 𝑖,𝑙,𝑚 by counting
𝑡 𝑀𝐿 𝑓 𝑒 = 𝐶𝑜𝑢𝑛𝑡 𝑒,𝑓 𝐶𝑜𝑢𝑛𝑡 𝑒
𝑞 𝑀𝐿 𝑗 𝑖,𝑙,𝑚 = 𝐶𝑜𝑢𝑛𝑡 𝑗 𝑖,𝑙,𝑚 𝐶𝑜𝑢𝑛𝑡 𝑖,𝑙,𝑚

16. EM for IBM Model 2
For 𝑠=1…𝑆 Set all counts 𝑐 … =0 For 𝑘=1…𝑛 For 𝑖=1… 𝑚 𝑘 , For 𝑗=0… 𝑙 𝑘 𝑐 𝑒 𝑗 𝑘 , 𝑓 𝑖 𝑘 ←𝑐 𝑒 𝑗 𝑘 , 𝑓 𝑖 𝑘 +𝛿 𝑘, 𝑖, 𝑗 𝑐 𝑒 𝑗 𝑘 ←𝑐 𝑒 𝑗 𝑘 +𝛿 𝑘, 𝑖, 𝑗 𝑐 𝑗 𝑖, 𝑙, 𝑚 ←𝑐 𝑗 𝑖, 𝑙, 𝑚 +𝛿 𝑘, 𝑖, 𝑗 𝑐 𝑖, 𝑙, 𝑚 ←𝑐 𝑖, 𝑙, 𝑚 +𝛿 𝑘, 𝑖, 𝑗 Recalculate the parameters: 𝑡 𝑓 𝑒 = 𝑐 𝑒, 𝑓 𝑐 𝑒 𝑞 𝑗 𝑖, 𝑙, 𝑚 = 𝑐 𝑗 𝑖, 𝑙, 𝑚 𝑐 𝑖, 𝑙, 𝑚

17. EM for IBM Model 2
e (𝑙 words) = (NULL) And the program has been implemented
f (𝑚 words)= Le programme a ete mis en application
if alignments are observed
𝛿 𝑘,𝑖,𝑗 =1 𝑖𝑓 𝑎 𝑖 𝑘 =𝑗
do not have alignments on training data
𝛿 𝑘,𝑖,𝑗 = 𝑞 𝑗 𝑖, 𝑙 𝑘 , 𝑚 𝑘 𝑡 𝑓 𝑖 𝑘 𝑒 𝑗 𝑘 𝑗=0 𝑙 𝑘 𝑞 𝑗 𝑖, 𝑙 𝑘 , 𝑚 𝑘 𝑡 𝑓 𝑖 𝑘 𝑒 𝑗 𝑘
𝛿 𝑘,𝑖,𝑗 =𝑝 𝑎 𝑖 𝑘 =𝑗 𝑒 𝑘 , 𝑓 𝑘 ;𝑡,𝑞