1. Translation Error Rate Metric
Reference: “the Iraqi weapons are to be handed over to the army within two weeks”
MT output: “in two weeks Iraq’s weapons will give army”
March 10, 2005
METEOR

2. Step-1: Remove Stop Words
the Iraqi weapons are to be handed over to the army within two weeks
in two weeks Iraq’s weapons will give army
March 10, 2005
METEOR

3. Step-2: Word Align with Matcher
the Iraqi weapons are to be handed over to the army within two weeks
in two weeks Iraq’s weapons will give army
March 10, 2005
METEOR

4. Step-3: Count Flips (Reorder)
the Iraqi weapons are to be handed over to the army within two weeks
in two weeks Iraq’s weapons will give army
Reordered:
in Iraq’s weapons will give army two weeks
Flip count = 4
March 10, 2005
METEOR

5. Step-4: Count Insertions/Deletions/Substitutions
the Iraqi weapons are to be handed over to the army within two weeks
in two weeks Iraq’s weapons will give army
Reordered:
in Iraq’s weapons will give army two weeks
Flip count = 4
Substitutions: 2
Insertions (from Ref): 5
Deletions (from MT): 1
March 10, 2005
METEOR

6. Step-5: Score
the Iraqi weapons are to be handed over to the army within two weeks
in two weeks Iraq’s weapons will give army
Reordered:
in Iraq’s weapons will give army two weeks
Flip count = 4
Substitutions: 2
Insertions (from Ref): 5
Deletions (from MT): 1
TER = flips + subst + ins + del / ref length = /12 = 12/12 = 1.0
March 10, 2005
METEOR

7. Another Example
Ref: north korea is prepared to allow washington to verify that it does not manufacture nuclear weapons
MT: North Korea prepared to allow Washington to the verification of that is to manufacture nuclear weapons
Flips: 0
Substitutions: 3
Insertions: 1
Deletions: 0
TER Score = /16 = 4/16 = 0.25
March 10, 2005
METEOR