Sheffield at ImageCLEF 2003 Paul Clough and Mark Sanderson

Aims 1.To build a CLIR system with little knowledge of CLIR 2.To evaluate translation quality 3.To correlate retrieval performance and translation quality

A rudimentary CLIR system CL image retrieval based on matching translated queries to captions (no other enhancements) Used Systran as a “black-box” translation module Images ranked by BM25 score and “ANDing” operator Translated ImageCLEF queries in Chinese, Dutch, Spanish, Italian, French and German into English

How good is retrieval? Language Mean Average Precision % of monolingual Monolingual Chinese % Dutch % Italian % Spanish % German % French % Baseline results for CL image retrieval using MT

Translation quality Can assess quality of MT output based on Adequacy – how well the translation conveys meaning Fluency – how well the translation presents its content Informativeness – how well the content is understood Measuring adequacy Manually – ImageCLEF titles in source language and the Systran English output Automatically – ImageCLEF English titles and the Systran English output

Manual assessment (1) Translators assessed adequacy using a 5- point scale 5 (very good) to 1 (very poor) Translators asked to rate quality on adequacy for retrieval All 50 ImageCLEF topic titles rated Problems with assessor variability between languages

Manual assessment (2) Avg manual score % topics with manual score of “very good” % topics with manual score of “very poor” Chinese3.3428%14% Dutch3.3230%8% German3.6444%8% French3.3840%24% Italian3.6550%12% Spanish3.6434%6%

Example translations LanguageQuery with low average score Query with high average score EnglishDogs rounding up sheepRuined castles in England ItalianDogs that assemble sheep Ruins of castles in England GermanDogs with sheep hatsCastle ruins in England DutchDogs which sheep bejeendrijven Ruin of castles in the United Kingdom FrenchDogs gathering of the preois Castle of ruins in England SpanishDogs urging on ewesCastle of ruins in England ChineseCatches up with the sheep of the dog Becomes the ruins of the English castle

Automatic assessment (1) Used NIST’s mteval (version 9) Computed co-occurrence of words Reference translation is ImageCLEF English title Test translation is Systran English title Assumption: reference translation is best English equivalent of the non-English title

Automatic assessment (2) Avg NIST score Man-NIST correlation (Spearman’s rho) % topics with NIST score of 0 Chinese *38% Dutch *12% German *10% French *8% Italian % Spanish *10% Italian and Chinese rated highly manually but not automatically – semantic equivalence *significant at p<0.01

Retrieval and translation Language MAP-manual correlation MAP-auto correlation Chinese0.472*0.384* Dutch0.412*0.426* Italian0.394*0.378* Spanish-0.061?0.462* German0.503*0.324* French0.460*0.456* Generally high MAP-manual/auto correlation *significant at p<0.01

Summary Competitive CL image retrieval with no knowledge of CLIR We have shown how translation quality can be assessed using manual and automatic methods In general retrieval performance using captions is influenced by translation quality The challenge is to improve CL image retrieval over a baseline approach using Systran