The CLEF 2005 Cross-Language Image Retrieval Track Organised by Paul Clough, Henning Müller, Thomas Deselaers, Michael Grubinger, Thomas Lehmann, Jeffery Jensen and William Hersh

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Overview Image Retrieval and CLEF Motivations Tasks in 2005 Ad-hoc retrieval of historic photographs and medical images Automatic annotation of medical images Interactive task Summary and future work

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Image Retrieval and CLEF Cross-language image retrieval Images often accompanied by text (used for retrieval) Began in 2003 as pilot experiment Aims of ImageCLEF Investigate retrieval combining visual features and associated text Promote the exchange of ideas Provide resources for IR evaluation

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Motivations Image retrieval a good application for CLIR Assume images are language-independent Many images have associated text (e.g. captions, metadata, Web page links) CLIR has potential benefits for image vendors and users Image retrieval can be performed using Low-level visual features (e.g. texture, colour and shape) Abstracted features expressed using text Combining both visual and textual approaches

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 ImageCLEF participants from 11 countries Specific domains and tasks Retrieval of historic photographs (St Andrews) Retrieval and annotation of medical images (medImageCLEF and IRMA) Additional co-ordinators William Hersh and Jeffrey Jensen (OHSU) Thomas Lehmann and Thomas Deselaers (Aachen) Michael Grubinger (Melbourne) Links with MUSCLE NoE including pre-CLEF workshop

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Ad-hoc retrieval from historic photographs Paul Clough (University of Sheffield) Michael Grubinger (Victoria University)

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 From: St Andrews Library historic photographic collection イングランドにある灯台の写真 Изображения английских маяков Fotos de faros ingleses Pictures of English lighthouses Kuvia englantilaisista majakoista Bilder von englischen Leuchttürmen صور لمنارات انجليزيه St Andrews image collection

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Topics 28 search tasks (topics) Consist of title, narrative and example images Topics more general than 2004 and more “visual” e.g. waves breaking on beach, dog in sitting position Topics translated by native speakers 8 languages for title & narrative (e.g. German, Spanish, Chinese, Japanese) 25 languages for title (e.g. Russian, Bulgarian, Norwegian, Hebrew, Croatian) 2004 topics and qrels used as training data

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Relevance judgements Staff from Sheffield University were assessors Assessors judged topic pools Top 50 images from all 349 runs Average of 1,376 images per pool 3 assessments per image (inc. topic creator) Ternary relevance judgements Qrels: images judged as relevant/partially relevant by topic creator and at least one other assessor

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Submissions & Results (1) 11 groups (5 new*) CEA* NII* Alicante CUHK* DCU Geneva Indonesia* Miracle NTU Jaen* UNED DimensionType#Runs (%)Avg. MAP LanguageEnglish119 (34%) Non-English230 (66%) Run typeAutomatic349 (100%) Feedback (QE)Yes142 (41%) No207 (59%) ModalityImage4 (1%) Text318 (91%) Text + Image27 (8%) Initial QueryImage4 (1%) Title274 (79%) Narrative6 (2%) Title + Narr57 (16%) Title + Image4 (1%) All4 (1%)0.3953

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Submissions & Results (2) Language#RunsMax. MAPGroupInitial QueryFeedbackModality English CUHKTitle+imgYesText+img Chinese (trad.) NTUTitle+narrYesText+img Spanish (Lat. Am.) Alicante/JaenTitleYesText Dutch Alicante/JaenTitleYesText Visual NTUVisualYesImage German Alicante/JaenTitleYesText Spanish (Euro.) UNEDTitleYesText Portuguese MiracleTitleNoText Greek DCUTitleYesText French JaenTitle+narrYesText Japanese AlicanteTitleYesText Russian DCUTitleYesText Italian MiracleTitleYesText

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Submissions & Results (3) Language#RunsMax. MAPGroupInitial QueryFeedbackModality Chinese (simpl.) AlicanteTitleYesText Indonesian IndonesiaTitleNoText+img Turkish MiracleTitleNoText Swedish JaenTitleNoText Norwegian MiracleTitleNoText Filipino MiracleTitleNoText Polish MiracleTitleNoText Romanian MiracleTitleNoText Bulgarian MiracleTitleNoText Czech MiracleTitleNoText Croatian MiracleTitleNoText Finnish MiracleTitleNoText Hungarian MiracleTitlenoText

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Summary Most groups focused on text retrieval Fewer combined runs than 2004 But still gives highest average MAP Translation main focus for many groups 13 languages have at least 2 groups More use of title & narrative than 2004 As Relevance feedback (QE) improves results Topics still dominated by semantics But typical of searches in this domain

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Ad-hoc medical retrieval task Henning Müller (University Hospitals Geneva) William Hersh, Jeffrey Jensen (OHSU)

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Collection 50,000 medical images 4 sub-collections with heterogeneous annotation Radiographs, photographs, Powerpoint slides and illustrations Mixed languages for annotations (French, German and English) In 2004 only 9,000 images available

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Search topics Topics based on 4 axes Modality (e.g. x-ray, CT, MRI) Anatomic region shown in image (e.g. head, arm) Pathology (disease) shown in image Abnormal visual observation (e.g. enlarged heart) Different types of topic identified from survey Visual (11) – visual approaches only expected to perform well Mixed (11) – text and visual approaches expected to perform well Semantic (3) – visual approaches not expected to perform well Topics consist of annotation in 3 languages and 1-3 query images

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 An example (topic # 20 - mixed) Show me microscopic pathologies of cases with chronic myelogenous leukemia. Zeige mir mikroskopische Pathologiebilder von chronischer Leukämie. Montre-moi des images de la leucémie chronique myélogène.

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Relevance assessments Medical doctors made relevance judgements Only one per topic for money and time constraints Some additional to verify consistency Relevant/partially relevant/non relevant For ranking only relevant vs. non-relevant Image pools created from submissions Top 40 images from 134 runs Average of 892 images per topic to assess

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Submissions 13 groups submitted runs (24 registered) Resources very interesting but lack of manpower 134 runs submitted Several categories for submissions Manual vs. Automatic Data source used Visual/textual/mixed All languages could be used or a single one

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Results (1) Mainly automatic and mixed submissions some further to be classified as manual Large variety of text/visual retrieval approaches Ontology-based Simple tf/idf weighting Manual classification before visual retrieval Query typesAutomaticManual Visual283 Textual141 Mixed862

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Results (2) – highest MAP Query typesAutomaticManual VisualI2Rfus.txt i2r-vk-avg.txt TextualIPALI2R_Tn OHSUmanual.txt MixedIPALI2R_TIan OHSUmanvis.txt 0.160

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Average results per topic type

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Summary Text-only approaches perform better than image-only But some visual systems have high early precision Depends on the topics formulated Visual systems very bad on semantic queries Best overall systems use combined approaches GIFT as a baseline system used by many participants and still best visual completely automatic Few manual runs

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Automatic Annotation Task Thomas Deselaers, Thomas Lehmann (RWTH Aachen University)

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Automatic annotation Goal Compare state-of-the-art classifiers for medical image annotation task Purely visual task Task 9,000 training & 1,000 test medical images from Aachen University Hospital 57 classes identifying modality, body orientation, body region and biological system (IRMA code) e.g. 01: plain radiography, coronal, cranuim, musculosceletal system Classes in English and German and unevenly distributed

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Example of IRMA code Example: Example: radiography, plain, analog, overview radiography, plain, analog, overview coronal, AP, supine coronal, AP, supine abdomen, middle abdomen, middle uropoetic system uropoetic system

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Example Images

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Participants Groups 26 registered 12 submitted runs Runs In total 41 submitted CEA (France) CINDI (Montreal,CA) medGift (Geneva, CH) Infocomm (Singapore, SG) Miracle (Madrid, ES) Umontreal (Montreal, CA) Mt. Holyoke College (Mt. Hol., US) NCTU-DBLAB (TW) NTU (TW) RWTH Aachen CS (Aachen, DE) IRMA Group (Aachen, DE) U Liège (Liège, BE)

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Results... Baseline error rate: 36.8%

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 Conclusions Continued global participation from variety of research communities Improvements in ad-hoc medical task Realistic topics Larger medical image collection Introduction of medical annotation task Overall combining text and visual approaches works well for ad-hoc task

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 ImageCLEF2006 and beyond …

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 ImageCLEF 2006 … New ad-hoc IAPR collection of 25,000 personal photographs Annotations in English, German and Spanish Medical ad-hoc Same data; new topics Medical annotation Larger collection; more fine-grained classification New interactive task Using Flickr.com … more in iCLEF talk

22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005 … and beyond Image annotation task Annotate general images with simple concepts Using the LTU 80,000 Web images (~350 categories) MUSCLE collaboration Create visual queries for ad-hoc task (IAPR) Funding workshop in 2006 All tasks involve cross-language in some way