1. Welcome to CLEF 2009 Carol Peters ISTI-CNR, Pisa, Italy

2. Happy 10 th Birthday CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

3. CLEF Objectives  Stimulate the development of multilingual IR systems (for European languages !)  Create a CLIR/MLIA community  Construct publicly available test-suites BY  Conducting annual evaluation campaigns  Designing tracks/tasks to meet emerging needs and to stimulate research in the”right” direction Objective: truly multilingual/multimedia systems CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

4. Evolution of CLEF CLEF 2000 Tracks mono-, bi- & multilingual text doc retrieval (Ad Hoc) mono- and cross-language information on structured scientific data (Domain-Specific) CLEF 2001 New interactive cross-language retrieval (iCLEF) CLEF 2002 New cross-language spoken document retrieval (CL-SR) CLEF 2003 New multiple language question answering (QA@CLEF) cross-language retrieval in image collections (ImageCLEF) CLEF 2005 New multilingual retrieval of Web documents (WebCLEF) cross-language geographical retrieval (GeoCLEF) CLEF 2008 New cross-language video retrieval (VideoCLEF) multilingual information filtering (INFILE@CLEF) CLEF 2009 New intellectual property (CLEF-IP) log file analysis (LogCLEF) large-scale grid experiments (Grid@CLEF)

5. CLEF Tracks: 2000 - 2009

6.  Multilingual textual document retrieval (Ad Hoc)  Interactive cross-language retrieval (iCLEF)  Multiple language question answering (QA@CLEF)  Cross-language retrieval in image collections (ImageCLEF)  Multilingual information filtering (INFILE@CLEF)  Cross-language video retrieval (VideoCLEF)  Intellectual property (CLEF-IP) – New this year  Log file analysis (LogCLEF) – New this year  MorphoChallenge (in collaboration with Pascal NoE) CLEF 2009 Tracks Pilot: Grid Experiments (Grid@CLEF) CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

7. CLEF2009 Coordination  Adaptive Informatics Research Centre, Helsinki University of Technology, Finland  Berlin School of Library and Information Science, Humboldt-Universität zu Berlin  Business Information Systems, U. Applied Sciences Western Switzerland, Sierre, Switzerland  CEA LIST, France  Centre for Evaluation of Human Language & Multimodal Communication Technologies (CELCT), Italy  Centruum vor Wiskunde en Informatica, Amsterdam, NL  Computer Science Department, U. Basque Country, Spain  Computer Vision and Multimedia Lab, U. Geneva, CH  Data Base Research Group, U. Tehran, Iran  Dept. of Computer Science and Information Systems, U. Limerick, Ireland  Dept. of Information Engineering, U. Padua, Italy  Dept. of Information Science, U. Hildesheim, Germany  Dept. of Information Studies, U. Sheffield, UK  Dept. of Medical Informatics and Clinical Epidemiology, Oregon Health and Science U., USA  Dept. Medical Informatics, U. Hospitals and University of Geneva, Switzerland  Department of Medical Informatics, Aachen University of Technology (RWTH), Germany  Evaluations and Language Resources Distribution Agency, Paris, France  GERiiCO, Université de Lille, France  Idiap Research Institute, Switzerland  Information Retrieval Facility (IRF), Austria  Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur (LIMSI), Orsay, France  U. Nacional de Educación a Distancia, Madrid, Spain  Linguateca, Sintef ICT, Norway  Linguistic Modelling Lab., Bulgarian Acad Sci  Matrixware Information Services, Austria  Mediamatics, Delft University of Technology, The Netherlands  Mitre Corporation, USA  NIST, USA  NLE Lab., Universidad Politènica de Valencia, Spain  Research Institute for Artificial Intelligence, Romanian Academy, Romania  Romanian Institute for Computer Science, Romania  Royal Institute of Technology (KTH), Stockholm, Sweden  School of Computing, DCU, Ireland  Swedish Institute of Computer Science, Sweden  TALP Research Center, Universitat Politécnica de Catalunya, Barcelona, Spain. CLEF is coordinated by the Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche, Pisa The following Institutions are contributing to the organisation of the different tracks of the CLEF 2008 campaign: CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

8. CLEF Steering Committee  Maristella Agosti, U.Padove, Italy  Martin Braschler, Zurich, Switzerland  Amedeo Cappelli, ISTI-CNR & CELCT, Italy  Hsin-Hsi Chen, National Taiwan U., Taipei, Taiwan  Khalid Choukri, ELRA/ELDA, Paris, France  Paul Clough, University of Sheffield, UK  Thomas Deselaers, RWTH Aachen University, Germany  Giorgio Di Nunzio, U. Padova, Italy  David A. Evans, Clairvoyance Corporation, USA  Nicola Ferro, U. Padova, Italy  Christian Fluhr, CEA-LIST, Fontenay-aux-Roses, France  Norbert Fuhr, University of Duisburg, Germany  Frederic C. Gey, U.C. Berkeley, USA  Julio Gonzalo, LSI-UNED, Madrid, Spain  Donna Harman, NIST, USA  Gareth Jones, Dublin City University, Ireland  Franciska de Jong, University of Twente, Netherlands  Noriko Kando, NII, Tokyo, Japan  Jussi Karlgren, SICS, Sweden  Michael Kluck, German Institute for International and Security Affairs, Berlin, Germany  Natalia Loukachevitch, Moscow State University, Russia  Bernardo Magnini, ITC-irst, Trento, Italy  Thomas Mandl, U. Hildesheim, Germany  Paul McNamee, Johns Hopkins University, USA  Henning Müller, University & University Hospitals of Geneva, Switzerland  Douglas W. Oard, University of Maryland, USA  Anselmo Peňas, LSI-UNED, Madrid, Spain  Vivien Petras, Humboldt University, Berlin, Germany  Maarten de Rijke, University of Amsterdam, Netherlands  Diana Santos, Linguateca, Sintef, Oslo, Norway  Jacques Savoy, University of Neuchatel, Switzerland  Peter Schäuble, Eurospider Information Technologies, Switzerland  Richard Sutcliffe, University of Limerick, Ireland  Hans Uszkoreit, German Research Center for Artificial Intelligence (DFKI), Germany  Felisa Verdejo, LSI-UNED, Madrid, Spain  José Luis Vicedo, University of Alicante, Spain  Ellen Voorhees, NIST, USA  Christa Womser-Hacker, University of Hildesheim, Germany CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

9. CLEF2009: Track Coordinators  Ad Hoc: Abolfazl AleAhmad, Hadi Amiri, Eneko Agirre, Giorgio Di Nunzio, Nicola Ferro, Nicolas Moreau, Arantxa Otegi, Vivien Petras  iCLEF: Paul Clough, Julio Gonzalo, Jussi Karlgren  QA@CLEF: Iñaki Alegria, Davide Buscaldi, Luís Miguel Cabral, Pere R. Comas, Corina Forascu, Pamela Forner, Olivier Galibert, Danilo Giampiccolo, Nicolas Moreau, Djamel Mostefa, Petya Osenova, Anselmo Peñas, Álvaro Rodrigo, Sophie Rosset, Paolo Rosso, Diana Santos, Richard Sutcliff and Jordi Turmo  ImageCLEF: Brian Bakke, Steven Bedrick, Barbara Caputo, Paul Clough, Peter Dunker, Thomas Deselaers, Thomas Deserno, Ivan Eggel, Mark Oliver Güld, William Hersh, Patric Jensfelt, Charles E. Kahn Jr., Jana Kludas, Jayashree Kalpathy–Cramer, Henning Müller, Stefanie Nowak, Monica Lestari Paramita, Andrzej Pronobis, Saïd Radhouani, Mark Sanderson, Tatiana Tommasi, Theodora Tsikrika and Petra Welter  VideoCLEF: Gareth J.F. Jones, Martha Larson and Eamonn Newman  INFILE: Romaric Besançon, Stéphane Chaudiron, Khalid Choukri, Meriama Laïb, Djamel Mostefa and Ismaïl Timimi  CLEF-IP: Florina Piroi, Giovanna Roda, John Tait and Veronika Zenz  LogCLEF: Maristella Agosti, Giorgio Di Nunzio, Christine Doran, Inderjeet Mani, Thomas Mandl, Julia Maria Schulz and Alexander Yeh  Grid@CLEF: Nicola Ferro and Donna Harman  MorphoChallenge: Graeme W. Blackwood, William Byrne Mikko Kurimo, Ville T. Turunen and Sami Virpioja CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

10. CLEF 2009: Participating Groups CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

11. CLEF: Trend in Participation CLEF2009: Europe = 81(69); N. America = 18(12); Asia = 16(15), S. America = 1(3), Africa = 1(1) CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

12. CLEF2009: Participants per Track  Ad Hoc: 28(26)  iCLEF: 6(6)  QA@CLEF: 25(29)  ImageCLEF: 62 (42)  INFILE: 5(1)  VideoCLEF: 7(5)  MorphoChallenge: 9(6)  CLEF-1P: 15 (new)  LogCLEF: 6 (new)  Grid@CLEF: 2 (pilot) CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

13. Participation by Track CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

14. CLEF 2000 – 2009 Participation per Track CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

15. CLEF Achievements  Creation of very active multidisciplinary international research community  Investigation of core issues in MLIA  development of multiple language processing tools  creation of linguistic resources  implementation of appropriate cross-language retrieval models and algorithms for different tasks and languages ;  Creation of important reusable test collections and resources in diverse media for a large number of European languages  Significant and quantifiable improvements in the performance of MLIA systems CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

16. Test Collections 2000  News documents in 4 languages  GIRT German Social Science database 2009  A subset of CLEF multilingual corpus of news documents (Ad Hoc WSD-Robust)  Hamshahri Persian newspaper corpus (Ad Hoc Persian task)  TEL Library catalog records plus log files (Ad Hoc TEL task and LogCLEF)  Flickr web-based image database (iCLEF)  ResPubliQA document collection, (QAatCLEF: ResPubliQA)  Transcripts of European parliamentary sessions & French news broadcasts (QAST)  BELGAPICTURE image collection (ImageCLEFPhoto)  Multilingual collections of Wikipedia documents and images (ImageCLEFwiki)  Articles and images from Radiology and Radiography; IRMA collection for medical  image annotation (ImageCLEFmed and medAnnotation)  Dutch and English documentary television programs (VideoCLEF)  Agence France Press (AFPnewswire stories in Arabic, French and English (INFILE)  Patent documents from the European Patent Office (CLEF-IP)

17. Outline of Talk Why, How, What DIRECT@work in CLEF CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

18. CLEF Publications  9 CLEF Post-workshop Proceedings, Springer LNCS  Nearly 1000 Working Notes Reports on-line  Google Scholar > 15,000 publications with CLEF – cross-language information retrieval CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

19. CLEF 2008 Proceedings Evaluating Systems for Multilingual and Multimodal Information Access Lecture Notes in Computer Science Lecture Notes in Computer Science 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17- 19, 2008, Revised Selected Papers Series: Lecture Notes in Computer Science, Vol. 5706Lecture Notes in Computer Science Peters, C.; Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, Th.; Peñas, A.; Petras, V (Eds.) CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece Many, many thanks to Danilo Giampiccolo, CELCT

20. CLEF & TrebleCLEF  CLEF 2008 & 2009 is an activity of the TrebleCLEF Coordination Action under the Seventh Framework Programme of the European Commission.  TrebleCLEF organises a set of dissemination activities in the multilingual information access field. CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

21. Treble-CLEF The CLEF research results have led to development of a new generation of multilingual retrieval system prototypes BUT lack of technology transfer Treble-CLEF extends the CLEF activity by:  continuing to promote MLIA R&D via evaluation campaigns;  providing a consistent training activity: tutorials, workshops, summer school;  producing best practice guidelines for system implementation;  providing resources to encourage the multilingual system development. www.trebleclef.eu CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

22. Approach  Evaluation  test collections and laboratory evaluation  user evaluation  log analysis  Best Practices & Guidelines  system-oriented aspects of MLIA applications  collaborative user studies  user-oriented aspects of MLIA interfaces  Dissemination and Training  tutorials  workshops  summer school CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

23. Treble-CLEF Events  Workshop on Novel Methodologies for Evaluation in Information Retrieval, ECIR’08, Glasgow, Scotland  Workshop on Best Practices for the Development of Multilingual Information Access Systems, Segovia, Spain, June 08  Workshop on Best Practices for System Developers: Bringing Multilingual Information Access to Operational Systems, Winterthur, Switzerland, October 2008  Workshop on Best Practices in Query Log Analysis, London, May 2009  Summer School on Multilingual information Access, Pisa, June 2009  MLIA Technology Day – Dissemination of Results of Best Practices Workshops, 8 December 2009 CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

24. CLEF 2010 Can CLEF continue without external funding for the central coordination? ?????? Participate in the discussion on Friday afternoon CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

25. Before I Forget 1.Posters: Put up between 8.00 & 13.30 Thursday morning  Pool Room 2: ImageCLEF Posters  Pool Room 1: All other Posters & Demos 2.USB Stick  ECDL Conference Proceedings  All Workshop Proceedings 3.Social Dinner - Tonight  Hotel restaurant – 8 pm (Badge)

26. CLEF 2002 Rome CLEF 2006 Alicante CLEF 2005 Vienna CLEF 2004 Bath CLEF 2003 Trondheim CLEF 2008 Aarhus CLEF 2001 Darmstadt CLEF 2007 Budapest CLEF 2000 Lisbon CLEF2009 Corfu CLEF2010 ????? Come to CLEF – and see Europe!

27. CLEF 2009 Thank you for your attention and ENJOY THE Workshop ! CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008