What’s happening in iCLEF? (the iCLEF Flickr Challenge) Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED),

Slides:

Advertisements

Similar presentations

Elearning Quality for Learning Repositories in Secondary Education Elearning Quality for Learning Repositories in Secondary Education e-Learning Quality:

Advertisements

DELOS Highlights COSTANTINO THANOS ITALIAN NATIONAL RESEARCH COUNCIL.

The Cost of Authoring with a Knowledge Layer Judy Kay and Lichao Li School of Information Technologies The University of Sydney, Australia.

ECB PowerPoint Presentation Template Page 1 Welcome to Kia Oval ECB Club Strategy Implementation Workshop 25 th October 2012.

Project Proposal.

7 Strategies of Assessment for Learning

Interviews With Individuals  Structured  Semi-structured  Un-structured (open, in-depth) With Groups  Focus groups.

Overview of Collaborative Information Retrieval (CIR) at FIRE 2012 Debasis Ganguly, Johannes Leveling, Gareth Jones School of Computing, CNGL, Dublin City.

The Challenges of Multilingual Search Paul Clough The Information School University of Sheffield ISKO UK conference 8-9 July 2013.

The art and science of measuring people l Reliability l Validity l Operationalizing.

Applying blogs to a language learning context Tríona Hourigan Institute for the Study of Knowledge in Society University of Limerick.

CS CS 5150 Software Engineering Lecture 12 Usability 2.

The art and science of measuring people l Reliability l Validity l Operationalizing.

Web Search – Summer Term 2006 II. Information Retrieval (Basics Cont.) (c) Wolfgang Hürst, Albert-Ludwigs-University.

SE 555 Software Requirements & Specification 1 SE 555 Software Requirements & Specification Prototyping.

With or without users? Julio Gonzalo UNEDhttp://nlp.uned.es.

Damian Gordon.  Summary and Relevance of topic paper  Definition of Usability Testing ◦ Formal vs. Informal methods of testing  Testing Basics ◦ Five.

Welcome to the Wild World of Wikis Ed Considine Boston Public Schools.

Paper Prototyping Source:

Lecturer: Ghadah Aldehim

A few more strategies to explore 4 MAT: This is a complex model of instruction that incorporates brain research, learning styles, and actually mirrors.

Best practices in parliament web sites and the new Eduskunta.fi Mr. Aki Asola, information officer.

ATLAS Outreach & Education News & Collaboration. News Reporting ATLAS progress and results to the world ATLAS Week - 11 Oct 2011S. Goldfarb - ATLAS Outreach.

Evaluation Experiments and Experience from the Perspective of Interactive Information Retrieval Ross Wilkinson Mingfang Wu ICT Centre CSIRO, Australia.

Incorporating Pragmatic Usability Testing Into a Software Test Plan Carla Merrill, Ph.D. Focused Design focuseddesign.com

Final Project Presentation Heath Davis 21:228 Hypertext Hypermedia Systems May 5, 2009.

JULIO GONZALO, VÍCTOR PEINADO, PAUL CLOUGH & JUSSI KARLGREN CLEF 2009, CORFU iCLEF 2009 overview tags : image_search, multilinguality, interactivity, log_analysis,

CLEF 2004 – Interactive Xling Bookmarking, thesaurus, and cooperation in bilingual Q & A Jussi Karlgren – Preben Hansen –

Hao Wu Nov Outline Introduction Related Work Experiment Methods Results Conclusions & Next Steps.

The CLEF 2003 cross language image retrieval task Paul Clough and Mark Sanderson University of Sheffield

Heuristic evaluation Functionality: Visual Design: Efficiency:

Cross-Language Evaluation Forum (CLEF) IST Expected Kick-off Date: August 2001 Carol Peters IEI-CNR, Pisa, Italy Carol Peters: blabla Carol.

Web Page Aesthetics Analysis Paper Main Points. Visual Appeal & Effectiveness Coherence, clarity, balance, innovation, form, size, perspective, layout,

Usability, the User Experience & Interface Design: The Role of Reference July 30, 2013.

Proposal for Term Project J. H. Wang Mar. 2, 2015.

PBA Front-End Programming Universal Usability. Two aspects of usability in web design – Enabling the typical user to accomplish her task as efficiently.

My.umich.edu Partial Integration of Dynamic Services with Visual Design.

Methods and Models Choice of methods for Development of IT related products and systems SVINGSVING Conference held in Gothenburg, Sweden, October 2000.

Instructional Technology & Design Office or The World of Wikis Presented by Rebecca McGuire.

1 ISE 412 Usability Testing Purpose of usability testing:  evaluate users’ experience with the interface  identify specific problems in the interface.

Human Factors in Information Seeking and Use Wooseob Jeong.

CLEF 2008 Final Session CLEF 2008 Workshop, Aarhus, Denmark September 2008.

Elaine Ménard & Margaret Smithglass School of Information Studies McGill University [Canada] July 5 th, 2011 Babel revisited: A taxonomy for ordinary images.

UNED at iCLEF 2008: Analysis of a large log of multilingual image searches in Flickr Victor Peinado, Javier Artiles, Julio Gonzalo and Fernando López-Ostenero.

Article by Dunja Mladenic, Marko Grobelnik, Blaz Fortuna, and Miha Grcar, Chapter 3 in Semantic Knowledge Management: Integrating Ontology Management,

How robust is CLIR? Proposal for a new robust task at CLEF Thomas Mandl Information Science Universität Hildesheim 6 th Workshop.

Internet for Teaching and Learning. Understanding the Web The Web is A collection of publicly accessible pages (web sites) on the Internet All use the.

Data and Evaluation Workgroup 9/10/2015 | 8:30-10:30am| Chinook 115.

CLEF 2007 Workshop Budapest, Hungary, 19–21 September 2007 Nicola Ferro Information Management Systems (IMS) Research Group Department of Information Engineering.

WEB 2.0 PATTERNS Carolina Marin. Content  Introduction  The Participation-Collaboration Pattern  The Collaborative Tagging Pattern.

The Cross Language Image Retrieval Track: ImageCLEF Breakout session discussion.

Patti Duncan DEN Virtual Conference October 25, 2008.

Interface design Multimedia and Web. Today’s Objectives Defined User Interface Design Introduce User-Center Design Design guidelines Check Web accounts.

Secondary Evidence for User Satisfaction With Community Information Systems Gregory B. Newby University of North Carolina at Chapel Hill ASIS Midyear Meeting.

Fact or Opinion? How can one tell? A Teacher’s Guide to a web-based lesson. Created by Deborah Swisher Coker Last revised June 28, 2009.

The CLEF 2005 interactive track (iCLEF) Julio Gonzalo 1, Paul Clough 2 and Alessandro Vallin Departamento de Lenguajes y Sistemas Informáticos, Universidad.

INMA Project 2.0 European Virtual Community InnoNetBook proposal | |

How to write a CRITIQUE Laura Ruel Assistant Professor, UNC-Chapel Hill.

NSU Website Structure By: Debbie Jones, NSU Webmaster 1 NSU Web Services Publication - Author: NSU Webmaster Norfolk State University.

Grant Writing for Digital Projects September 2012 IODE Project Office IODE Project Office Oostende, Belgium Oostende, Belgium Sustainability and.

Blog A Web site that contains dated text entries in reverse chronological order (most recent first) about a particular topic. Blogs serve many purposes.

WP8: Demonstrators (UniCam – Regione Marche)

Proposal for Term Project

Contracting Officer Podcast Slides

Thematic Independent Studies (TIS)

Web Mining Ref:

1 NSU Website Structure By: Debbie Lyn Jones, Information Technology Manager I / Norfolk State University Webmaster NSU Webmaster Publication – Created.

College of Information

Presentation transcript:

What’s happening in iCLEF? (the iCLEF Flickr Challenge) Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED)

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news A piece of good news for CLEF! People have started to hyperlink MT versions of web pages!

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news The original URL

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news The world is in danger

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news The world needs Multilingual Information Access research/solutions …(that do care about users)

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Retrospective: goals for iCLEF 2006 Find realistic scenarios where multilingual search arises naturally  multilingual web communities growing around (mostly) non-textual items (ebay, Flickr)  Explore Web 2.0 “new text” challenges: folksonomies, blogs, wikis, etc. Reduce the cost of entry for new participants.  Provide a default multilingual search interface to Flickr. Improve the cost-effectiveness of the traditional experiment design.  Let participants work on their own experimental designs.

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Flickr

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Flickr advantages Naturally multilingual Web 2.0 challenges (folksonomies) Replicability (unlike general web image search facilities)

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2006 Topics  Ad hoc: find as many photographs of (different) european parliaments as possible.  Creative: find five illustrations for this article about saffron in Italy.  Visual: What is the name of the beach where this crab is lying on? Methodology  Participants must propose their own evaluation measures and experiment design.

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Experiments UNED focused on user’s behaviour  How users deal with native/passive/unknown languages?  Do they actually use CLIR facilities when available? SICS focused on user’s perceptions  Satisfaction (all tasks)  Completeness (creative,ad-hoc)  Quality (creative) Sheffield also measured search effectiveness  How many facets were retrieved (creative, ad-hoc)  Was the image found? (visual)

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Example result: choice of target languages along time.

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Achievements Found a natural CLIR task Flickr works for CLIR testing New perspectives for experiment design. Moved from bilingual to multilingual search problems.  Stick to Flickr in 2007

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Why participation was low? Kept on recruiting users and forcing them to search whatever we asked for.  Experiment design still too costly.  User populations still too small. Cost of entry still too high  Building a Flickr interface / adapting the provided interface was not trivial.  Experiment design too open for newcomers. Schedule collapses with standard tracks.  A problem for CLEF groups potentially interested.

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news How to improve? Kept on recruiting users and forcing them to search whatever we asked for.  Experiment design still too costly.  User populations still too small. Cost of entry still too high  Building a Flickr interface / adapting the provided interface was not trivial.  Experiment design too open for newcomers. Schedule collapses with standard tracks.  A problem for CLEF groups potentially interested.  Move to Flickr/Web users  Make search task a game Provide experiment design  Participants ≈ Searchers  Adjust iCLEF calendar

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2007 (i)Target search task 海 (sea, wave) Find this image Clear goal for the user Clearly defined measures of success (appropriate for a “Hall of Fame” experience) Invokes different searching strategies

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2007 (ii) Evaluation aspects User’s behaviour (Observational/Log Analysis)  Search strategies with native/passive/unknown languages User’s perceptions  Usefulness  Difficulty  Clarity, etc. Search effectiveness  Success rate and its correlation with user profiles, search strategies, etc.

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2007 (iii) CL search interface Single web interface for all experiments. Available for:web interface  Groups willing to recruit users and conduct their own experiments.  Any web user wishing to enter the iCLEF Hall of Fame (by finding more images than anyone else!).  Groups willing to experience CL searching from the user’s perspective: who is the best CLEF team searching cross-language? Full experiments Log analysis

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news iCLEF 2007/2008 schedule End of October: Feedback from you! on  the multilingual search interface  experiment design  evaluation measures Mid-December: Guidelines + interface goes public  Groups start their own experiments.  People can start competing in the search challenge.  Organization starts collecting logs from Flickr users Mid-April:  Flickr/Web user Logs released to interested groups  Thematic session in Chorus Conference on Multimedia Search and Access (hosted by Yahoo!). Paper submission: aligned with CLEF 2008

Retrospective FlickriCLEF 2006Lessons learntiCLEF 2007/8Good news Start testing the interface now! Feedback to