Text REtrieval Conference (TREC) The TREC Conferences Ellen Voorhees

Text REtrieval Conference (TREC) TREC Philosophy TREC is a modern example of the Cranfield tradition –system evaluation based on test collections Emphasis on advancing the state of the art from evaluation results –TREC’s primary purpose is not competitive benchmarking –experimental workshop: sometimes experiments fail!

Text REtrieval Conference (TREC) Cranfield at Fifty Evaluation methodology is still valuable… carefully calibrated level of abstraction –has sufficient fidelity to real user tasks to be informative –general enough to be broadly applicable, feasible, relatively inexpensive …but is showing some signs of age size is overwhelming our ability to evaluate new abstractions need to carefully accommodate variability to maintain power

Text REtrieval Conference (TREC) Evaluation Difficulties Variability despite stark abstraction, user effect still dominates Cranfield results Size matters effective pooling has corpus size dependency test collection construction costs depend on number of judgments Model coarseness even slightly different tasks may not be good fit –e.g., legal discovery, video features

Text REtrieval Conference (TREC) TREC 2009 All tracks used some new, large document set Different trade-offs in adapting evaluation strategy tension between evaluating current participants’ ability to do the task and building reusable test collections variety of tasks that are not simple ranked-list retrieval

Text REtrieval Conference (TREC) ClueWeb09 Document Set Snapshot of the WWW in early 2009 crawled by CMU with support from NSF distributed through CMU used in four TREC 2009 tracks: Web, Relevance Feedback, Million Query, and Entity Full corpus about one billion pages and 25 terabytes of text about half is in English Category B English-only subset of about 50 million pages (including Wikipedia) to permit wider participation

Text REtrieval Conference (TREC) TREC 2009 Participants Applied DiscoveryLogik Systems, Inc.University of Applied Science Geneva Beijing Institute of TechnologyMicrosoft Research AsiaUniversity of Arkansas, Little Rock Beijing U. of Posts and Telecommunications Microsoft Research CambridgeUniversity of California, Santa Cruz Cairo Microsoft Innovation CenterMilwaukee School of EngineeringUniversity of Delaware (2) Carnegie Mellon UniversityMugla UniversityUniversity of Glasgow Chinese Academy of Sciences (2)National Institute of Information and Communications Technology University of Illinois, Urbana-Champaign Clearwell Systems, Inc.Northeastern UniversityUniversity of Iowa Clearly Gottlieb Steen & Hamilton, with Backstop LLC Open Text CorporationUniversity of Lugano Dalian University of TechnologyPeking UniversityUniversity of Maryland, College Park Delft University of TechnologyPohang U. of Science & TechnologyUniversity of Massachusetts, Amherst EMC - CMA - R&DPurdue UniversityThe University of Melbourne EquivioQueensland University of TechnologyUniversity of Padova Fondazione Ugo BordoniRMIT UniversityUniversity of Paris Fraunhofer SCAISabir ResearchUniversity of Pittsburgh Fudan UniversitySouth China University of TechnologyUniversity of Twente H5SUNY BuffaloUniversity of Waterloo (2) Heilongjiang Inst. of TechnologyTsinghua UniversityUrsinus College IntegreonUniversidade do PortoYahoo! Research International Inst. of Information Technology, Hyderabad University College DublinYork University (2) Know-CenterUniversity of Alaska, FairbanksZL Technologies, Inc. Lehigh UniversityUniversity of Amsterdam (2)

Text REtrieval Conference (TREC) The TREC Tracks Blog Spam Chemical IR Genomics Novelty QA, Entity Legal Enterprise Terabyte, Million Query Web VLC Video Speech OCR Cross-language Chinese Spanish Interactive, HARD, Feedback Filtering Routing Ad Hoc, Robust Personal documents Retrieval in a domain Answers, not documents Searching corporate repositories Size, efficiency, & web search Beyond text Beyond just English Human-in-the-loop Streamed text Static text

Text REtrieval Conference (TREC) TREC 2010 Blog, Chemical IR, Entity, Legal, Relevance Feedback, Web continuing Million Query merged with Web New “Sessions” track: investigate search behavior over a series of queries (series of length 2 for first running in 2010)

Text REtrieval Conference (TREC) TREC 2011 Track proposals due Monday (Sept 27) New track on searching free text fields of medical records likely