1 Discussion Class 5 TREC. 2 Discussion Classes Format: Questions. Ask a member of the class to answer. Provide opportunity for others to comment. When.

Slides:

Advertisements

Similar presentations

1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.

Advertisements

Developing and Evaluating a Query Recommendation Feature to Assist Users with Online Information Seeking & Retrieval With graduate students: Karl Gyllstrom,

1 Retrieval Performance Evaluation Modern Information Retrieval by R. Baeza-Yates and B. Ribeiro-Neto Addison-Wesley, (Chapter 3)

1 Discussion Class 3 The Porter Stemmer. 2 Course Administration No class on Thursday.

Evaluating Search Engine

1 Discussion Class 2 A Vector Space Model for Automated Indexing.

1 Discussion Class 12 Medical Subject Headings (MeSH) and Unified Medical Language System (UML)

1 Discussion Class 11 Click through Data as Implicit Feedback.

1 CS 430: Information Discovery Lecture 10 Cranfield and TREC.

Modern Information Retrieval

1 CS 430 / INFO 430 Information Retrieval Lecture 2 Searching Full Text 2.

1 CS 430 / INFO 430 Information Retrieval Lecture 11 Evaluation of Retrieval Effectiveness 2.

1 Discussion Class 4 Latent Semantic Indexing. 2 Discussion Classes Format: Question Ask a member of the class to answer. Provide opportunity for others.

1 Discussion Class 10 Informedia. 2 Discussion Classes Format: Question Ask a member of the class to answer. Provide opportunity for others to comment.

1 CS 430 / INFO 430 Information Retrieval Lecture 11 Evaluation of Retrieval Effectiveness 2.

Reference Collections: Task Characteristics. TREC Collection Text REtrieval Conference (TREC) –sponsored by NIST and DARPA (1992-?) Comparing approaches.

©2007 H5 Simultaneous Achievement of high Precision and high Recall through Socio-Technical Information Retrieval Robert S. Bauer, Teresa Jade

SIMS 202 Information Organization and Retrieval Prof. Marti Hearst and Prof. Ray Larson UC Berkeley SIMS Tues/Thurs 9:30-11:00am Fall 2000.

1 Discussion Class 12 User Interfaces and Visualization.

1 Discussion Class 3 Inverse Document Frequency. 2 Discussion Classes Format: Questions. Ask a member of the class to answer. Provide opportunity for.

1 Discussion Class 2 A Vector Space Model for Automated Indexing.

1 Discussion Class 6 Crawling the Web. 2 Discussion Classes Format: Questions. Ask a member of the class to answer. Provide opportunity for others to.

1 Discussion Class 8 The Google File System. 2 Discussion Classes Format: Question Ask a member of the class to answer. Provide opportunity for others.

1 Final Discussion Class User Interfaces. 2 Discussion Classes Format: Question Ask a member of the class to answer Provide opportunity for others to.

Maryam Karimzadehgan (U. Illinois Urbana-Champaign)*, Ryen White (MSR), Matthew Richardson (MSR) Presented by Ryen White Microsoft Research * MSR Intern,

1 Discussion Class 1 Three Information Retrieval Systems.

ISP 433/633 Week 6 IR Evaluation. Why Evaluate? Determine if the system is desirable Make comparative assessments.

CS580: Building Web Based Information Systems Roger Alexander & Adele Howe The purpose of the course is to teach theory and practice underlying the construction.

Evaluation Information retrieval Web. Purposes of Evaluation System Performance Evaluation efficiency of data structures and methods operational profile.

Search and Retrieval: Relevance and Evaluation Prof. Marti Hearst SIMS 202, Lecture 20.

Information Retrieval and Web Search IR Evaluation and IR Standard Text Collections.

Physics CPD Presentation Programme Welcome and outline for event Overview of Internal Assessment in Physics Assessment of Outcome 1 Workshop 1, assessing.

Philosophy of IR Evaluation Ellen Voorhees. NIST Evaluation: How well does system meet information need? System evaluation: how good are document rankings?

Project Guidelines. Introduction Introduction should include support/justification “why” the research should be done. The focus is on the dependent variable.

How do I begin?. Read the assignment carefully as soon as it is given to you. Ask your teacher to explain any unclear details to you. UNDERSTAND THE ASSIGNMENT.

Evaluation INST 734 Module 5 Doug Oard. Agenda Evaluation fundamentals  Test collections: evaluating sets Test collections: evaluating rankings Interleaving.

1 CS 501 Spring 2002 CS 501: Software Engineering Lecture 9 Techniques for Requirements Definition and Specification I.

1 CS 501 Spring 2002 CS 501: Software Engineering Lecture 23 Reliability III.

1 Discussion Class 9 Thesaurus Construction. 2 Discussion Classes Format: Question Ask a member of the class to answer Provide opportunity for others.

Xiaoying Gao Computer Science Victoria University of Wellington Intelligent Agents COMP 423.

1 Discussion Class 8 MARC. 2 Discussion Classes Format: Question Ask a member of the class to answer. Provide opportunity for others to comment. When.

Chapter 8 Evaluating Search Engine. Evaluation n Evaluation is key to building effective and efficient search engines  Measurement usually carried out.

Measuring How Good Your Search Engine Is. *. Information System Evaluation l Before 1993 evaluations were done using a few small, well-known corpora of.

WERST – Methodology Group

1 CS 430: Information Discovery Lecture 8 Evaluation of Retrieval Effectiveness II.

1 13/05/07 1/20 LIST – DTSI – Interfaces, Cognitics and Virtual Reality Unit The INFILE project: a crosslingual filtering systems evaluation campaign Romaric.

Topic by Topic Performance of Information Retrieval Systems Walter Liggett National Institute of Standards and Technology TREC-7 (1999)

1 Discussion Class 1 Examples of Digital Libraries and Electronic Publications.

1 CS 430 / INFO 430 Information Retrieval Lecture 9 Evaluation of Retrieval Effectiveness 2.

Evaluation of Information Retrieval Systems Xiangming Mu.

1 Discussion Class 1 Three Information Retrieval Systems.

Information Retrieval Quality of a Search Engine.

1 CS 430: Information Discovery Lecture 11 Cranfield and TREC.

Xiaoying Gao Computer Science Victoria University of Wellington COMP307 NLP 4 Information Retrieval.

Relevance Feedback Prof. Marti Hearst SIMS 202, Lecture 24.

1 Discussion Class 1 Inverted Files. 2 Discussion Classes Format: Question Ask a member of the class to answer Provide opportunity for others to comment.

1 Discussion Class 10 Thesaurus Construction. 2 Discussion Classes Format: Question Ask a member of the class to answer Provide opportunity for others.

Islamic University Nursing College.  A literature review involves the systematic identification, location, search, and summary of written materials that.

Chapter 20 Asking Questions, Finding Sources. Characteristics of a Good Research Paper Poses an interesting question and significant problem Responds.

1 Discussion Class 2 A Vector Space Model for Automated Indexing.

Report writing skills A Trade union training on research methodology, TMLC, Kisumu, Kenya 6-10 December 2010 Presentation by Mohammed Mwamadzingo,

1 INFILE - INformation FILtering Evaluation Evaluation of adaptive filtering systems for business intelligence and technology watch Towards real use conditions.

Evaluation Anisio Lacerda.

Discussion Class 7 Lucene.

Relevance Feedback and Query Modification

Discussion Class 3 Stemming Algorithms.

Introduction to information retrieval

Discussion Class 9 Google.

Discussion Class 7 User Requirements.

Discussion Class 8 User Interfaces.

Presentation transcript:

1 Discussion Class 5 TREC

2 Discussion Classes Format: Questions. Ask a member of the class to answer. Provide opportunity for others to comment. When answering: Stand up. Give your name. Make sure that the TA hears it. Speak clearly so that all the class can hear. Suggestions: Do not be shy at presenting partial answers. Differing viewpoints are welcome.

3 Question 1: Objectives The TREC workshop series has four goals: (a)Encourage research in text based retrieval based on large test collections (b)Communication among industry, academia and government (c)Transfer of technology from research labs into products by demonstrating methodologies on real-world problems (d)Increase availability of appropriate evaluation techniques What does the ad hoc task contribute to each of these goals?

4 Question 2: The TREC Corpus SourceSize# DocsMedian (Mbytes)words/doc Wall Street Journal, , Associated Press newswire, , Computer Selects articles24275, Federal Register, , abstracts of DOE publications184226, Wall Street Journal, , Associated Press newswire, , Computer Selects articles17556, Federal Register, ,860396

5 Question 2: The TREC Corpus (a)What characteristics of this data are likely to impact the results of experiments? (b)Explain the statement, "Disks 1-5 were used as training data." (c)Suppose that you were designing two search engines: (i) for use with a library catalog, (ii) for use with a Web search service. How does your data differ from the TREC corpus?

6 Question 3: TREC Topic Statement Number: 409 legal, Pan Am, 103 Description: What legal actions have resulted from the destruction of Pan Am Flight 103 over Lockerbie, Scotland, on December 21, 1988? Narrative: Documents describing any charges, claims, or fines presented to or imposed by any court or tribunal are relevant, but documents that discuss charges made in diplomatic jousting are not relevant. A sample TREC topic statement

7 Question 3: TREC Topic Statement (a)What is the relationship between TREC topic statements and queries? (b)Distinguish between manual and automatic methods of query generation. (c)Explain the process used by the manual methods. (d)Some of the results used a time limit (e.g., "limited to no more than 10 minutes clock time"). What was being timed?

8 Question 4: Relevance Assessments (a)Explain the statement, "All TRECs have used the pooling method to assemble the relevance assessments." (b)How is relevance assessed? (c)What is the impact of some relevant documents being missed from the pool? (d)What is the problem of some relevant documents in the pool coming from only a single run? How serious is this?

9 Question 5: Evaluation

10 Question 5: What are: (a)The recall-precision curve? (b)The mean (non-interpolated) average precision? The report commented that, "two topics are fundamental to effective retrieval performance." What are they? How do the automatic tests differ from the manual?

11 Question 6: The future (a)Why was TREC-8 the last year for the ad hoc task? (b)Does this mean that text-based information retrieval is now solved?