Information Retrieval and Extration 期末專題實驗 — Relevant Sentence Detection
Overview 實驗目標 實驗目標 – 句子層次的主題相關性偵測 分組 分組 – 每組 1~4 人 Deadline and Demo Deadline and Demo –6/21
Typical Ad Hoc Retrieval
Relevant Sentence Detection
Topics and Collection TREC 2004 Novelty Track TREC 2004 Novelty Track –10 topics and their relevance judgments for system training and parameter tuning “ TrainingTopics.txt ” in the dataset “ TrainingTopics.txt ” in the dataset –3-5 testing topics for demo
Topic Example <top> Number: N54 Number: N54<title> Firestone Tire Recall <toptype>Event Description: Description: The widespread affects of the Firestone tire recall Narrative: Narrative: Opinion of the public, personal, business, or company asto the general scope of the recall (too much, not enough);as to the type of tires or vehicle that should beincluded; as well as any customer complaints or onany actions taken are relevant. Documents that brieflyreport on the recall with no enlightening details are notrelevant. <documents>APW NYT </top>
Relevant Document Example <DOC><DOCNO> APW APW </DOCNO><DOCTYPE> NEWS STORY NEWS STORY </DOCTYPE><DATE_TIME> : :19 </DATE_TIME><BODY><HEADLINE> Source: Firestone To Recall Tires Source: Firestone To Recall Tires </HEADLINE> By NEDRA PICKLER By NEDRA PICKLER <TEXT> Most of the Firestone ATX, ATX II and Wilderness AT tires are on Ford Explorers, the industry's top-selling SUV, but the recall will include tires on all brands of vehicles, the source said on condition of anonymity. Most of the Firestone ATX, ATX II and Wilderness AT tires are on Ford Explorers, the industry's top-selling SUV, but the recall will include tires on all brands of vehicles, the source said on condition of anonymity. The recalled tires will be replaced by other Firestone tires, the source said. The recalled tires will be replaced by other Firestone tires, the source said. </TEXT></BODY></DOC>
Evaluation Precision, Recall and F-measure Precision, Recall and F-measure –Usage of the evaluation program ( “ 04.eval_novelty_run.pl ” in the dataset) Usage of the evaluation programUsage of the evaluation program
List of Content in the Dataset “ TrainingTopics.txt ” (file) “ TrainingTopics.txt ” (file) –Training topics for system development “ RelevantDocsForTrainingTopics ” (dir) “ RelevantDocsForTrainingTopics ” (dir) –Relevant documents for each training topic “ 04.qrels.relevant(TrainingTopics).txt ” (file) “ 04.qrels.relevant(TrainingTopics).txt ” (file) –The set of relevant sentences for training topics “ 04.eval_novelty_run.pl ” (file) “ 04.eval_novelty_run.pl ” (file) –Program for evaluation “ AdditionalDocuments(LATIMES) ” (dir) “ AdditionalDocuments(LATIMES) ” (dir) –Additional (not necessary) document collection