Query Based Event Extraction along a Timeline H.L. Chieu and Y.K. Lee DSO National Laboratories, Singapore (SIGIR 2004)

Slides:



Advertisements
Similar presentations
DISCOVERING EVENT EVOLUTION GRAPHS FROM NEWSWIRES Christopher C. Yang and Xiaodong Shi Event Evolution and Event Evolution Graph: We define event evolution.
Advertisements

Automatic Timeline Generation from News Articles Josh Taylor and Jessica Jenkins.
Learning to Suggest: A Machine Learning Framework for Ranking Query Suggestions Date: 2013/02/18 Author: Umut Ozertem, Olivier Chapelle, Pinar Donmez,
Chapter 5: Introduction to Information Retrieval
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
Choosing a Topic and Developing Research Questions
Effective Keyword Based Selection of Relational Databases Bei Yu, Guoliang Li, Karen Sollins, Anthony K.H Tung.
Linked data: P redicting missing properties Klemen Simonic, Jan Rupnik, Primoz Skraba {klemen.simonic, jan.rupnik,
Automatic Timeline Generation Jessica Jenkins Josh Taylor CS 276b.
Comparing Twitter Summarization Algorithms for Multiple Post Summaries David Inouye and Jugal K. Kalita SocialCom May 10 Hyewon Lim.
Query Dependent Pseudo-Relevance Feedback based on Wikipedia SIGIR ‘09 Advisor: Dr. Koh Jia-Ling Speaker: Lin, Yi-Jhen Date: 2010/01/24 1.
Systems Engineering and Engineering Management The Chinese University of Hong Kong Parameter Free Bursty Events Detection in Text Streams Gabriel Pui Cheong.
Evaluating Search Engine
Search Engines and Information Retrieval
Search and Retrieval: More on Term Weighting and Document Ranking Prof. Marti Hearst SIMS 202, Lecture 22.
Modern Information Retrieval
Learning to Advertise. Introduction Advertising on the Internet = $$$ –Especially search advertising and web page advertising Problem: –Selecting ads.
Reference Collections: Task Characteristics. TREC Collection Text REtrieval Conference (TREC) –sponsored by NIST and DARPA (1992-?) Comparing approaches.
Approaches to automatic summarization Lecture 5. Types of summaries Extracts – Sentences from the original document are displayed together to form a summary.
Chapter 5: Information Retrieval and Web Search
Query session guided multi- document summarization THESIS PRESENTATION BY TAL BAUMEL ADVISOR: PROF. MICHAEL ELHADAD.
Search Engines and Information Retrieval Chapter 1.
Newsjunkie: Providing Personalized Newsfeeds via Analysis of Information Novelty Gabrilovich et.al WWW2004.
1 Formal Models for Expert Finding on DBLP Bibliography Data Presented by: Hongbo Deng Co-worked with: Irwin King and Michael R. Lyu Department of Computer.
A Simple Unsupervised Query Categorizer for Web Search Engines Prashant Ullegaddi and Vasudeva Varma Search and Information Extraction Lab Language Technologies.
PAUL ALEXANDRU CHIRITA STEFANIA COSTACHE SIEGFRIED HANDSCHUH WOLFGANG NEJDL 1* L3S RESEARCH CENTER 2* NATIONAL UNIVERSITY OF IRELAND PROCEEDINGS OF THE.
A Markov Random Field Model for Term Dependencies Donald Metzler W. Bruce Croft Present by Chia-Hao Lee.
UOS 1 Ontology Based Personalized Search Zhang Tao The University of Seoul.
When Experts Agree: Using Non-Affiliated Experts To Rank Popular Topics Meital Aizen.
Processing of large document collections Part 7 (Text summarization: multi- document summarization, knowledge- rich approaches, current topics) Helena.
25/03/2003CSCI 6405 Zheyuan Yu1 Finding Unexpected Information Taken from the paper : “Discovering Unexpected Information from your Competitor’s Web Sites”
Search and Information Extraction Lab IIIT Hyderabad.
A Machine Learning Approach to Sentence Ordering for Multidocument Summarization and Its Evaluation D. Bollegala, N. Okazaki and M. Ishizuka The University.
RCDL Conference, Petrozavodsk, Russia Context-Based Retrieval in Digital Libraries: Approach and Technological Framework Kurt Sandkuhl, Alexander Smirnov,
Pete Bohman Adam Kunk.  ChronoSearch: A System for Extracting a Chronological Timeline ChronoChrono.
INTERESTING NUGGETS AND THEIR IMPACT ON DEFINITIONAL QUESTION ANSWERING Kian-Wei Kor, Tat-Seng Chua Department of Computer Science School of Computing.
Chapter 6: Information Retrieval and Web Search
CS 533 Information Retrieval Systems.  Introduction  Connectivity Analysis  Kleinberg’s Algorithm  Problems Encountered  Improved Connectivity Analysis.
Introduction to Digital Libraries hussein suleman uct cs honours 2003.
LexPageRank: Prestige in Multi- Document Text Summarization Gunes Erkan and Dragomir R. Radev Department of EECS, School of Information University of Michigan.
Binxing Jiao et. al (SIGIR ’10) Presenter : Lin, Yi-Jhen Advisor: Dr. Koh. Jia-ling Date: 2011/4/25 VISUAL SUMMARIZATION OF WEB PAGES.
Enhancing Cluster Labeling Using Wikipedia David Carmel, Haggai Roitman, Naama Zwerdling IBM Research Lab (SIGIR’09) Date: 11/09/2009 Speaker: Cho, Chin.
1 Web-Page Summarization Using Clickthrough Data* JianTao Sun, Yuchang Lu Dept. of Computer Science TsingHua University Beijing , China Dou Shen,
LANGUAGE MODELS FOR RELEVANCE FEEDBACK Lee Won Hee.
Chapter 8 Evaluating Search Engine. Evaluation n Evaluation is key to building effective and efficient search engines  Measurement usually carried out.
DOCUMENT UPDATE SUMMARIZATION USING INCREMENTAL HIERARCHICAL CLUSTERING CIKM’10 (DINGDING WANG, TAO LI) Advisor: Koh, Jia-Ling Presenter: Nonhlanhla Shongwe.
Retroactive Answering of Search Queries Beverly Yang Glen Jeh.
Measuring How Good Your Search Engine Is. *. Information System Evaluation l Before 1993 evaluations were done using a few small, well-known corpora of.
Creating Subjective and Objective Sentence Classifier from Unannotated Texts Janyce Wiebe and Ellen Riloff Department of Computer Science University of.
Advantages of Query Biased Summaries in Information Retrieval by A. Tombros and M. Sanderson Presenters: Omer Erdil Albayrak Bilge Koroglu.
Multi-level Bootstrapping for Extracting Parallel Sentence from a Quasi-Comparable Corpus Pascale Fung and Percy Cheung Human Language Technology Center,
Probabilistic Text Structuring: Experiments with Sentence Ordering Mirella Lapata Department of Computer Science University of Sheffield, UK (ACL 2003)
Finding document topics for improving topic segmentation Source: ACL2007 Authors: Olivier Ferret (18 route du Panorama, BP6) Reporter:Yong-Xiang Chen.
The Loquacious ( 愛說話 ) User: A Document-Independent Source of Terms for Query Expansion Diane Kelly et al. University of North Carolina at Chapel Hill.
Event-Based Extractive Summarization E. Filatova and V. Hatzivassiloglou Department of Computer Science Columbia University (ACL 2004)
Statistical Machine Translation Part II: Word Alignments and EM Alex Fraser Institute for Natural Language Processing University of Stuttgart
DISTRIBUTED INFORMATION RETRIEVAL Lee Won Hee.
2005/09/13 A Probabilistic Model for Retrospective News Event Detection Zhiwei Li, Bin Wang*, Mingjing Li, Wei-Ying Ma University of Science and Technology.
1 Random Walks on the Click Graph Nick Craswell and Martin Szummer Microsoft Research Cambridge SIGIR 2007.
Toward Entity Retrieval over Structured and Text Data Mayssam Sayyadian, Azadeh Shakery, AnHai Doan, ChengXiang Zhai Department of Computer Science University.
Annotating and measuring Temporal relations in texts Philippe Muller and Xavier Tannier IRIT,Université Paul Sabatier COLING 2004.
CS791 - Technologies of Google Spring A Web­based Kernel Function for Measuring the Similarity of Short Text Snippets By Mehran Sahami, Timothy.
A Survey on Automatic Text Summarization Dipanjan Das André F. T. Martins Tolga Çekiç
INFORMATION RETRIEVAL MEASUREMENT OF RELEVANCE EFFECTIVENESS 1Adrienn Skrop.
Text Similarity: an Alternative Way to Search MEDLINE James Lewis, Stephan Ossowski, Justin Hicks, Mounir Errami and Harold R. Garner Translational Research.
Using Blog Properties to Improve Retrieval Gilad Mishne (ICWSM 2007)
University Of Seoul Ubiquitous Sensor Network Lab Query Dependent Pseudo-Relevance Feedback based on Wikipedia 전자전기컴퓨터공학 부 USN 연구실 G
Unsupervised Learning Part 2. Topics How to determine the K in K-means? Hierarchical clustering Soft clustering with Gaussian mixture models Expectation-Maximization.
Yi-Chia Wang LTI 2nd year Master student
Retrieval Performance Evaluation - Measures
Presentation transcript:

Query Based Event Extraction along a Timeline H.L. Chieu and Y.K. Lee DSO National Laboratories, Singapore (SIGIR 2004)

2/35 Abstract We present a framework and a system that extracts events relevant to a query and place such event along a timeline. Each event is represented by a sentence, based on the assumption that “important” event are widely cited in many documents for a period time within which these events are of interest. Evaluation was performed using G8 leader names as queries: comparison made by human evaluators between manually and system generated timelines.

3/35 Introduction The growth of the WWW means that users need tools to find what they want. In this paper, we attempt to summarize a big collection of documents by placing sentences that report “important” events related to the query along a timeline. For this purpose, we assume that documents are time-tagged, and each sentence is assigned a date by using simple rules

4/35 Introduction Events that are of interest to many people are often reported in many different news articles from different sources. Updates and commentaries of such event would also persist for a period of time. Our system made use of this property to determine if events are “important”

5/35 Related Work Past work on summarization focused on either summarizing single document, or summarizing similarities and difference in small clusters of documents. Recent DUC introduced summarization focused by a viewpoint or a question, but still from small clusters of documents. Mckeown et al. (2001) used different summarizers for different types of document clusters (MultiGen, DEMS) Schiffman et al. (2000) summarized collections of documents to produce biographical summaries without taking into account temporal information

6/35 Related Work Work has been done on abstracting, where sentences are generated instead of extracted (Barzilay, 1999, Mckeown, 1999) Kevin and Marcu (2000) explored summarization by sentence compression Summarization system can also be based on information extraction systems, where summaries are generated from extracted templates (Radev, 1998) For multidocument summarization, the ordering of sentences within a summary has also been explored to produce coherent summaries for the reader (Barzilay, 2002, 2003) Sentences extracted on a timeline refer to different events that happened on different dates

7/35 Related Work Most work on temporal aspects of linguistics addresses the annotation of temporal expressions and temporal attributes of events (Mani, 2000, 2001) The DARPA ACE task of Relation Detection and Tracking involves attaching temporal attributes to relations The area of TDT brought forth some work on summarization of incoming streams of news Swan and Allan (2000) pioneered work on timeline overviews Allan et al. (2001) addressed temporal summarization of news topics by providing sentence-based summaries, where they defined precision and recall measures that capture relevance and novelty

8/35 Related Work Smith (2002) extracted associations between date and location and showed that associations detected corresponded well with known event Kleinberg (2002) modeled data streams to infer a hierarchical structure of bursts in the stream and extract bursts of term usage in title of papers from conference databases In this paper, our system summarizes documents returned by a query-base search and aims to extract events related to an entity or type of event by sentence extraction along a timeline

9/35 Framework For Timeline Construction

10/35 Framework For Timeline Construction Ranking Sentences Ranking Metric: “Interest” We define “interesting event” in SC(q) as events that are a subject of interest in many sentences in SC(q) The sentences in the set of SC(q) can then be raked according to their interest

11/35 Framework For Timeline Construction Ranking Sentences Ranking Metric: “Burstiness” Reports of “important” events often cluster around the date of their occurrence To sift out “important” events, we compare the number of reports of an event within and outside a date duration We model reports of events as a random process with an unknown binominal distribution, we can check for associations between event and date

12/35 Framework For Timeline Construction Ranking Sentences Ranking Metric: “Burstiness” In stead of taking a fixed value for k, we calculate the log likelihood ratio for each value of K from 1 to 10

13/35 Implementation Assumptions In step (1), a sentence s is deemed relevant to query q, if one of the terms in q appears in s In step (1) In step (2), we resolve date(s) with a few rules In step (2) Only one event is mentioned per sentence: each sentence is mapped to one day

14/35 Implementation Resolving Date We further assume that the first date expression detected in a sentence s is the date of the event mentioned in s We crafted simple rules to detect date expression in sentence, and resolve them to absolute dates using the article date as reference In the case where no date expression is detected in the entire sentence s, date(s) is taken to be the date of publication of the article containing s

15/35 Implementation Mapping sentences to vectors In order to compare sentences, we mapped each sentence to a vector In stead of using the traditional idf, we use inverse date frequency (iDf) for term weighting: sentences are grouped by dates rather than documents We use a cutoff of 3 for date-frequency: all terms that occur in 3 dates or less would have iDf calculated based on a date-frequency of 3 Besides removing stop words from each sentence vector, words that are found to be date expression are also removed

16/35 Implementation Sentences reporting the same event The decision of whether two sentences report the same event boils down to the problem of paraphrasing We use the cosine similarity between the vectors of the 2 sentences as the probability that they paraphrase each other

17/35 Implementation Sentences reporting the same event Interest and burstiness were redefined Sentences reporting events happening on different dates are unlikely to be paraphrases of each other where T is a constant, T=10

18/35 Implementation Sentences reporting the same event Interest and burstiness were redefined The log likelihood ratio can still be calculated

19/35 Implementation Removing duplicated sentences In order to decide the range of the neighborhood in which sentences are deleted, we define extent of each sentence for each of the measures interest and burstiness For both measures, extents can take a maximum value of 10 days When picking up the top N sentences after ranking, sentences are discards if they are within any of the extents of sentences already selected P=80%

20/35 Implementation Efficiency Direct computation of the sum of cosines would still be expensive if SC(q) contains a large number of sentences As the dot product is distributive with respect to addition, the sum of dot products used in the measures can be converted to a dot product of sums

21/35 Case Study on Development Corpus For development, we conducted our experiments on articles from the corpus of Reuters Volume 1, dated from January to August, 1997 We discuss the performance of our system using the query “earthquake(s)” and “quake(s)” We found a chronology of major earthquakes in Reuters articles on internet (All quakes in this list “killed more than 1,000 people”) Although many earthquakes occurred in January to August 1997, only two of them were major enough to figure in this list 10/5/1997, Eastern Iran 28/2/1997, Northeast Iran

22/35 Case Study on Development Corpus

23/35 Case Study on Development Corpus

24/35 Evaluation We performed a small experiment to compare our timelines against manually constructed timelines We chose to use articles from the first 6 months of 2002 from the English Gigaword corpus: Agence France Press English Service, Associated Press Worldstream English Service, and The New York Times Newswire Service For our experiment, we used person names, namely, the eight leaders of the countries in G8 in 2002

25/35 Evaluation

26/35 Evaluation From machine generated timelines, we automatically removed events that are not in 1/2002~6/2002 For this evaluation, we used only the top ten sentences generated by each measure We employed four arts undergraduates majoring in history to help us evaluate our system The evaluation was carried out in three phases

27/35 Evaluation Phase 1 Evaluators were told to construct timelines of ten sentences for queries assigned to them by doing their own research (4 queries/person) Phase 2 They were given 4 timelines (2:machine, 2:manually) for each of the 8 queries, and asked to answer questions on each timeline They were also told to rank the 4 timelines for each query from best to worst, and to rank all 40 sentences from 1 to 40, in term of importance While ranking sentences, they were asked to tie ranks for sentences that relate “equivalent” events Phase 3 They were asked to judge on the dates of events for the machine generated timelines, given the sentence and its source document

28/35 Evaluation

29/35 Evaluation

30/35 Evaluation

31/35 Evaluation

32/35 Evaluation

33/35 Evaluation

34/35 Evaluation

35/35 Conclusion and Future Work We have described a system for extracting sentences from a big corpus to be placed along a timeline given a query from the user The advantage of our work Efficient Sentences are better units of information as they allow quick access to their source documents and may be informative enough for the user It is our intention to integrate it with a search engine so that it can work real time on user queries