1. Timestamped Graphs: Evolutionary Models of Text for Multi-document Summarization Ziheng Lin and Min-Yen Kan Department of Computer Science National University of Singapore, Singapore

2. Using Evolutionary Models of Text for Multi-document summarization 2TextGraphs 2 at HLT/NAACL 2007 Summarization Traditionally, heuristics for extractive summarization – Cue/stigma phrases – Sentence position (relative to document, section, paragraph) – Sentence length – TF×IDF, TF scores – Similarity (with title, context, query) With the advent of machine learning, heuristic weights for different features are tuned by supervised learning In last few years, graphical representations of text have shed new light on the summarization problem

3. Using Evolutionary Models of Text for Multi-document summarization 3TextGraphs 2 at HLT/NAACL 2007 Prestige as sentence selection One motivation of using graphical methods was to model the problem as finding prestige of nodes in a social network PageRank used random walk to smooth the effect of non-local context HITS and SALSA to model hubs and authorities In summarization, lead to TextRank and LexRank Contrast with previous graphical approaches (Salton et al. 1994) Did we leave anything out of our representation for summarization? Yes, the notion of an evolving network

4. Using Evolutionary Models of Text for Multi-document summarization 4TextGraphs 2 at HLT/NAACL 2007 Social networks change! Natural evolving networks (Dorogovtsev and Mendes, 2001) – Citation networks: New papers can cite old ones, but the old network is static – The Web: new pages are added with an old page connecting it to the web graph, old pages may update links

5. Using Evolutionary Models of Text for Multi-document summarization 5TextGraphs 2 at HLT/NAACL 2007 Evolutionary models for summarization Writers and readers often follow conventional rhetorical styles - articles are not written or read in an arbitrary way Consider the evolution of texts using a very simplistic model – Writers write from the first sentence onwards in a text – Readers read from the first sentence onwards of a text A simple model: sentences get added incrementally to the graph

6. Using Evolutionary Models of Text for Multi-document summarization 6TextGraphs 2 at HLT/NAACL 2007 Timestamped Graph Construction Approach – These assumptions suggest us to iteratively add sentences into the graph in chronological order. – At each iteration, consider which edges to add to the graph. – For single document: simple and straightforward: add 1 st sentence, followed by the 2 nd, and so forth, until the last sentence is added – For multi-document: treat it as multiple instances of single documents, which evolve in parallel; i.e., add 1 st sentences of all documents, followed by all 2 nd sentences, and so forth Doesn’t really model chronological ordering between articles, fix later

7. Using Evolutionary Models of Text for Multi-document summarization 7TextGraphs 2 at HLT/NAACL 2007 Timestamped Graph Construction Model: Documents as columns – d i = document i Sentences as rows –s j = j th sentence of document

8. Using Evolutionary Models of Text for Multi-document summarization 8TextGraphs 2 at HLT/NAACL 2007 Timestamped Graph Construction A multi document example doc1 doc2 doc3 sent1 sent2 sent3

9. Using Evolutionary Models of Text for Multi-document summarization 9TextGraphs 2 at HLT/NAACL 2007 An example TSG: DUC 2007 D0703A-A

10. Using Evolutionary Models of Text for Multi-document summarization 10TextGraphs 2 at HLT/NAACL 2007 Properties of nodes Timestamped Graph Construction Properties of edges Input text transformation function These are just one instance of TSGs Let’s generalize and formalize them Def: A timestamped graph algorithm tsg(M) is a 9-tuple (d, e, u, f,σ, t, i, s, τ) that specifies a resulting algorithm that takes as input the set of texts M and outputs a graph G

11. Using Evolutionary Models of Text for Multi-document summarization 11TextGraphs 2 at HLT/NAACL 2007 Edge properties (d, e, u, f) Edge Direction (d) – Forward, backward, or undirected Edge Number (e) – number of edges to instantiate per timestep Edge Weight (u) – weighted or unweighted edges Inter-document factor (f) – penalty factor for links between documents in multi-document sets.

12. Using Evolutionary Models of Text for Multi-document summarization 12TextGraphs 2 at HLT/NAACL 2007 Node properties ( σ, t, i, s) Vertex selection function σ(u, G) – One strategy: among those nodes not yet connected to u in G, choose the one with highest similarity according to u – Similarity functions: Jaccard, cosine, concept links (Ye et al.. 2005) Text unit type (t) – Most extractive algorithms use sentences as elementary units Node increment factor (i) – How many nodes get added at each timestep Skew degree (s) – Models how nodes in multi-document graphs are added – Skew degree = how many iterations to wait before adding the 1 st sentence of the next document – Let’s illustrate this …

13. Using Evolutionary Models of Text for Multi-document summarization 13TextGraphs 2 at HLT/NAACL 2007 Skew Degree Examples time(d1) < time(d2) < time(d3) < time(d4) d1 d2 d3 d4 Skewed by 1Skewed by 2 Freely skewed d1 d2 d3 d4 Freely skewed = Only add a new document when it would be linked by some node using vertex function σ

14. Using Evolutionary Models of Text for Multi-document summarization 14TextGraphs 2 at HLT/NAACL 2007 Input text transformation function (τ) Document Segmentation Function (τ) – Problem observed in some clusters where some documents in a multi-document cluster are very long – Takes many timestamps to introduce all of the sentences, causing too many edges to be drawn –Τ(G) segments long documents into several sub docs Solution is too hacked – hope to investigate more in current and future work d5 d5bd5a

15. Using Evolutionary Models of Text for Multi-document summarization 15TextGraphs 2 at HLT/NAACL 2007 Timestamped Graph Construction Representations – We can model a number of different algorithms using this 9-tuple formalism: (d, e, u, f,σ, t, i, s,τ) – The given toy example: (f, 1, 0, 1, max-cosine-based, sentence, 1, 0, null) – LexRank graphs: (u, N, 1, 1, cosine-based, sentence, L max, 0, null) N = total number of sentences in the cluster; L max = the max document length i.e., all sentences are added into the graph in one timestep, each connected to all others, and cosine scores are given to edge weights

16. TSG-based summarization Methodology Evaluation Analysis

17. Using Evolutionary Models of Text for Multi-document summarization 17TextGraphs 2 at HLT/NAACL 2007 System Overview Sentence splitting –Detect and mark sentence boundaries –Annotate each sentence with the doc ID and the sentence number –E.g., XIE19980304.0061: 4 March 1998 from Xinhua News; XIE19980304.0061-14: the 14 th sentence of this document Graph construction –Construct TSG in this phase

18. Using Evolutionary Models of Text for Multi-document summarization 18TextGraphs 2 at HLT/NAACL 2007 System Overview Sentence Ranking – Apply topic-sensitive random walk on the graph to redistribute the weights of the nodes Sentence extraction – Extract the top-ranked sentences – Two different modified MMR re- rankers are used, depending on whether it is main or update task

19. Using Evolutionary Models of Text for Multi-document summarization 19TextGraphs 2 at HLT/NAACL 2007 Evaluation Dataset: DUC 2005, 2006 and 2007. Evaluation tool: ROUGE: n-gram based automatic evaluation Each dataset contains 50 or 45 clusters, each cluster contains a query and 25 documents Evaluate on some parameters –Do different e values affect the summarization process? –How do topic-sensitivity and edge weighting perform in running PageRank? –How does skewing the graph affect the information flow in the graph?

20. Using Evolutionary Models of Text for Multi-document summarization 20TextGraphs 2 at HLT/NAACL 2007 Evaluation on number of edges (e) Tried different e values Optimal performance: e = 2 At e = 1, graph is too loosely connected, not suitable for PageRank → very low performance At e = N, a LexRank system N N N e = 2

21. Using Evolutionary Models of Text for Multi-document summarization 21TextGraphs 2 at HLT/NAACL 2007 Evaluation (other edge parameters) PageRank: generic vs topic-sensitive Edge weight (u): unweighted vs weighted Optimal performance: topic-sensitive PageRank and weighted edges Topic- sensitive Weighted edges ROUGE-1ROUGE-2 No 0.393580.07690 YesNo0.394430.07838 NoYes0.398230.08072 Yes 0.398450.08282

22. Using Evolutionary Models of Text for Multi-document summarization 22TextGraphs 2 at HLT/NAACL 2007 Evaluation on skew degree (s) Different skew degrees: s = 0, 1 and 2 Optimal performance: s = 1 s = 2 introduces a delay interval that is too large Need to try freely skewed graphs Skew degreeROUGE-1ROUGE-2 00.369820.07580 10.372680.07682 20.369980.07489

23. Using Evolutionary Models of Text for Multi-document summarization 23TextGraphs 2 at HLT/NAACL 2007 Holistic Evaluation in DUC We participated in DUC 2007 with an extractive-based TSG system Main task: 12 th for ROUGE-2, 10 th for ROUGE-SU4 among 32 systems Update task: 3 rd for ROUGE-2, 4 th for ROUGE-SU4 among 24 systems Used a modified version of maximal marginal relevance to penalize links in previously read articles – Extension of inter-document factor (f) TSG formalism better tailored to deal with update / incremental text tasks New method that may be competitive with current approaches – Other top scoring systems may do sentence compression, not just extraction

24. Using Evolutionary Models of Text for Multi-document summarization 24TextGraphs 2 at HLT/NAACL 2007 Conclusion Proposed a timestamped graph model for text understanding and summarization – Adds sentences one at a time Parameterized model with nine variables – Canonicalizes representation for several graph based summarization algorithms Future Work Freely skewed model Empirical and theoretical properties of TSGs (e.g., in-degree distribution)

25. Backup Slides 25 Minute talk total 26 Apr 2007, 11:50-12:15

26. Using Evolutionary Models of Text for Multi-document summarization 26TextGraphs 2 at HLT/NAACL 2007 Differences for main and update task processing Main task: 1.Construct a TSG for input cluster 2.Run topic-sensitive PageRank on the TSG 3.Apply first modified version of MMR to extract sentences Update task: Cluster A: – Construct a TSG for cluster A – Run topic-sensitive PageRank on the TSG – Apply the second modified version of MMR to extract sentences Cluster B: – Construct a TSG for clusters A and B – Run topic-sensitive PageRank on the TSG; only retain sentences from B – Apply the second modified version of MMR to extract sentences Cluster C: – Construct a TSG for clusters A, B and C – Run topic-sensitive PageRank on the TSG; only retain sentences from C – Apply the second modified version of MMR to extract sentences

27. Using Evolutionary Models of Text for Multi-document summarization 27TextGraphs 2 at HLT/NAACL 2007 Sentence Ranking Once a timestamped graph is built, we want to compute an prestige score for each node PageRank: use an iterative method that allows the weights of the nodes to redistribute until stability is reached Similarities as edges → weighted edges; query → topic-sensitive Topic sensitive (Q) portion Standard random walk term

28. Using Evolutionary Models of Text for Multi-document summarization 28TextGraphs 2 at HLT/NAACL 2007 Sentence Extraction – Main task Original MMR: integrates a penalty of the maximal similarity of the candidate document and one selected document Ye et al. (2005) introduced a modified MMR: integrates a penalty of the total similarity of the candidate sentence and all selected sentences Score(s) = PageRank score of s; S = selected sentences This is used in the main task Penalty: All previous sentence similarity

29. Using Evolutionary Models of Text for Multi-document summarization 29TextGraphs 2 at HLT/NAACL 2007 Sentence Extraction – Update task Update task assumes readers already read previous cluster(s) – implies we should not select sentences that have redundant information with previous cluster(s) Propose a modified MMR for the update task: – consider the total similarity of the candidate sentence with all selected sentences and sentences in previously-read cluster(s) P contains some top-ranked sentences in previous cluster(s) Previous cluster overlap

30. Using Evolutionary Models of Text for Multi-document summarization 30TextGraphs 2 at HLT/NAACL 2007 References Günes Erkan and Dragomir R. Radev. 2004. LexRank: Graph-based centrality as salience in text summari-zation. Journal of Artificial Intelligence Research, (22). Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring-ing order into texts. In Proceedings of EMNLP 2004. S.N. Dorogovtsev and J.F.F. Mendes. 2001. Evolution of networks. Submitted to Advances in Physics on 6th March 2001. Sergey Brin and Lawrence Page. 1998. The anatomy of a large-scale hypertextual Web search engine. Com-puter Networks and ISDN Systems, 30(1-7). Jon M. Kleinberg. 1999. Authoritative sources in a hy-perlinked environment. In Proceedings of ACM-SIAM Symposium on Discrete Algorithms, 1999. Shiren Ye, Long Qiu, Tat-Seng Chua, and Min-Yen Kan. 2005. NUS at DUC 2005: Understanding docu-ments via concepts links. In Proceedings of DUC 2005.