1. Statistical NLP Spring 2011 Lecture 25: Summarization Dan Klein – UC Berkeley TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAA

2. Document Summarization

3. Multi-document Summarization … 27,000+ more

4. Extractive Summarization

5. Selection Maximum Marginal Relevance mid-‘90s present Maximize similarity to the query Minimize redundancy [Carbonell and Goldstein, 1998] s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Q Greedy search over sentences

6. Maximum Marginal Relevance Graph algorithms [Mihalcea 05++] mid-‘90s present Selection

7. Maximum Marginal Relevance Graph algorithms mid-‘90s present s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Nodes are sentences Selection

8. Maximum Marginal Relevance Graph algorithms mid-‘90s present s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Nodes are sentences Edges are similarities Selection

9. Maximum Marginal Relevance Graph algorithms mid-‘90s present s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Nodes are sentences Edges are similarities Stationary distribution represents node centrality Selection

10. Maximum Marginal Relevance Graph algorithms Word distribution models mid-‘90s present Input document distributionSummary distribution ~w P A (w) Obama? speech? health? Montana?w P D (w) Obama0.017 speech0.024 health0.009 Montana0.002 Selection

11. Maximum Marginal Relevance Graph algorithms Word distribution models mid-‘90s present SumBasic [Nenkova and Vanderwende, 2005] Value(w i ) = P D (w i ) Value(s i ) = sum of its word values Choose s i with largest value Adjust P D (w) Repeat until length constraint Selection

12. Maximum Marginal Relevance Graph algorithms Word distribution models Regression models mid-‘90s present s1s1s1s1 s2s2s2s2 s3s3s3s3 word values positionlength 12124 4214 6318 s2s2s2s2 s3s3s3s3 s1s1s1s1 F(x) frequency is just one of many features Selection

13. Maximum Marginal Relevance Graph algorithms Word distribution models Regression models Topic model-based [Haghighi and Vanderwende, 2009] mid-‘90s present Selection

21. H & V 09PYTHY

24. Maximum Marginal Relevance Graph algorithms Word distribution models Regression models Topic models Globally optimal search mid-‘90s present [McDonald, 2007] s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Q Optimal search using MMR Integer Linear Program Selection

25. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration. s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 conceptvalue Selection

26. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration. conceptvalue obama3 s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 Selection

27. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration. conceptvalue obama3 health2 s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 Selection

28. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration. conceptvalue obama3 health2 house1 s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 Selection

29. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. conceptvalue obama3 health2 house1 s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 The health care bill is a major test for the Obama administration.summarylengthvalue {s 1, s 3 }175 {s 2, s 3, s 4 }176 Length limit: 18 words greedy optimal Selection

30. Maximize Concept Coverage [Gillick and Favre 09] Optimization problem: Set Coverage Value of concept c Set of concepts present in summary s Set of extractive summaries of document set D Results 23.5 35.0 4.00 6.85

31. [Gillick, Riedhammer, Favre, Hakkani-Tur, 2008] total concept value summary length limit maintain consistency between selected sentences and concepts Integer Linear Program for the maximum coverage model Selection

32. [Gillick and Favre, 2009] This ILP is tractable for reasonable problems Selection

33. 52 submissions 27 teams 44 topics 10 input docs 100 word summaries Gillick & Favre Rating scale: 1-10 Humans in [8.3, 9.3] Rating scale: 1-10 Humans in [8.5, 9.3] Rating scale: 0-1 Humans in [0.62, 0.77] Rating scale: 0-1 Humans in [0.11, 0.15] Results [G & F, 2009]

34. [Gillick and Favre, 2008] Error Breakdown?

35. How to include sentence position? First sentences are unique Selection

36. Some interesting work on sentence ordering [Barzilay et. al., 1997; 2002] But choosing independent sentences is easier First sentences usually stand alone well Sentences without unresolved pronouns Classifier trained on OntoNotes: <10% error rate Baseline ordering module (chronological) is not obviously worse than anything fancier Selection

37. Problems with Extraction It is therefore unsurprising that Lindsay pleaded not guilty yesterday afternoon to the charges filed against her, according to her publicist. What would a human do?

38. Problems with Extraction It is therefore unsurprising that Lindsay pleaded not guilty yesterday afternoon to the charges filed against her, according to her publicist. What would a human do?

39. Sentence Rewriting [Berg-Kirkpatrick, Gillick, and Klein 11]

40. Sentence Rewriting [Berg-Kirkpatrick, Gillick, and Klein 11]

41. Sentence Rewriting [Berg-Kirkpatrick, Gillick, and Klein 11]

42. Sentence Rewriting [Berg-Kirkpatrick, Gillick, and Klein 11] New Optimization problem: Safe Deletions Set branch cut deletions made in creating summary s Value of deletion d How do we know how much a given deletion costs?

43. Learning [Berg-Kirkpatrick, Gillick, and Klein 11] Features: Embed ILP in cutting plane algorithm. Results 41.3

44. Sentence extraction is limiting... and boring! But abstractive summaries are much harder to generate… in 25 words? Beyond Extraction / Compression?

45. http://www.rinkworks.com/bookaminute/

46. Brown first saw the new planet Jan. 8 using the 48-inch Samuel Oschin Telescope Sample Errors:... news that the U.K. Panel on Takeover and Mergers... outside the U.S. Amoco already has shed such operations Preprocessing

47. Sentence Boundary Detection (SBD) is non-trivial because periods are ambiguous (sentence boundaries, abbreviations, or both). Wall Street Journal Corpus

48. Preprocessing Local context (one word on each side of the period) is sufficient. 80% of sentence-ending abbrs. in the WSJ test set Key innovations: Using an abbreviation list is too coarse.Abbr. Ends sentence TotalRatio Inc.1096830.16 Co.805660.14 Corp.676990.10 U.S.458000.06 Calif.24860.28 Ltd.231120.21