1 Do Summaries Help? A Task-Based Evaluation of Multi-Document Summarization Kathleen McKeown, Rebecca Passonneau, David Elson, Ani Nenkova, Julia Hirschberg Department of Computer Science Columbia University

2

3 Status of Multi-Document Summarization  Robust  Many existing systems (e.g. DUC 2004)    Extensive quantitative evaluation (intrinsic)  DUC 2001 – DUC 2005  Comparison of system summary content against human models  Do system generated summaries help end- users to make better use of the news?

4 Extrinsic Evaluation  Task-based evaluation of single document summarization using IR  TIPSTER-II, Brandow et al, Mani et al, Mochizuki&Okumura  Other factors can determine result (Jing et al)  Evaluation of evaluation metrics using similar task as ours  Amigo et al

5 Task Evaluation  Hypothesis: multi-document summaries enable users to find information efficiently  Task: fact-gathering given topic and questions  Resembles intelligence analyst task  Compared 4 parallel news browsing systems  Level 1: Source documents only  Level 2: One sentence multi-document summaries (e.g., Google News) linked to documents  Level 3: Newsblaster multi-document summaries linked to documents  Level 4: Human written multi-document summaries linked to documents

6 Results Preview  Quality of facts gathered significantly better  Newsblaster vs. documents alone  User satisfaction higher  Newsblaster and human summaries vs. documents and 1 sentence summaries  Summaries contributed important facts  Newsblaster and human summaries vs. 1 sentence summaries  Full multi-document summarization more powerful than no documents or single sentence summarization

7 Outline  Study design and execution  Scoring  Results

8 Evaluation Goals  Do summaries help users find information needed to perform a fact gathering task?  Do users use information from the summary in gathering their facts?  Do summaries increase user satisfaction with the online news system?  Do users create better fact sets with an online news system that includes summaries than one without?  How does type of summary (i.e., 1-sentence, system generated, human generated) affect quality of task output and user satisfaction?

9 Experimental Design  Subjects performed four 30-minute fact- gathering scenarios  Prompt: topic description plus three questions  Given a web page as sole resource  Space in which to compose response  Instructed to cut and paste from summary or article  Four event clusters per page u Two centrally relevant, two less relevant u 10 documents per cluster on average  Complete survey after each scenario

10 Prompt  The conflict between Israel and the Palestinians has been difficult for government negotiators to settle. Most recently, implementation of the "road map for peace," a diplomatic effort sponsored by the United States, Russia, the E.U. and the U.N., has suffered setbacks. However unofficial negotiators have developed a plan known as the Geneva Accord for finding a permanent solution to the conflict.  Who participated in the negotiations that produced the Geneva Accord?  Apart from direct participants, who supported the Geneva Accord preparations and how?  What has the response been to the Geneva Accord by the Palestinians and Israelis?

11 Experimental Design  Subjects performed four 30-minute fact- gathering scenarios  Prompt: topic description plus three questions  Produced a report containing a list of facts  Given a web page as sole resource  Space in which to compose response  Instructed to cut and paste from summary or article and make citation  Four event clusters per page u Two centrally relevant, two less relevant u 10 documents per cluster on average  Complete survey after each scenario

12 Level 1: Documents only, no summary

13 Level 2: 1-sentence summary for each event cluster, 1-sentence summary for each article

14 Full multi-document summaries Neither humans nor systems had access to the prompt  Level 3: Generated by Newsblaster for each event cluster  Level 4  Human written summary for each event cluster  Summary writers hired to write summaries u English or Journalism students with high verbal SAT

15 Levels 3 and 4: full summary for each event cluster

16 Experimental Design  Subjects performed four 30-minute fact- gathering scenarios  Prompt: topic description plus three questions  Produced a report containing a list of facts  Given a web page as sole resource  Space in which to compose response  Instructed to cut and paste from summary or article and make citation  Four event clusters per page u Two centrally relevant, two less relevant u 10 documents per cluster on average  Complete survey after each scenario

17 Study Execution  45 Subjects with varied background  73% students (BS, BA, journalism, law)  Native speakers of English  Paid, with promise of monetary prize for best report  3 studies, controlling for scenario and level order, ~11 subjects/scenario/level

18 Results – What was Measured  Report content across summary conditions: levels 1-4  User satisfaction per summary condition based on user surveys  Source of report content (summary or article) by counting fact citations

19 Scoring Report Content  Compare subject reports against a gold standard  Used the Pyramid method [HLT2004]  Avoids postulating an ideal exhaustive report  Predicts multiple equally good reports  Provides a metric for comparison  Gold standard for report x = pyramid of facts constructed from all reports except x  Relative importance of facts determined by report writers  34 reports per pyramid on average -> very stable

20 Pyramid representation  Tiers of differentially weighted facts  Top: few facts, high weight  Bottom: many facts, low weight  Report facts that don’t appear in pyramid have weight 0  Duplicate report facts get weight 0 W=1 W=33 W=34 …

21 Ideally informative report  Does not include a fact from a lower tier unless all facts from higher tiers are included as well

22 Ideally informative report  Does not include a fact from a lower tier unless all facts from higher tiers are included as well

23 Ideally informative report  Does not include a fact from a lower tier unless all facts from higher tiers are included as well

24 Ideally informative report  Does not include a fact from a lower tier unless all facts from higher tiers are included as well

25 Ideally informative report  Does not include a fact from a lower tier unless all facts from higher tiers are included as well

26 Ideally informative report  Does not include a fact from a lower tier unless all facts from higher tiers are included as well

Report Length  Wide variation in length impacts scores  We restricted report length < 1 standard deviation above the mean by truncating question answers

28 Results - Content Summary LevelPyramid Score Level 1 (docs only).3354 Level 2 (1 sentence).3757 Level 3 (Newsblaster).4269 Level 4 (Human).4027 Report quality improves from level 1 to level 3. (One scenario was dropped from results as it was problematic for subjects)

29 Statistical Analysis  ANOVA shows summary is marginally significant factor  Bonferonni method applied to determine differences in summary levels  Difference between Newsblaster and documents-only significant (P=.05)  Differences between Newsblaster and 1-sentence or human not significant  ANOVA shows that scenario, question and subject also significant factors

30 Results - User Satisfaction  6 questions in exit survey required response from a 1-5 scale  Average increases by summary type Level 1Level 2Level 3Level 4 Average

31 With full summaries, users read less Question Level 1Level 2Level 3Level 4 A. What best describes your experience reading source articles? I read a LOT more than I needed to 5. I only read those articles I needed to read

32 With Summaries, easier to write report and tended to have more time Question Level 1Level 2Level 3Level 4 B. How difficult do you think it was to write the report? Very difficult 5. Very easy C. Do you feel you had enough time to write the report? I needed more time 5. I had more than enough time

33 Usefulness improves with summary quality Human summaries help best with time Question Level 1Level 2Level 3Level 4 D. What best describes your experience using article summaries? n/a They had nothing useful to say 5. Everything I needed to know E. Did you feel that the automatic summaries saved you time, wasted time, or had no impact on your time budget? n/a Summaries wasted time 5. Summaries saved me time

34 Multiple Choice Survey Questions QuestionLevel 2Level 3Level 4 1. Which was most helpful? Source articles helped most 64%48%29% Equally helpful 32%29% Summaries helped most 5%24%43% 2. How did you budget your time? Most searching, some writing 55%48%67% Half searching, half writing 39%29%19% Most writing, some searching 7%24%14%

35 Citation Patterns  Report writers were significantly more likely to extract facts from summaries with Newsblaster and human summaries Level 2Level 3Level 4 Citations from summaries 8%17%27%

36 What we Learned  With summaries, a significant increase in quality of report  We hypothesized summaries would reduce reading time  As summary quality increases, users significantly more often draw facts from summary without decrease in report quality  Users claim they read fewer full documents with level 3 and 4 summaries  Full multi-document summarization better than 1 sentence summaries  Almost 5 times the proportion of subjects using Newsblaster summaries say summaries are helpful than subjects using 1 sentence summaries

37 Need for Follow-on Studies  Why no significant increase in report quality from level 2 to level 3?  Interface differences u Level 2 had summary for each article, level 3 did not u Level 3 required extra clicks to see list of articles  Studies to investigate controlling report length  Studies to investigate impact of scenario and question

38

39

40 Need for Follow-on Studies  Why no significant increase in report quality from level 2 to level 3?  Interface differences u Level 2 had summary for each article, level 3 did not u Level 3 required extra clicks to see list of articles  Studies to investigate controlling report length  Studies to investigate impact of scenario and question

41 Conclusions  Do summaries help?  Yes  Our task-based, extrinsic evaluation yielded significant conclusions  Full multi-document summarization (Newsblaster, human summaries) helps users perform better at fact-gathering than documents only  Users are more satisfied with full multi- document summarization than Google News style 1-sentence summaries