1. r Comparing Presentation Summaries: Slides vs. Reading vs. Listening Liwei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and Multimedia Group Microsoft Research Liwei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and Multimedia Group Microsoft Research

2. CHI 2000 2 Motivation Multimedia presentations are being archived for on-demand access  University courses  Corporate training and seminars Effective summarization and skimming can help users utilize time better Multimedia presentations are being archived for on-demand access  University courses  Corporate training and seminars Effective summarization and skimming can help users utilize time better

3. CHI 2000 3 Video Skimming Techniques Time compression  1.5 – 2.5 saving factor at most Video summary  2.5+ saving factor is possible Time compression  1.5 – 2.5 saving factor at most Video summary  2.5+ saving factor is possible

4. CHI 2000 4 Previous Summarization Study Compared 4 video summary techniques  1 by authors of the presentation  3 by computer algorithms  Pre- and post quizzes and subjective ratings  More details in our paper in ACM Multimedia 99  “Auto-Summarization of Audio-Video Presentations” Compared 4 video summary techniques  1 by authors of the presentation  3 by computer algorithms  Pre- and post quizzes and subjective ratings  More details in our paper in ACM Multimedia 99  “Auto-Summarization of Audio-Video Presentations”

5. CHI 2000 5 Auto Summary Study Results All four summaries improve quiz scores  Human-generated summary is significantly better than computers  No difference among computer-generated summaries Overall, all are appreciated by subjects All four summaries improve quiz scores  Human-generated summary is significantly better than computers  No difference among computer-generated summaries Overall, all are appreciated by subjects

6. CHI 2000 6 Questions Raised What about other forms of summary?  Amount of information from slides?  Skimming text transcript vs. watching video?  Transcripts with key points highlighted vs. video summaries? What about other forms of summary?  Amount of information from slides?  Skimming text transcript vs. watching video?  Transcripts with key points highlighted vs. video summaries?

7. CHI 2000 7 Experimental Design (1) 4 summarization techniques  PowerPoint slides only  Raw text transcripts  Transcripts with key points highlighted  Author-generated video summaries 4 summarization techniques  PowerPoint slides only  Raw text transcripts  Transcripts with key points highlighted  Author-generated video summaries

8. CHI 2000 8 Experimental Design (2) To compare summarization techniques  Objective measure: quiz score improvement before and after watching a summary  Subjective measure: user ratings 4 talks chosen from Microsoft training site  Original presenters wrote quiz questions To compare summarization techniques  Objective measure: quiz score improvement before and after watching a summary  Subjective measure: user ratings 4 talks chosen from Microsoft training site  Original presenters wrote quiz questions

9. CHI 2000 9 Experimental Design (3) 24 Microsoft employees were subjects  Quiz before and after each summary  Watches 4 summaries, each of a different type  Summary types and talks are counter-balanced within each subject 24 Microsoft employees were subjects  Quiz before and after each summary  Watches 4 summaries, each of a different type  Summary types and talks are counter-balanced within each subject

10. CHI 2000 10 PowerPoint Slides Only

11. CHI 2000 11 Raw Text Transcript

12. CHI 2000 12 Text Transcript w/ Highlights

13. CHI 2000 13 Video Summary

14. CHI 2000 14 Four Presentations Used P1P2P3P4 Length (min) 71414771 # of slides 17182752 # of pages 1510815 Highlighted19%24%25%20%

15. CHI 2000 15 Quiz Score Improvement (1) Plot by summary types

16. CHI 2000 16 Quiz Score Improvement (2) Highlight text and video summary > (at p (at p<0.001) slide only and raw text Highlight text and video summary > (at p (at p<0.001) slide only and raw text

17. CHI 2000 17 Quiz Score Improvement (3) Video summary > highlight text ? p = 0.087 Video summary > highlight text ? p = 0.087

18. CHI 2000 18 Quiz Score Improvement (4) Plot by presentations

19. CHI 2000 19 Subjective Ratings (1) Slide only Raw text Highlight text Video summary Synopsis3.133.584.704.96 Key points 41%62%64%69% Skip talk 1.963.504.524.41 Concise2.923.504.525.13 Coherent2.834.174.354.13 Table by summarization techniques

20. CHI 2000 20 Subjective Ratings (2) Highlight text and video summary > (at p=0.01) slide only and raw text Highlight text and video summary > (at p=0.01) slide only and raw text Slide only Raw text Highlight text Video summary Synopsis3.133.584.704.96 Key points 41%62%64%69% Skip talk 1.963.504.524.41 Concise2.923.504.525.13 Coherent2.834.174.354.13

21. CHI 2000 21 Subjective Ratings (3) Slide only Raw text Highlight text Video summary Synopsis3.133.584.704.96 Key points 41%62%64%69% Skip talk 1.963.504.524.41 Concise2.923.504.525.13 Coherent2.834.174.354.13 Highlight text and video summary are not significantly different (at p=0.05) Highlight text and video summary are not significantly different (at p=0.05)

22. CHI 2000 22 Subjective Ratings (4) Slide only Raw text Highlight text Video summary Synopsis3.133.584.704.96 Key points 41%62%64%69% Skip talk 1.963.504.524.41 Concise2.923.504.525.13 Coherent2.834.174.354.13 Raw text, highlight text, and video summary > (at p=0.05) slide only Raw text, highlight text, and video summary > (at p=0.05) slide only

23. CHI 2000 23 User Comments 13 out of 24 like video summaries  “It is more enjoyable listening and seeing the presenter.” 11 prefer highlighted transcripts  “I liked having the option of being able to get more detailed info when I need it.” 13 out of 24 like video summaries  “It is more enjoyable listening and seeing the presenter.” 11 prefer highlighted transcripts  “I liked having the option of being able to get more detailed info when I need it.”

24. CHI 2000 24 Conclusions Effective summary techniques are key This study compared 4 summarization techniques  Slide only does not work well for most talks  Raw text transcript is hard to read  Human produced summaries work better Effective summary techniques are key This study compared 4 summarization techniques  Slide only does not work well for most talks  Raw text transcript is hard to read  Human produced summaries work better

25. CHI 2000 25 Conclusions (cont.) Slide authoring style makes a difference What to do?  For authors: tools to generate summaries  For users: interactive and intelligent video browser Slide authoring style makes a difference What to do?  For authors: tools to generate summaries  For users: interactive and intelligent video browser

26. CHI 2000 26 Compare with AutoSum Study (1) Current study and auto summary study are comparable  4 talks and quiz are the same  Both have author-generated summary  Slides are shown in all conditions for both  Evaluation methods are the same Current study and auto summary study are comparable  4 talks and quiz are the same  Both have author-generated summary  Slides are shown in all conditions for both  Evaluation methods are the same

27. CHI 2000 27 Compare with AutoSum Study (2) AutoSum Study Current Study

28. CHI 2000 28 Compare with AutoSum Study (3) SA*ASO Synopsis4.335.004.963.13 Key points 56%76%69%41% Skip talk 3.214.964.411.96 Concise4.085.635.132.92 Coherent3.575.334.132.83 Subject ratings (AutoSum vs. Current)

29. CHI 2000 29 Compare with AutoSum Study (4) SA*ASO Synopsis4.335.004.963.13 Key points 56%76%69%41% Skip talk 3.214.964.411.96 Concise4.085.635.132.92 Coherent3.575.334.132.83 A* in AutoSum consistently > A

30. CHI 2000 30 Compare with AutoSum Study (3) SA*ASO Synopsis4.335.004.963.13 Key points 56%76%69%41% Skip talk 3.214.964.411.96 Concise4.085.635.132.92 Coherent3.575.334.132.83 Slide-based summary (S) > slide only (SO)