r Comparing Presentation Summaries: Slides vs. Reading vs. Listening Liwei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and Multimedia Group Microsoft Research Liwei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and Multimedia Group Microsoft Research

CHI Motivation Multimedia presentations are being archived for on-demand access  University courses  Corporate training and seminars Effective summarization and skimming can help users utilize time better Multimedia presentations are being archived for on-demand access  University courses  Corporate training and seminars Effective summarization and skimming can help users utilize time better

CHI Video Skimming Techniques Time compression  1.5 – 2.5 saving factor at most Video summary  2.5+ saving factor is possible Time compression  1.5 – 2.5 saving factor at most Video summary  2.5+ saving factor is possible

CHI Previous Summarization Study Compared 4 video summary techniques  1 by authors of the presentation  3 by computer algorithms  Pre- and post quizzes and subjective ratings  More details in our paper in ACM Multimedia 99  “Auto-Summarization of Audio-Video Presentations” Compared 4 video summary techniques  1 by authors of the presentation  3 by computer algorithms  Pre- and post quizzes and subjective ratings  More details in our paper in ACM Multimedia 99  “Auto-Summarization of Audio-Video Presentations”

CHI Auto Summary Study Results All four summaries improve quiz scores  Human-generated summary is significantly better than computers  No difference among computer-generated summaries Overall, all are appreciated by subjects All four summaries improve quiz scores  Human-generated summary is significantly better than computers  No difference among computer-generated summaries Overall, all are appreciated by subjects

CHI Questions Raised What about other forms of summary?  Amount of information from slides?  Skimming text transcript vs. watching video?  Transcripts with key points highlighted vs. video summaries? What about other forms of summary?  Amount of information from slides?  Skimming text transcript vs. watching video?  Transcripts with key points highlighted vs. video summaries?

CHI Experimental Design (1) 4 summarization techniques  PowerPoint slides only  Raw text transcripts  Transcripts with key points highlighted  Author-generated video summaries 4 summarization techniques  PowerPoint slides only  Raw text transcripts  Transcripts with key points highlighted  Author-generated video summaries

CHI Experimental Design (2) To compare summarization techniques  Objective measure: quiz score improvement before and after watching a summary  Subjective measure: user ratings 4 talks chosen from Microsoft training site  Original presenters wrote quiz questions To compare summarization techniques  Objective measure: quiz score improvement before and after watching a summary  Subjective measure: user ratings 4 talks chosen from Microsoft training site  Original presenters wrote quiz questions

CHI Experimental Design (3) 24 Microsoft employees were subjects  Quiz before and after each summary  Watches 4 summaries, each of a different type  Summary types and talks are counter-balanced within each subject 24 Microsoft employees were subjects  Quiz before and after each summary  Watches 4 summaries, each of a different type  Summary types and talks are counter-balanced within each subject

CHI PowerPoint Slides Only

CHI Raw Text Transcript

CHI Text Transcript w/ Highlights

CHI Video Summary

CHI Four Presentations Used P1P2P3P4 Length (min) # of slides # of pages Highlighted19%24%25%20%

CHI Quiz Score Improvement (1) Plot by summary types

CHI Quiz Score Improvement (2) Highlight text and video summary > (at p (at p<0.001) slide only and raw text Highlight text and video summary > (at p (at p<0.001) slide only and raw text

CHI Quiz Score Improvement (3) Video summary > highlight text ? p = Video summary > highlight text ? p = 0.087

CHI Quiz Score Improvement (4) Plot by presentations

CHI Subjective Ratings (1) Slide only Raw text Highlight text Video summary Synopsis Key points 41%62%64%69% Skip talk Concise Coherent Table by summarization techniques

CHI Subjective Ratings (2) Highlight text and video summary > (at p=0.01) slide only and raw text Highlight text and video summary > (at p=0.01) slide only and raw text Slide only Raw text Highlight text Video summary Synopsis Key points 41%62%64%69% Skip talk Concise Coherent

CHI Subjective Ratings (3) Slide only Raw text Highlight text Video summary Synopsis Key points 41%62%64%69% Skip talk Concise Coherent Highlight text and video summary are not significantly different (at p=0.05) Highlight text and video summary are not significantly different (at p=0.05)

CHI Subjective Ratings (4) Slide only Raw text Highlight text Video summary Synopsis Key points 41%62%64%69% Skip talk Concise Coherent Raw text, highlight text, and video summary > (at p=0.05) slide only Raw text, highlight text, and video summary > (at p=0.05) slide only

CHI User Comments 13 out of 24 like video summaries  “It is more enjoyable listening and seeing the presenter.” 11 prefer highlighted transcripts  “I liked having the option of being able to get more detailed info when I need it.” 13 out of 24 like video summaries  “It is more enjoyable listening and seeing the presenter.” 11 prefer highlighted transcripts  “I liked having the option of being able to get more detailed info when I need it.”

CHI Conclusions Effective summary techniques are key This study compared 4 summarization techniques  Slide only does not work well for most talks  Raw text transcript is hard to read  Human produced summaries work better Effective summary techniques are key This study compared 4 summarization techniques  Slide only does not work well for most talks  Raw text transcript is hard to read  Human produced summaries work better

CHI Conclusions (cont.) Slide authoring style makes a difference What to do?  For authors: tools to generate summaries  For users: interactive and intelligent video browser Slide authoring style makes a difference What to do?  For authors: tools to generate summaries  For users: interactive and intelligent video browser

CHI Compare with AutoSum Study (1) Current study and auto summary study are comparable  4 talks and quiz are the same  Both have author-generated summary  Slides are shown in all conditions for both  Evaluation methods are the same Current study and auto summary study are comparable  4 talks and quiz are the same  Both have author-generated summary  Slides are shown in all conditions for both  Evaluation methods are the same

CHI Compare with AutoSum Study (2) AutoSum Study Current Study

CHI Compare with AutoSum Study (3) SA*ASO Synopsis Key points 56%76%69%41% Skip talk Concise Coherent Subject ratings (AutoSum vs. Current)

CHI Compare with AutoSum Study (4) SA*ASO Synopsis Key points 56%76%69%41% Skip talk Concise Coherent A* in AutoSum consistently > A

CHI Compare with AutoSum Study (3) SA*ASO Synopsis Key points 56%76%69%41% Skip talk Concise Coherent Slide-based summary (S) > slide only (SO)