Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant Replay (AIR) Conferencing

Make meetings (and play) better with telepresence technology

Verizon Conferencing Report

Meeting Capture Cutler et al., 2002; Jian, Kim & Li, 2003; Ranjan, Birnholtz, & Balakrishnan, 2008 Summaries Video Skimming & Summarization - Smith & Kanade, 1995; He, Gupta & Grudin, 1999 ; Christel, 2006; Money & Agius, 2008; Indexing Recorded Meetings Moran et al., 1997 Time-Compression Omoigui et al., 1999; Orr, 1971; Wildemuth et al., 2003 Real-time Catchup Tucker et al., 2010

AIR Conferencing

Explore the benefits (and challenges) of Accelerated Instant Replay for meetings

18 Participants (1 female) Three conditions Background Interviews Task: Live Status Meeting 3-way Videoconference Each person gets a chance to present Each person gets interrupted 4 times and has to catch up Recall assessment (quiz on facts + explanations) Transcript-OnlyMuted ReplayFull Replay

All occasionally miss parts of meetings Current catch-up strategies (most indicated they do not want to disrupt the meeting) often just listen and try to deduce what they missed, possibly interrupt with a question if they are confused wait for a recap by the presenter browse meeting material to figure out what was missed discreetly ask someone what was missed (whisper or IM)

Useful to replay what was missed during a videoconference? Yes (10), maybe (7), no (1). “Often you miss critical conversations when you step out or are interrupted during a meeting and then you try to play catch-up during the rest of the meeting. Getting to know what was covered and who said it and the body language would put me back into the meeting very quickly.” “It would depend mostly upon the importance of the meeting, followed by the duration of how much I missed, and finally, on how discreetly I could review the video”

Full Replay enabled users to recall as much as they did when they were not interrupted. Recall Test FactsExplanations Baseline78%43% Transcript only45%16% Muted Replay40%13% Full Replay80%49%

Current System Perfect Speech-to-Text Transcript Only27 Muted Replay02 Full Replay166 – Full Replay: “it was fast, easy to concentrate and auto catch-up” “easiest to follow; leads most smoothly into rejoining live” “seems like the only way to catch up in a focused way” “fast audio was really cool …. I feel like I can cheat in time with fast forward”

Very Difficult Somewhat Difficult Neither Easy nor Difficult Somewhat Easy Very Easy Past + Present: “I can listen to audio while watching the live slide show and transcript” “you can listen to current conversations and read transcripts”

1. How important were the additional modalities (over just audio alone)? 2. How successful would the system be with perfect speech-to-text transcription? 3. How can we better support multitasking?

Examining individual modalities and impact of perfect speech-to-text transcription

Videos Shared Desktop Speech- to-text Transcript

58 Participants (25 female) Pre-recorded status meeting Mixed Design Everyone did audio only + 1 other condition Focused review (no divided attention)

Transcript only Audio only Audio + Transcript Audio + Video Audio + Shared Workspace Audio + All (full replay)

RECALLFactExplanationIdentification Live93%77%84% Audio Only74%50%61% Enhanced Audio83% 1 62% 1 72% 1 1 Significantly better than audio only but worse than live

RECALLFactExplanationIdentification Live93%77%84% Audio Only74%50%61% Audio + All89% 1 70% 1 81% 1 Audio + Workspace90% 1 65%72% Audio + Transcript83%67% 2 66% Audio + Video70%48%68% 1 Significantly better than audio only and no difference with live 2 Significantly better than audio only but worse than live

RECALLFactExplanationIdentification Live93%77%84% Audio Only74%50%61% Audio + All89% 1 70% 1 81% 1 Audio + Workspace90% 1 65%72% Audio + Transcript83%67% 2 66% Audio + Video70%48%68% 1 Significantly better than audio only and no difference with live 2 Significantly better than audio only but worse than live Users were significantly more confident with their answers in the Audio + All condition (p<.05 ).

RECALLFactExplanationIdentification Live93%77%84% Audio Only74%50%61% Audio + All89% 1 70% 1 81% 1 Audio + Workspace90% 1 65%72% Audio + Transcript83%67% 2 66% Audio + Video70%48%68% 1 Significantly better than audio only and no difference with live 2 Significantly better than audio only but worse than live “I used the audio to see who was talking, and every time I missed something, I had the transcript which kept a recording of everything and I could just look back at it.” “Being able to see who was talking during catch-up helped to associate a face, name, and voice with the answers given.”

RECALLFactExplanationIdentification Audio Only71%50%56% Transcript Only77% 1 52%43% 2 1 Significantly better than audio only 2 Significantly worse than audio only

RECALLFactExplanationIdentification Audio Only71%50%56% Transcript Only77% 1 52%43% 2 1 Significantly better than audio only 2 Significantly worse than audio only Transcript alone was way too fast to really understand what was going on. I could kind of skim, and I got some information but I felt like I was just bouncing along.

Enhanced audio catch-up is superior to audio-only review. Users preferred, felt more confident with, and performed better with enhanced-audio than audio-only review. As good as live! Results consistent between two studies Audio + All showed the strongest benefit Audio + Shared Workspace improved recall of facts Audio + Transcript improved recall of explanations Speech-to-Text transcript not as beneficial as expected

Cost/benefit tradeoff: importance of information vs. overhead Ensure seamless use so as to not detract from the meeting Next Steps: Enhanced timeline index Multi-tasking AIR for face-to-face meetings (with mobile phones/laptops & headsets)

Replaying life should be “a given” But sometimes participation in real-time is important, so we need to enable catch-up

Questions?