User Testing Results October 09

1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google Videos 3.4 TED 4. Recommendations User Testing Results

1. Goals and approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations 1. Goals and approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations User Testing Results

Goals and Approach Goal: Evaluate the usability of the following media players: Virtpresenter 173_10_32&related=studipuos&token=fb2eb23d9d20d662d508f95cb180eb98#app=e43a&81f8-selectedIndex=1 Youtube Google videos AaB5ZDgBg&q=tedtalks TED

Goals and Approach Approach: Systematically observe potential users of the examined media players performing representative, determinate tasks. Goal: Evaluate the effectiveness, efficiency, and satisfaction of Virtpresenter, Google Videos, YouTube, and TED. The participants were given a series of tasks to complete with each player and were asked follow up questions directed towards understanding: whether the structure of the players are easy to understand whether users can find the necessary information to complete their tasks whether the user can navigate the content easily

Participants were comprised of 5 students from The Open University of Catalonia (UOC). Each participant completed the tasks individually in minute sessions. Study was conducted on October 3 rd at the University Autónoma. Goals and Approach

1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations 1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations User Testing Results

Task Summary Each user was given the a number of scenarios with corresponding tasks. While completing the tasks, the participants were observed closely and each task was attributed either a success or failure score. Next we present a table displaying the tasks and their corresponding results.

Task Summary Task Success Ratios [Virtpresenter]

Task Summary Task Success Ratios [YouTube]

Task Summary Task Success Ratios [Google Videos]

Task Summary Task Success Ratios [TED]

1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations 1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations User Testing Results

Results (Virtpresenter) Scenario 1 You have registered for a class and found out that all the lectures are available online. You have opened the link to the lecture viewer with the intention of watching the first day of class. Task 1.1 – Begin watching the first lecture and watch for 15 seconds. 100%0% Had Little to No Difficulty (5 users): All users clicked the large play button found in the video screen. None of the participants noticed the play button next to the scrubber. One user commented that he would like to download the video to watch on his computer at another time but he couldn't find this button (doesn’t exist). “I want to download the video to watch whenever and wherever I want”. User 4. Watch video

Results (Virtpresenter) Scenario 2 You just received a phone call that you need to take. Task 2.1 – Pause the lecture so you can take your call. 100%0% Had Little to No Difficulty (5 users): 3 participants clicked the larger pause button in the video section [button not visible until video section is hovered over] 2 participants clicked the smaller pause button next to the scrubber. They didn’t notice the large button in the video section] “If the icons would be under the video, like in YouTube, everybody would se them”. User 4. Watch video

Results (Virtpresenter) Scenario 3 You will need to meet the person you were talking to in an hour meaning you need to leave in 30 minutes. You are curious whether you will be able to finish the lecture before you need to leave. Task 3.1 – Find out how much of the lecture you have watched and how much is remaining. 80%20% Had Little to No Difficulty (4 users): Participants found the time quickly and without problems. When they tried to move the timeline, they didn't distinguish the blue circle from the orange square. They didn’t understand the concept of “bookmarks” and they didn’t understand the two oranges squares. They believed the orange square was the playhead because it is positioned on top of the scrubber [leading them to think they could move it]. Other students believed this was a delimiter and the video would stop at the last orange box. They expected to be able to type the numbers into the timecode. Had Difficulty (1 user): Didn’t see the numbers and didn’t understand the system of blue circles and oranges squares. Watch video

Results (Virtpresenter) Scenario 4 Now that you’ve realized you won’t have enough time to watch the entire lecture, you decide to skip to the __________ section which you think is the most important. You’ll watch the rest when you get home tonight. Task 4.1 – Begin watching the __________ section of the lecture. 40%60% Had Little to No Difficulty (2 users): They had a hard time finding where to look. Two participants tried using the forward and backward buttons found above the scrubber but became confused. One participant tried dragging the scrubber but it took a while for them to find the playhead; they expected to see it larger and at the top of the scrubber. Had Difficulty (3 users): They did not see the forward and back buttons. They did not see the accordion under the video. No participants understood the chapters located in the scrubber and did not realize that it was possible to browse through them. “Maybe if the accordion were in a different color I would notice it. I prefer to move with the scrubber”. User 4

Results (Virtpresenter) Scenario 5 Your roommate is listening to his music loudly and you are having trouble concentrating. You would like to turn the volume up and make the slides bigger so you can focus on the lecture. Task 5.1 – Turn up the volume and make the video of the professor larger and continue watching. [no applications] Had Little to No Difficulty (5 users): They expected to find a button like on other players that would open full screen. They thought this full screen was not big. Two users found the icons plus and minus on the video, but they associated them with volume. When clicking on them, didn’t understand the functionality and when they reached the big video screen, they did not know how to get back. They expected to find a volume icon as on other players. “This has confused me. I thought that it was the volume. I expect an icon for “full screen" that makes the video big ”. User 1. Watch video

1. Goals and approach 2. Summary of tasks and dysfunctions detected 3. Task’s Details 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Improvement opportunities 1. Goals and approach 2. Summary of tasks and dysfunctions detected 3. Task’s Details 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Improvement opportunities User Testing Results

Results (Youtube) Scenario 1 You cannot hear (or the video is in the wrong language) and you would like captions. Task 1.1 – Turn on the captions so you can read what is being said. 60%40% Had Little to No Difficulty (3 users): Had difficulty finding it and needed a lot of time to find the button. All of them said that the captions icon wasn’t intuitive and the reason they spent lot of time to find it. Had Difficulty (2 users): One of them found the icon when she tried to find the full screen button. In general, she didn't understand the buttons. The other user couldn’t find the icon. “The icon and interaction is not intuitive”. User 2. Watch video

Results (Youtube) Scenario 2 You received a phone call and would like to pause the video. Task 2.1 – Pause and play the video. 100%0% Had Little to No Difficulty (5 users): Three participants used the pause button in the scrubber. Two participants clicked the video to pause the content. Watch video

Results (Youtube) Scenario 3 The video is too small and you cannot see any details. Task 3.1 – Make the video larger so it takes up the entire screen. 100%0% Had Little to No Difficulty (5 users): They found the full screen button very quickly. When they needed to close the window, they couldn’t find the icon. They associated the icon (X) with the action of closing the entire window; they assumed it would close the entire browser. One user closed the full screen by double-clicking on the video. “This icon is for closing youtube”. User 3. Watch video YouTube (Close Full Screen) YouTube (Close Full Screen Hover)

Results (YouTube) Scenario 4 Your roommate is playing their music too loudly and you can’t hear the video. Task 4.1 – Turn up the volume to its maximum level. 100%0% Had Little to No Difficulty (5 users): All users knew how to use the volume icon. They knew if they clicked the button, the volume would mute. They know that dragging would raise and lower the volume. Watch video

Results (YouTube) Scenario 5 You have already seen the first half of the video and would like to watch the second half. Task 5.1 – Jump to the second half of the video and watch it from there. 100%0% Had Little to No Difficulty (5 users): They found it dragging the scrubber to the X point. Two students tried writing in the “timecard box” “I like that I can see the time dragging in the scrubber”. User 4. Watch video

1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations 1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations User Testing Results

Results (Google videos) Scenario 1 You cannot hear (or the video is in the wrong language) and you would like captions. Task 1.1 – Turn on the captions so you can read what is being said. 100%0% Had Little to No Difficulty (5 users): It wasn’t hard to find it. All users said this icon was more intuitive than the one found in YouTube. “I prefer this icon more than the other one”. User 2. Watch video GoogleYouTube

Results (Google videos) Scenario 2 You received a phone call and would like to pause the video. Task 2.1 – Pause and play the video. 100%0% Had Little to No Difficulty (5 users): All users used the icons in the scrubber. Some users paused the media by clicking the video. Watch video

Results (Google videos) Scenario 3 The video is too small and you cannot see any details. Task 3.1 – Make the video larger so it takes up the entire screen. 100%0% Had Little to No Difficulty (5 users): All participants found the button very quickly. All the users said that this icon was easier to understand than the one in YouTube. When trying to exit full screen, three participants hit the Escape key but were asked to only use the interface. GoogleYouTube YouTube Hover

Results (Google videos) Scenario 4 Your roommate is playing their music too loudly and you can’t hear the video. Task 4.1 – Turn up the volume to its maximum level. 100%0% Had Little to No Difficulty (5 users): All the users knew how to use the volume icon. They knew clicking would mute the volume. They know that dragging will raise or lower the volume. They prefer the YouTube icon more than this one. Watch video GoogleYouTube

Results (Google videos) Scenario 5 You have already seen the first half of the video and would like to watch the second half. Task 5.1 – Jump to the second half of the video and watch it from there. 100%0% Had Little to No Difficulty (5 users): Three found it by dragging the scrubber to the X point. Two participants used the video preview images

1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations 1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations User Testing Results

Results (TED) Scenario 1 You cannot hear (or the video is in the wrong language) and you would like captions. Task 1.1 – Turn on the captions so you can read what is being said. 100%0% Had Little to No Difficulty (5 users): For three of them, this was more difficult than both YouTube and Google Videos because they expected to have all the icons in the same place. They liked it because it’s not an icon but they still wanted the Google Videos icon for captions and this one for subtitles. “The icon and interaction is not intuitive”. User 2. Watch video YouTubeTED

Results (TED) Scenario 2 You received a phone call and would like to pause the video. Task 2.1 – Pause and play the video. 100%0% Had Little to No Difficulty (5 users): All participants used the icons in the scrubber and found it very quickly. Watch video

Results (TED) Scenario 3 The video is too small and you cannot see any details. Task 3.1 – Make the video larger so it takes up the entire screen. 100%0% Had Little to No Difficulty (5 users): Participants had some difficulty finding the full screen button. All of them said they expect to have this button at the same level than the others, like YouTube. “Oh! It´s here! I didn’t expect to find it here!”. User 3. Watch video TEDYouTube

Results (TED) Scenario 4 Your roommate is playing their music too loudly and you can’t hear the video. Task 4.1 – Turn up the volume to its maximum level. 100%0% Had Little to No Difficulty (5 users): All participants knew how to use the volume button. They thought it wasn’t easy to adjust the volume. “I want to play and pause the video by clicking the video screen”. User 3.

Results (TED) Scenario 5 You have already seen the first half of the video and would like to watch the second half. Task 5.1 – Jump to the second half of the video and watch it from there. 100%20% Had Little to No Difficulty (5 users): All participants dragged the scrubber to the X point. One participant tried to type in the timecode. Watch video

1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations 1. Goals and Approach 2. Task Summary 3. Results 3.1 Virtpresenter 3.2 YouTube 3.3 Google videos 3.4 TED 4. Recommendations User Testing Results

Recommendations Based in the user test we present the following recommendations for improvement that some should be made to resolve of the most severe problems that we found during the usability test carried out on:

Recommendations Play and Pause -Location: the majority of participants clicked the play button found in the video section and the pause button found in the scrubber (aka controls bar). The Matterhorn media player should offer both options. - Video Section Button Visibility: users are reluctant to have elements that obstruct the video content while they are watching. It is not recommend to show buttons in this section unless their cursor is in the center. Only have the Pause button visible when the cursor is directly over it. Scrubber - Participants did not understand the difference between the playhead and the bookmark sliders. Consider separating this functionality. - Some participants wanted to be able to specify the timecode by typing it in. Include editable timecode box so users may jump to specific sections of the content. - The users liked the dynamic tooltip that appears while dragging the scrubber. This made it easy for them to identify the part of the lecture to which they are navigating. Include a timecode tooltip while playhead is dragged across the scrubber.

Recommendations Full screen - Some participants expected the screen-size buttons to adjust the volume. They expected to find a full screen button on the lower right. - In Virtpresenter, some participants expected the screen-size buttons to adjust the volume. Follow common conventions of full screen button such as YouTube and Google Videos. - Participants expected to have the functionality of making the video take up the entire screen with no controls visible. Include a full screen implementation like YouTube. Volume - Users preferred collapsible volume controls (YouTube) over slider (Google Videos). Utilize collapsible control convention for volume. Slider - Users were confused by the chapter sections in the scrubber. Consider a different implementation or inclusion of color to make this functionality more apparent. - Users were confused and unsure what to expect by the playback navigation arrows. Group and place controls appropriately to ensure transparency.

Recommendations Captions - Of the two versions of captions provided, users prefer the first [A] which overlaid text on the video content. Have captions overlay video content and include a semitransparent character background to improve contrast and therefore readability. [A] Inside de video screen. [B] Outside de video screen. YouTubeTED

Recommendations Other Suggestions - Students want the opportunity to download the video and view it offline. - Students do not want the video to begin playing on page load. Do not have video play automatically when the page loads. - Control buttons should be kept together and not separated like the full screen button and subtitles button in TED. Organize controls and keep them together. - They do not understand the concept of bookmarking. Consider redesigning this interaction. - Participants did not search inside the accordion. Reconsider placement and design of accordion.