Usability Evaluation of the Course Management Features of Sakai Jonathan Howarth Rex Hartson Aaron Zeckoski Virginia Tech Sakai Conference, Atlanta, 2006

| | | | | | | 2 Overview  Introduction  Usability  Setup  Positive Feedback  Performance and Satisfaction  Usability Problems  Next Steps

| | | | | | | 3 Introduction  - Scholar is Virginia Tech’s implementation of Sakai  Using Scholar in a series of dissertation studies  Usability evaluation in exchange for input from Scholar developers  Summary report { 0/ScholarUsabEvalReportSummary.pdf?version=1} 0/ScholarUsabEvalReportSummary.pdf?version=1  Full report { 0/ScholarUsabEvalReportFull.pdf?version=1 } 0/ScholarUsabEvalReportFull.pdf?version=1

| | | | | | | 4 Usability  Examples (1)  Examples from the real world

| | | | | | | 5 Usability  Examples (2)  Examples from the digital world

| | | | | | | 6 Usability  Examples (3)  Some additional contrived examples World Usability Day posters provided by the Usability Professional’s Association { }

| | | | | | | 7 Usability  Examples (4)  Usability in the news

| | | | | | | 8 Usability  Definition (1)  Usability is a characteristic of an interactive system that indicates ease of use, usefulness  Usability includes Effectiveness Efficiency Satisfaction

| | | | | | | 9 Usability  Definition (2)  It is NOT "dummy proofing”  “Doing usability” is NOT (just) usability testing  Usability is NOT “user friendliness”  Misconceptions: Peanut butter theory Priest in a parachute theory

| | | | | | | 10 Usability  Motivation  Why should you care For the user, the UI is the system Usability can make or break your product Reduces training and help-desk costs Increases productivity Stand out among competition  An example Shutterfly increased print orders by 200% by redesigning their website { }

| | | | | | | 11 Setup (1)  Comparison with other Sakai usability evaluations { }  Scholar version 2.2.x  Course management features of Scholar  Goals Satisfaction Learnability Performance

| | | | | | | 12 Setup (2)  Walkthrough  Usability testing 5 participants with different backgrounds Sessions lasted 2 hours 17 tasks 2 questionnaires Gratuity

| | | | | | | 13 Positive Feedback  Variety of tools  Help system  Mechanisms for supporting collaboration

| | | | | | | 14 Performance and Satisfaction (1)  Performance Time on task  Long amounts of time required for initial tasks  Example – 10 minutes average to add lecture and lab events to the schedule  Much variance among participants  Improvement for repeated tasks Assists  As a group, the participants were only able to complete 2 initial tasks  Task of adding an assignment resulted in 14 assists

| | | | | | | 15 Performance and Satisfaction (2)  Satisfaction Scores on questionnaire  Average around a 5 (scale of 1 to 10, 1 = poor, 10 = excellent)  No change in scores between 1 st and 2 nd questionnaire  Summary Need to decrease time on task Results suggest that participants were able to learn quickly Performance/perceived performance did not have a substantial effect on satisfaction

| | | | | | | 16 Usability Problems (1) GroupP1P2P3P4P5Total Terminology Consistency Sections and groups Conformance with standards Assignments and gradebook assignments Workspaces and worksites Distribution of functionality Miscellaneous  Usability problems – verbal feedback, hesitations, deviation from correct sequence of actions

| | | | | | | 17 Usability Problems (2)  Terminology Roster {video – explaining what a roster is},video Resources {video – what are resources?}video

| | | | | | | 18 Usability Problems (3)  Consistency Lists, confirmation, feedback,...

| | | | | | | 19 Usability Problems (4)  Sections and Groups Sections are tools (appear in the menu bar) and are exclusive Groups are not tools (appear in site info) and are not exclusive  Conformance with standards Partial saving of page state {video – trying to get to the main Site Info page}video Functioning of the browser’s back button

| | | | | | | 20 Usability Problems (5)  Assignments and gradebook assignments {video – working in both tools to correct a point value}video

| | | | | | | 21 Usability Problems (6)  Workspaces and Worksites {video – trying to explain the difference}video

| | | | | | | 22 Usability Problems (7)  Distribution of functionality Adding and removing students {video – separation of add and remove causes problems on repeated task}video Editing assignments {video – ability to access assignment from multiple locations}, etcvideo Example – RSmart.comRSmart.com

| | | | | | | 23 Usability Problems (8)  Miscellaneous problems Reoccurring schedule entries, system-centered permissions, clean Word text {video – difficulties pasting text}video

| | | | | | | 24 Next Steps  Usability evaluation must be iterative It’s not possible to get it right the first time (in any complex design domain) Need: Ready, Fire, Aim!

| | | | | | | 25 Questions ?