1. Usability Evaluation of the Course Management Features of Sakai Jonathan Howarth {jhowarth@vt.edu} Rex Hartson {hartson@vt.edu} Aaron Zeckoski {aaronz@vt.edu} Virginia Tech Sakai Conference, Atlanta, 2006

2. | | | | | | | 2 Overview  Introduction  Usability  Setup  Positive Feedback  Performance and Satisfaction  Usability Problems  Next Steps

3. | | | | | | | 3 Introduction  - Scholar is Virginia Tech’s implementation of Sakai  Using Scholar in a series of dissertation studies  Usability evaluation in exchange for input from Scholar developers  Summary report { http://issues.sakaiproject.org/confluence/download/attachments/3115 0/ScholarUsabEvalReportSummary.pdf?version=1} http://issues.sakaiproject.org/confluence/download/attachments/3115 0/ScholarUsabEvalReportSummary.pdf?version=1  Full report { http://issues.sakaiproject.org/confluence/download/attachments/3115 0/ScholarUsabEvalReportFull.pdf?version=1 } http://issues.sakaiproject.org/confluence/download/attachments/3115 0/ScholarUsabEvalReportFull.pdf?version=1

4. | | | | | | | 4 Usability  Examples (1)  Examples from the real world

5. | | | | | | | 5 Usability  Examples (2)  Examples from the digital world

6. | | | | | | | 6 Usability  Examples (3)  Some additional contrived examples World Usability Day posters provided by the Usability Professional’s Association { http://www.worldusabilityday.org/tools/world-usability-day-posters } http://www.worldusabilityday.org/tools/world-usability-day-posters

7. | | | | | | | 7 Usability  Examples (4)  Usability in the news

8. | | | | | | | 8 Usability  Definition (1)  Usability is a characteristic of an interactive system that indicates ease of use, usefulness  Usability includes Effectiveness Efficiency Satisfaction

9. | | | | | | | 9 Usability  Definition (2)  It is NOT "dummy proofing”  “Doing usability” is NOT (just) usability testing  Usability is NOT “user friendliness”  Misconceptions: Peanut butter theory Priest in a parachute theory

10. | | | | | | | 10 Usability  Motivation  Why should you care For the user, the UI is the system Usability can make or break your product Reduces training and help-desk costs Increases productivity Stand out among competition  An example Shutterfly increased print orders by 200% by redesigning their website { http://www.jaredresearch.com/why_it_worked.html } http://www.jaredresearch.com/why_it_worked.html

11. | | | | | | | 11 Setup (1)  Comparison with other Sakai usability evaluations { http://bugs.sakaiproject.org/confluence/display/UI/Usage+Reports } http://bugs.sakaiproject.org/confluence/display/UI/Usage+Reports  Scholar version 2.2.x  Course management features of Scholar  Goals Satisfaction Learnability Performance

12. | | | | | | | 12 Setup (2)  Walkthrough  Usability testing 5 participants with different backgrounds Sessions lasted 2 hours 17 tasks 2 questionnaires Gratuity

13. | | | | | | | 13 Positive Feedback  Variety of tools  Help system  Mechanisms for supporting collaboration

14. | | | | | | | 14 Performance and Satisfaction (1)  Performance Time on task  Long amounts of time required for initial tasks  Example – 10 minutes average to add lecture and lab events to the schedule  Much variance among participants  Improvement for repeated tasks Assists  As a group, the participants were only able to complete 2 initial tasks  Task of adding an assignment resulted in 14 assists

15. | | | | | | | 15 Performance and Satisfaction (2)  Satisfaction Scores on questionnaire  Average around a 5 (scale of 1 to 10, 1 = poor, 10 = excellent)  No change in scores between 1 st and 2 nd questionnaire  Summary Need to decrease time on task Results suggest that participants were able to learn quickly Performance/perceived performance did not have a substantial effect on satisfaction

16. | | | | | | | 16 Usability Problems (1) GroupP1P2P3P4P5Total Terminology911154948 Consistency69105737 Sections and groups21065427 Conformance with standards2851521 Assignments and gradebook assignments5163419 Workspaces and worksites2445116 Distribution of functionality2345115 Miscellaneous1112 9852  Usability problems – verbal feedback, hesitations, deviation from correct sequence of actions

17. | | | | | | | 17 Usability Problems (2)  Terminology Roster {video – explaining what a roster is},video Resources {video – what are resources?}video

18. | | | | | | | 18 Usability Problems (3)  Consistency Lists, confirmation, feedback,...

19. | | | | | | | 19 Usability Problems (4)  Sections and Groups Sections are tools (appear in the menu bar) and are exclusive Groups are not tools (appear in site info) and are not exclusive  Conformance with standards Partial saving of page state {video – trying to get to the main Site Info page}video Functioning of the browser’s back button

20. | | | | | | | 20 Usability Problems (5)  Assignments and gradebook assignments {video – working in both tools to correct a point value}video

21. | | | | | | | 21 Usability Problems (6)  Workspaces and Worksites {video – trying to explain the difference}video

22. | | | | | | | 22 Usability Problems (7)  Distribution of functionality Adding and removing students {video – separation of add and remove causes problems on repeated task}video Editing assignments {video – ability to access assignment from multiple locations}, etcvideo Example – RSmart.comRSmart.com

23. | | | | | | | 23 Usability Problems (8)  Miscellaneous problems Reoccurring schedule entries, system-centered permissions, clean Word text {video – difficulties pasting text}video

24. | | | | | | | 24 Next Steps  Usability evaluation must be iterative It’s not possible to get it right the first time (in any complex design domain) Need: Ready, Fire, Aim!

25. | | | | | | | 25 Questions ?