Evaluation ECE 695 Alexander J. Quinn March 30, 2018

Types of evaluation Lab study in-person Log analysis remote think aloud protocol Log analysis remote A-B testing Remote observation remote monitor screen Ethnography in-person, in-situ embed with users for extended period of time Interviews in-person e.g., contextual interview Cognitive walk-through no users Heuristic evaluation no users

Jakob Nielsen’s heuristics Match the real world Consistency & standards Help & documentation User control & freedom Visibility of system status Flexibility & efficiency Error prevention Recognition, not recall Error reporting, diagnosis, and recovery Aesthetic & minimalist design … and/or your own heuristics // Credit: https://www.nngroup.com/articles/ten-usability-heuristics/

Ben Shneiderman’s Golden Rules Consistency Shortcuts Feedback Dialog closure Error prevention and easy recovery Undo User in control Reduce short-term memory load Credit: https://www.cs.umd.edu/users/ben/goldenrules.html + http://web.mit.edu/6.813/www/sp18/classes/17-heuristic-evaluation/

Heuristic evaluation Evaluators are experts ≥ 5 evaluators 1-2 hours to examine application Rate severity on a controlled scale Severity I don't agree that this is a usability problem at all 1 Cosmetic problem only: need not be fixed unless extra time is available on project 2 Minor usability problem: fixing this should be given low priority 3 Major usability problem: important to fix, so should be given high priority 4 Usability catastrophe: imperative to fix this before product can be released https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/

Exercise We will do a simplified heuristic evaluation. Go to your app store and download any free “outliner” app on your phone Task: Create an outline with at least 3 items Use the evaluation form to note flaws found Note: In proper heuristic evaluation, we would have ≥5 evaluators per app spending 1-2 hours Severity I don't agree that this is a usability problem at all 1 Cosmetic problem only: need not be fixed unless extra time is available on project 2 Minor usability problem: fixing this should be given low priority 3 Major usability problem: important to fix, so should be given high priority 4 Usability catastrophe: imperative to fix this before product can be released