CS305, HW1, Spring 2008 Evaluation Assignment Part 1: Proposal (Email to horton@cs.virginia.edu by 5pm, Friday, Feb. 8) A short description of what you plan to evaluate (2-3 lines is enough) Names of group members (3-4 people) Possible participants for your evaluation Choose any software / system to evaluate as long as: You are at least somewhat familiar with it You can find two or more participant users who are not experts You can run the software / system in an experimental manner It must involve some input/interaction as well as output. (Don’t want it to be passive for participant. GUI not required) Web site? Only if it has some interesting complexity Device? Probably OK. Game? Probably OK. Ask me.

Evaluation Assignment Focus on some aspect of the software Just part of a system if it’s complex Focus on several usability goals The user interactions that implement one to three tasks that all together can be done by an expert in, say, 20 minutes Ideally a highly used or complex part of the software You choose what goals matter to you. E.g. maybe you do or don’t want to include learnability or efficiency

Evaluation Assignment Perform a co-operative evaluation and analysis of usability defects Do a dry run with someone in your team Remember that co-operative evaluation requires both you and the participant to verbalize Your participants should not be experts but should be able to use or learn the software fairly quickly If asked, you must volunteer to be someone else’s participant! Also, create a table like that presented in class to evaluate success in tasks Keeps good notes and records!

Evaluation Assignment: Your Report Turn in a report as follows on Mon., Feb, 18, 11 am A description of the specific software or system and its aspects that are being evaluated. Be sure to what the specific goals are for your evaluation! Just a subset of the system? Specific goals for this evaluation? A summary of your procedures When, where, how did you do your evaluation? (Only enough detail so I can see that you followed good procedures.) Also, what tasks did you ask the participant to do? Should include copies of the task descriptions you gave the user (see next page)

Your report (cont’d) List of all usability defects or problems by the user that you observe What happened? What usability principle is violated? OK if this has only a little detail, but feel free to add more detail if you wish A more detailed analysis of the most serious defects (at least four defects) Choose defects in the system’s design How it supports tasks, or its conceptual model, or its interaction style, or problems with specific UI elements State when error occurs in the goal/decide/execute cycle Excerpt from your notes (quotes if possible) on what happened with the user(s) in the evaluation Cause of the problem More details than given in step 3 above. Make appropriate use of concepts of usability principles etc. Any other appropriate detail from slides on usability defects Recommendations for changes in the system

Your report (cont’d) The table like Figure 12.9 (p. 610) for each task evaluated But, list high-level tasks based on a user-goal here! If something obvious, or easy, why bother? Use this to evaluate how well the system serves user needs Examples of good choices: Add new entry to phonebook Find and play specific MP3 file by name Compose and send an email Examples of bad choices: Type in phone number Pause the player (assuming there’s a pause-button) Enter the body of the email message A summary of any subjective comments made by your participants in the “wrap-up” discussion you have at the end of your evaluation.