Intro to Evaluation See how (un)usable your software really is…

Why evaluation is done? Summative – assess an existing system – judge if it meets some criteria Formative – assess a system being designed – gather input to inform design Summative or formative? – Depends on maturity of system how evaluation results will be used – Same technique can be used for either

Other distinctions Form of results of obtained – Quantitative – Qualitative Who is experimenting with the design – End users – HCI experts Approach – Experimental – Naturalistic – Predictive

Evaluation techniques Predictive Evaluation Interviews Questionnaires Observation Experiment Discount Evaluation techniques – Heuristic eval – Cognitive walkthrough

Predictive Evaluation Predict user performance and usability Rules or formulas based on experimentation – Quantitative – Predictive Next class…

Interviews & Questionnaires Ask users what they think about your prototype / design / ideas – Qualitative & quantitative – Subjective – End users or other stakeholders Often accompanies other methods to get subjective feedback

Observation Watch users perform tasks with your interface – Qualitative & quantitative – Objective – Experimental or naturalistic Variations – Think-aloud – Cooperative evaluation

Experiments Test hypotheses about your interface – Quantitative – Objective – Experimental Examine dependent variables against independent variables – Often used to compare two designs or compare performance between groups Next week…

Discount usability techniques Fast and cheap method to get broad feedback – Use HCI experts instead of users – Qualitative mostly Heuristic evaluation – Several experts examine interface using guiding heuristics (like the ones we used in design) Cognitive Walkthrough – Several experts assess learnability of interface for novices

And still more techniques Diary studies – Users relate experiences on a regular basis – Can write down, call in, etc. Experience Sampling Technique – Interrupt users with very short questionnaire on a random- ish basis Good to get idea of regular and long term use in the field (real world)

A “typical” usability study Bring users into a lab Introduce them to your interface Give them a script or several tasks and ask to complete them – Look for errors & problems, performance, etc. Interview or questionnaire after to get additional feedback

Usability Lab ation_room2.htm Large viewing area in this one- way mirror which includes an angled sheet of glass the improves light capture and prevents sound transmission between rooms. Doors for participant room and observation rooms are located such that participants are unaware of observers movements in and out of the observation room.

A “typical” usability study Questionnaire (biographical data) Observation of several tasks – Sometimes as part of an experiment Interview (for additional feedback)

Evaluation is Detective Work Goal: gather evidence that can help you determine whether your usability goals are being met Evidence (data) should be: – Relevant – Diagnostic – Credible – Corroborated

Data as Evidence Relevant – Appropriate to address the hypotheses e.g., Does measuring “number of errors” provide insight into how effective your new air traffic control system supports the users’ tasks? Diagnostic – Data unambiguously provide evidence one way or the other e.g., Does asking the users’ preferences clearly tell you if the system performs better? (Maybe)

Data as Evidence Credible – Are the data trustworthy? Gather data carefully; gather enough data Corroborated – Do more than one source of evidence support the hypotheses? e.g. Both accuracy and user opinions indicate that the new system is better than the previous system. But what if completion time is slower?

General Recommendations Identify evaluation goals Include both objective & subjective data – e.g. “completion time” and “preference” Use multiple measures, within a type – e.g. “reaction time” and “accuracy” Use quantitative measures where possible – e.g. preference score (on a scale of 1-7) Note: Only gather the data required; do so with minimum interruption, hassle, time, etc.

Evaluation planning Decide on techniques, tasks, materials – What are usability criteria? – How much required authenticity? How many people, how long How to record data, how to analyze data Prepare materials – interfaces, storyboards, questionnaires, etc. Pilot the entire evaluation – Test all materials, tasks, questionnaires, etc. – Find and fix the problems with wording, assumptions – Get good feel for length of study

Recruiting Participants Various “subject pools” – Volunteers – Paid participants – Students (e.g., psych undergrads) for course credit – Friends, acquaintances, family, lab members – “Public space” participants - e.g., observing people walking through a museum – , newsgroup lists Must fit user population (validity) Note: Ethics, Consent apply to *all* participants, including friends & “pilot subjects”

Performing the Study Be well prepared so participant’s time is not wasted Explain procedures without compromising results Session should not be too long, subject can quit anytime Never express displeasure or anger Data to be stored anonymously, securely, and/or destroyed Expect anything and everything to go wrong!! (a little story)

Consent Why important? – People can be sensitive about this process and issues – Errors will likely be made, participant may feel inadequate – May be mentally or physically strenuous What are the potential risks (there are always risks)?

Data Analysis Start just looking at the data – Were there outliers, people who fell asleep, anyone who tried to mess up the study, etc.? Identify issues: – Overall, how did people do? – “5 W’s” (Where, what, why, when, and for whom were the problems?) Compile aggregate results and descriptive statistics

Making Conclusions Where did you meet your criteria? Where didn’t you? What were the problems? How serious are these problems? What design changes should be made? – But don’t make things worse… Prioritize and plan changes to the design Iterate on entire process

Example: Heather’s study Software: MeetingViewer interface fully functional Criteria – learnability, efficiency, see what aspects of interface get used, what might be missing Resources – subjects were students in a research group, just me as evaluator, plenty of time Wanted completely authentic experience

Heather’s software

Heather’s evaluation Task: answer questions from a recorded meeting, use my software as desired Think-aloud Video taped, software logs Also had post questionnaire Wrote my own code for log analysis Watched video and matched behavior to software logs

Example materials

Data analysis Basic data compiled: – Time to answer a question (or give up) – Number of clicks on each type of item – Number of times audio played – Length of audio played – User’s stated difficulty with task – User’s suggestions for improvements More complicated: – Overall patterns of behavior in using the interface – User strategies for finding information

Data representation example

Data presentation

Some usability conclusions Need fast forward and reverse buttons (minor impact) Audio too slow to load (minor impact) Target labels are confusing, need something different that shows dynamics (medium impact) Need more labeling on timeline (medium impact) Need different place for notes vs. presentations (major impact)

Your turn: assignment Week from Thursday: draft of evaluation plan – What are your goals – How you will test each one – basic idea – Early drafts of any materials tasks you want people to do, questionnaires, interview questions, etc. Part 2 due in 2.5 weeks!

Your turn: in class In your project groups – Which usability goals are important for you? – How might you measure each one?

Reminder: (some) usability goals Learnability Predictability Synthesizability Familiarity Generalizability Consistency Error prevention Recoverability Observability Responsiveness Task conformance Flexibility Customizability Substitutivity Satisfying Engaging Motivating Efficient Aesthetic