Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal

Overview Evaluation is performed to determine how well a certain product design meets user needs. Need to decide what to evaluate? –Guided by goals, theory, model, etc. What to evaluate determines how to do the evaluation

Evaluation Paradigms Quick & Dirty Usability testing Field studies Predictive evaluation

Quick & Dirty Informal Designers or evaluators meet informally with users –Gather information about product design –Gather suggestions for design improvements Inexpensive Not time consuming

Usability Testing Formal assessment Measures user performance on predefined tasks –Tasks structured based on purpose of evaluation Controlled by evaluator Performance observed and/or captured –EXAMPLES? –based on questions guiding usability testing (i.e., what the evaluator wants to find)

Usability Testing Typically quantitative Interviews and questionnaires can result in qualitative assessments –User comments, quotes of likes/dislikes, etc. A mix method is ideal –WHY?

Usability Testing Not performed in a naturalistic setting Activities can be captured/recorded using software (e.g., Morae, HyperCam, Camtasia), or videotape Evaluator may take observational notes while activities being captured

Field Studies Naturalistic setting –User interacts with system as part of a daily routine –No tasks given by evaluator –Evaluator observes and records activities, OR uses software to capture activities, OR… Can be qualitative and quantitative –HOW?

Predictive Evaluation Experts place themselves in the users’ shoes to predict usability problems Guided by heuristics –Quick, inexpensive –Limitations WHAT ARE THEY?

Evaluation Techniques Observe user Gather user opinion Gather expert opinion Test user performance Model user performance Mix method (2 or more techniques)

DECIDE Framework Determine goals Explore/set questions to be answered Choose suitable paradigms and techniques Identify issues (e.g., how to recruit participants)

DECIDE Framework Decide on tackling ethical concerns (e.g., use of human subjects, privacy) Evaluate, interpret, present data