Evaluation Paradigms & Techniques IS 588 Spring 2007 Drs. Bilal & Normore

Overview Evaluation is performed to determine how well a certain product design meets user needs. Need to decide what to evaluate? Guided by goals, theory, model, etc. What to evaluate determines how to do the evaluation

Evaluation Paradigms Quick & Dirty Usability testing Field studies Predictive evaluation

Quick & Dirty Informal Designers or evaluators meet informally with users Gather information about product design Gather suggestions for design improvements Inexpensive Not time consuming

Usability Testing Formal assessment Measures user performance on predefined tasks Tasks structured based on purpose of evaluation Controlled by evaluator Performance observed and/or captured EXAMPLES? based on questions guiding usability testing (i.e., what the evaluator wants to find)

Usability Testing Typically quantitative Interviews and questionnaires can result in qualitative assessments User comments, quotes of likes/dislikes, etc. A mix method is ideal WHY?

Usability Testing Not performed in a naturalistic setting Activities can be captured/recorded using software (e.g., HyperCam, Camtasia), or videotaped Evaluator may take observational notes while activities being captured

Usability Testing Demo of HyperCam Recording and replaying activities Demo of a child’s captured activities in using the ICDL Based on Bilal’s 2006/2007

Field Studies Naturalistic setting Can be qualitative and quantitative User interacts with system as part of a daily routine No tasks given by evaluator Evaluator observes and records activities, OR uses software to capture activities, OR… Can be qualitative and quantitative HOW?

Predictive Evaluation Experts place themselves in the users’ shoes to predict usability problems Guided by heuristics Quick, inexpensive Limitations WHAT ARE THEY?

Evaluation Techniques Observe user Gather user opinion Gather expert opinion Test user performance Model user performance Mix method (2 or more techniques)

DECIDE Framework Determine goals Explore/set questions to be answered Choose suitable paradigms and techniques Identify issues (e.g., how to recruit participants)

DECIDE Framework Decide on tackling ethical concerns (e.g., use of human subjects, privacy) Evaluate, interpret, present data

Class Activity Work in a group Examine/scan Bilal’s study (parts I & 2) Determine/extract the components of DECIDE framework, as applicable Give an example of each existing component, as applicable.