1. Evaluation Paradigms & Techniques
IS 588
Spring 2007
Drs. Bilal & Normore

2. Overview
Evaluation is performed to determine how well a certain product design meets user needs.
Need to decide what to evaluate?
Guided by goals, theory, model, etc.
What to evaluate determines how to do the evaluation

3. Evaluation Paradigms Quick & Dirty Usability testing Field studies
Predictive evaluation

4. Quick & Dirty
Informal
Designers or evaluators meet informally with users
Gather information about product design
Gather suggestions for design improvements
Inexpensive
Not time consuming

5. Usability Testing Formal assessment
Measures user performance on predefined tasks
Tasks structured based on purpose of evaluation
Controlled by evaluator
Performance observed and/or captured
EXAMPLES?
based on questions guiding usability testing (i.e., what the evaluator wants to find)

6. Usability Testing Typically quantitative
Interviews and questionnaires can result in qualitative assessments
User comments, quotes of likes/dislikes, etc.
A mix method is ideal
WHY?

7. Usability Testing Not performed in a naturalistic setting
Activities can be captured/recorded using software (e.g., HyperCam, Camtasia), or videotaped
Evaluator may take observational notes while activities being captured

8. Usability Testing Demo of HyperCam
Recording and replaying activities
Demo of a child’s captured activities in using the ICDL
Based on Bilal’s 2006/2007

9. Field Studies Naturalistic setting Can be qualitative and quantitative
User interacts with system as part of a daily routine
No tasks given by evaluator
Evaluator observes and records activities, OR uses software to capture activities, OR…
Can be qualitative and quantitative
HOW?

10. Predictive Evaluation
Experts place themselves in the users’ shoes to predict usability problems
Guided by heuristics
Quick, inexpensive
Limitations
WHAT ARE THEY?

11. Evaluation Techniques
Observe user
Gather user opinion
Gather expert opinion
Test user performance
Model user performance
Mix method (2 or more techniques)

12. DECIDE Framework Determine goals Explore/set questions to be answered
Choose suitable paradigms and techniques
Identify issues (e.g., how to recruit participants)

13. DECIDE Framework
Decide on tackling ethical concerns (e.g., use of human subjects, privacy)
Evaluate, interpret, present data

14. Class Activity Work in a group
Examine/scan Bilal’s study (parts I & 2)
Determine/extract the components of DECIDE framework, as applicable
Give an example of each existing component, as applicable.