1. Evaluation How do we test the interaction design? Several Dimensions
Qualitative vs. Quantitative assessments
Conceptual vs. Physical Design

2. Why Evaluate Five good reasons:
Problems fixed before product is shipped
Team can concentrate on real (not imaginary) problems
Engineers can develop code instead of debating about their personal preferences
Time to market is sharply reduced
Solid, tested design will sell better

3. When to Evaluate Formative Evaluations Summative Evaluations
Conducted during requirements specification and design
Consider alternatives
Summative Evaluations
Assess the success of a finished product
Determine whether product satisfies requirements

4. What to Evaluate
A huge variety of User Interaction features can (and should) be evaluated, such as:
Sequence of links in a web search
Enjoyment experienced by game users
System response time
Signal detection performance in data analysis
Gould’s principles:
Focus on users and their tasks
Observe, measure, and analyze user performance
Design iteratively

5. Qualitative Assessment
Informal
Simply ask users how they like the system
Listen to “hallway” conversations about systems
Formal
Develop survey instruments to ask specific questions, e.g.
How long did it take you to become comfortable?
Which task is the most difficult to accomplish?
Hold focus group discussions about system

6. Quantitative Assessment
Identify Usability Criteria (from Requirements) to test
Design human performance experiments to test these, e.g.
Measure response time or time to complete a task
Measure error rate or incidence of “dead end”
This can be used during the design process to compare alternative designs

7. An Evaluation Framework
Evaluation must be an intentional, planned process
ad hoc evaluations are of very little value
The details of the particular framework can vary from team to team
What is important is that the framework be crafted in advance, and that all team members understand the framework

8. Evaluation Paradigms
Evaluation Paradigms are the beliefs and practices (perhaps underpinned by theory) that guide a user study
We’ll discuss four core evaluation paradigms:
Quick and Dirty evaluation
Usability Testing
Field Studies
Predictive Evaluation

9. Quick and Dirty Informal feedback from users
Can be conducted at any stage
Emphasis is on speed, not quality
Often consultants are used as surrogate users

10. Usability Testing
Measuring typical users’ performance on carefully prepared tasks that are typical for the system
Metrics can include such things as
Error rate and time to completion
Observations/recordings/logs of interaction
Questionnaires
Strongly controlled by the evaluator

11. What is usability? An Operational Definition As well as Efficient
Effective
Safe
Easy
To learn
To remember
To use
Productive
As well as
Satisfying
Enjoyable
Pleasing
Motivating
Fulfilling

12. Field Studies Done in natural settings
Try to learn what users do and how
Artifacts are collected
Video, notes, sketches, &c
Two approaches:
As an outsider looking on
Qualitative techniques used to gather data
Which may be analyzed qualitatively or quantitatively
As an insider
Easier to capture role of social environment

13. Predictive Evaluation
Uses models of typical users
Heuristic or theoretical
Users themselves need not be present
Cheaper, faster
Tried and true heuristics can be useful
E.g. speak the users’ language

14. Evaluation Techniques
Observing users
Asking users their opinions
Asking experts their opinions
Testing users’ performance
Modeling users’ task performance to predict efficacy of the interface

15. Techniques vs. Paradigms
Models used to predict efficacy
N/A
Modeling user’s performance
Can measure performance, but difficult
Test typical users, typical tasks
User testing
Heuristics early in design
Provide critiques
Asking experts
May interview
Questionnaires & interviews
Discussions, focus groups
Asking users
Central technique
Video and logging
See how users behave
Observing Users
Predictive
Field Studies
Usability Testing
Quick and Dirty
Techniques
Evaluation Paradigms

16. DECIDE Determine the overall goals that the evaluation addresses
Explore the specific questions to be answered
Choose the evaluation paradigm and techniques
Identify the practical issues that must be addressed
Decide how to deal with the ethical issues
Evaluate, interpret, and present the data

17. Determine the goals What are the high-level goals of the evaluation?
Who wants the evaluation and why?
Should guide the evaluation, i.e.:
Check that evaluators understood users’ needs
Identify the metaphor under the design
Ensure that interface is consistent
Investigate degree to which technology influences working practices
Identify how the interface of an existing product could be engineered to improve its usability

18. Explore the questions
This amounts to hierarchical question development:
“Is the user interface good?”
“Is the system easy to learn?”
“Are functions easy to find?”
“Is the terminology confusing?”
“Is response time too slow?”
“Is login time too slow?”
“Is calculation time too slow?” …

19. Choose the evaluation paradigm and techniques
Choosing one or more evaluation paradigms
Can use different paradigms in different stages
Can use multiple paradigms in a single stage
Combinations of techniques can be used to obtain different perspectives

20. Identify the practical issues
Practical issues must be considered BEFORE beginning any evaluation
Users
Adequate number of representative users must be found
Facilities and equipment
How many cameras? Where? Film?
Schedule and budget
Both always less than would be ideal
Expertise
Assemble the correct evaluation team

21. Decide how to deal with ethical issues
Experiments involving humans must be conducted within strict ethical guidelines
Tell participants the goals and what will happen
Explain that personal information is confidential
They’re free to stop at any time
Pay subjects when possible: formal relationship
Avoid using quotes that reveal identity
Ask users’ permission to quote them, show them the report
Example: Yale shock experiment

22. Evaluate, interpret and present the data
What data to collect, how to analyze them
Questions need to be asked
Reliability: is it reproducible?
Validity: measures what it’s supposed to
Biases: biases cause distortion
Scope: how generalizable?
Ecological validity: how important is the evaluation environment – does it match the real environment of interest?

23. Stopped Here

24. Observing Users
Ethnography – observing the social environment and recording observations which help to understand the function and needs of the people in it
Users can be observed in controlled laboratory conditions or in natural environments in which the products are used – i.e. the field

25. Goals, questions and paradigms
Goals and questions should guide all evaluation studies
Ideally, these are written down
Goals help to guide the observation because there is always so much going on

26. What and when to observe
Insider or outsider?
Laboratory or field?
Control vs. realism
What times are critical times (especially for field observations)?

27. Approaches to observation
Quick and dirty
Informal
Insider or outsider
Observation in usability testing
Formal
Video, interaction logs, performance data
Outsider
Observation in field studies
Outsider, participant, or ethnographer (participant or not?)

28. How to observe
Techniques of observation and data gathering vary

29. In controlled environments
Decide location
Temporary lab in user’s environment?
Remote laboratory?
Equipment
Hard to know what user is thinking
“Think Aloud” technique
But speaking can alter the interaction
Having two subjects work together can help

30. In the field Who is present? What is happening?
What are their roles?
What is happening?
Include body language, tone
When does activity occur?
Where is it happening?
Why is it happening
How is the activity organized?

31. Participant observation and ethnography
In this case, the observer/evaluator must be accepted into the group
Honesty about purpose is important both ethically and to gain trust
Disagreement in the field about the distinction between ethnography and participant observation
Do ethnographers begin with any assumptions?

32. Data collection Notes plus still camera
Audio recording plus still camera
Video

33. Indirect observation: tracking users’ activities
Diaries
Interaction logging

34. Analyzing, interpreting, and presenting data
Observation produces large quantities of data of various types
How to analyze and interpret depends on the research questions first developed

35. Qualitative analysis to tell a story
The ensemble of data (notes, video, diaries, &c) are used to help designers, as a team, understand the users
There is much room for evaluator bias in these techniques

36. Qualitative analysis for categorization
A taxonomy can be developed into which user’s behaviors can be placed
This can be done by different observers, with the discrepancies used as a measure of observer bias

37. Quantitative data analysis
Observations, interaction logs, and results are gathered and quantified
Counted
Measured
Analysis using statistical reasoning can be used to draw conclusions
What is statistical significance?
What is a T-test?

38. Case Study: Evaluating a Data Representation

39. Feeding the findings back into design
Ideally, the design team will participate in post-evaluation discussions of qualitative data
Reports to designers should include artifacts, such as quotes, anecdotes, pictures, video clips
Depending on the design team, quantitative data may or may not be compelling