1. CEN3722 Human Computer Interaction User Testing
Dr. Ron Eaglin

2. Objectives Define the term user testing
Describe how it differs from discount methods
Describe difference between performance and preference data
Describe ethical concerns with user testing, and things testers should do
Describe the procedure for user testing

3. Objectives Define the term controlled experiment
Define the following types of experiments
Factorial
Between-groups
Within-groups

4. User Testing Other names Goal is to determine usability problems
Usability testing
Performance testing
Goal is to determine usability problems
Difference with discount usability methods
Uses real users
Experimental situation
Highly controlled
Costs more

5. Types of Data Performance Preference Objective
How capable users are in using the system
Measures
Errors
Time
Preference
Subjective
How satisfied users are with the system
Rating scales
Absolute versus relative

6. Performance vs. Preference (Bailey, 1996)
Designer Preferences
Average Performance Time (sec)
Typed Entry 5%
9.9
1 Level Menu
18%
11.1
2 Level Menu
40%
12.2
Multiple Menus
37%
14.7

7. User Testing Procedure
Planning
Define the purpose of the test
Identify set of tasks
Adequately exercise the system
Plan how information will be collected/recorded
Decide questions for use as debriefing
Prepare the test facilities
Find a set of users representative of the potential user population
Use user profile

8. User Testing Procedure
Conduct the test while monitoring the users doing the tasks
Record information
Video
Audio
Written notes by experimenter
Software logging

9. User Testing Procedure
Evaluate the outcome
Analyze and summarize the performance data
Collate observations, comments, and questionnaire results
Identify problems with the design
Determine possible fixes to the design
Make changes to design
Make changes based on test results
Retest with new set of users

10. Ethical Concerns with Testing
Testing can be stressful and embarrassing
Protecting the participants
Voluntary
Informed consent
Testing the design NOT THE USERS
Ensure privacy
Video
Data
Explain purpose of test
Users can quit at any time
Debrief

11. Controlled Experiments
True experiments
A causes B
Independent variables
What is “manipulated”
Does turning up the thermostat cause sweating?
Dependent variables
What we assume is affected by the Independent Variables

12. Controlled Experiments
Experiment manipulates independent variables
Observes outcome on dependent variables.
Example: Change Task Flow and observe performance
Control: Structure experiment so that we can infer that the only cause of change in the dependent variable was the change in the independent variable.

13. User Test - Example
Task flow across 3 windows vs. single window

14. User Test - Example
Select 20 users (representative sample of real users)
Have 10 users in each system
Measure speed for completing a task
Average Time
System 1
23 seconds
System 2
12 seconds

15. Example – User Test Hypothesis Experimental conditions
System 2 is faster than System 1 because users don’t have to navigate 3 windows
Experimental conditions
2: System 1 and System 2
Random assignment of subjects
Randomly sample 20 users, 10 randomly assigned to each system
Random assignment ensures that samples within conditions are EQUAL on the relevant DV before manipulation

16. Experiment One independent variable with two levels
The two systems, different task flow (1 versus 3 windows)
One dependent variable
Time to complete a task
Possible alternatives include errors, satisfaction
Look at mean difference between condition
Differences in DV may be assumed to be caused by the difference in IV

17. Second User Test Same test, different results, what conclusion?
Average Time
System 1
15 seconds
System 2

18. Types of Experiments Tiger Woods Arnold Palmer Ernie Els Gary Player
Between-Subjects
Different groups of subjects are assigned to different conditions
Experimental Condition
Control Condition
Tiger Woods
Ernie Els
Payne Stewart
Jack Nicklaus
Arnold Palmer
Gary Player
John Daly
Philip Mickelson

19. Experimental Condition
Types of Experiments
Within-Subjects
One group of subjects which may get more than one condition
Time 1
Control Condition
Time 2
Experimental Condition
Tiger Woods
Ernie Els
Payne Stewart
Jack Nicklaus
Tiger Woods
Ernie Els
Payne Stewart
Jack Nicklaus 19

20. Factorial Designs Factorial Design
Where more than one IV is manipulated
Hypothesis: Font style and font size affect readability
Style 1
Style 2
Size 1
12 seconds
14 seconds
Size 2
26 seconds
22 seconds

21. Factorial Design Cell Means Marginal Means Grand Mean
Look at the averages of the Dependent Variables
Cell Means
Marginal Means
Style 1
Style 2
Mean
Size 1 12 14 13
Size 2 26 22 24 19 18
18.5
Grand Mean

22. Objectives Define the term user testing
Describe how it differs from discount methods
Describe difference between performance and preference data
Describe ethical concerns with user testing, and things testers should do
Describe the procedure for user testing

23. Objectives Define the term controlled experiment
Define the following types of experiments
Factorial
Between-groups
Within-groups