CEN3722 Human Computer Interaction User Testing Dr. Ron Eaglin

Objectives Define the term user testing Describe how it differs from discount methods Describe difference between performance and preference data Describe ethical concerns with user testing, and things testers should do Describe the procedure for user testing

Objectives Define the term controlled experiment Define the following types of experiments Factorial Between-groups Within-groups

User Testing Other names Goal is to determine usability problems Usability testing Performance testing Goal is to determine usability problems Difference with discount usability methods Uses real users Experimental situation Highly controlled Costs more

Types of Data Performance Preference Objective How capable users are in using the system Measures Errors Time Preference Subjective How satisfied users are with the system Rating scales Absolute versus relative

Performance vs. Preference (Bailey, 1996) Designer Preferences Average Performance Time (sec) Typed Entry 5% 9.9 1 Level Menu 18% 11.1 2 Level Menu 40% 12.2 Multiple Menus 37% 14.7

User Testing Procedure Planning Define the purpose of the test Identify set of tasks Adequately exercise the system Plan how information will be collected/recorded Decide questions for use as debriefing Prepare the test facilities Find a set of users representative of the potential user population Use user profile

User Testing Procedure Conduct the test while monitoring the users doing the tasks Record information Video Audio Written notes by experimenter Software logging

User Testing Procedure Evaluate the outcome Analyze and summarize the performance data Collate observations, comments, and questionnaire results Identify problems with the design Determine possible fixes to the design Make changes to design Make changes based on test results Retest with new set of users

Ethical Concerns with Testing Testing can be stressful and embarrassing Protecting the participants Voluntary Informed consent Testing the design NOT THE USERS Ensure privacy Video Data Explain purpose of test Users can quit at any time Debrief

Controlled Experiments True experiments A causes B Independent variables What is “manipulated” Does turning up the thermostat cause sweating? Dependent variables What we assume is affected by the Independent Variables

Controlled Experiments Experiment manipulates independent variables Observes outcome on dependent variables. Example: Change Task Flow and observe performance Control: Structure experiment so that we can infer that the only cause of change in the dependent variable was the change in the independent variable.

User Test - Example Task flow across 3 windows vs. single window

User Test - Example Select 20 users (representative sample of real users) Have 10 users in each system Measure speed for completing a task Average Time System 1 23 seconds System 2 12 seconds

Example – User Test Hypothesis Experimental conditions System 2 is faster than System 1 because users don’t have to navigate 3 windows Experimental conditions 2: System 1 and System 2 Random assignment of subjects Randomly sample 20 users, 10 randomly assigned to each system Random assignment ensures that samples within conditions are EQUAL on the relevant DV before manipulation

Experiment One independent variable with two levels The two systems, different task flow (1 versus 3 windows) One dependent variable Time to complete a task Possible alternatives include errors, satisfaction Look at mean difference between condition Differences in DV may be assumed to be caused by the difference in IV

Second User Test Same test, different results, what conclusion? Average Time System 1 15 seconds System 2

Types of Experiments Tiger Woods Arnold Palmer Ernie Els Gary Player Between-Subjects Different groups of subjects are assigned to different conditions Experimental Condition Control Condition Tiger Woods Ernie Els Payne Stewart Jack Nicklaus Arnold Palmer Gary Player John Daly Philip Mickelson

Experimental Condition Types of Experiments Within-Subjects One group of subjects which may get more than one condition Time 1 Control Condition Time 2 Experimental Condition Tiger Woods Ernie Els Payne Stewart Jack Nicklaus Tiger Woods Ernie Els Payne Stewart Jack Nicklaus 19

Factorial Designs Factorial Design Where more than one IV is manipulated Hypothesis: Font style and font size affect readability Style 1 Style 2 Size 1 12 seconds 14 seconds Size 2 26 seconds 22 seconds

Factorial Design Cell Means Marginal Means Grand Mean Look at the averages of the Dependent Variables Cell Means Marginal Means Style 1 Style 2 Mean Size 1 12 14 13 Size 2 26 22 24 19 18 18.5 Grand Mean

Objectives Define the term user testing Describe how it differs from discount methods Describe difference between performance and preference data Describe ethical concerns with user testing, and things testers should do Describe the procedure for user testing

Objectives Define the term controlled experiment Define the following types of experiments Factorial Between-groups Within-groups