Testing and Modeling Users Kristina Winbladh & Ramzi Nasr

Usability Usability means that the people who use the product can do so quickly and easily to accomplish their own tasks We want to build usable products How do we know whether our product is usable? Testing

Usability Testing Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Planning a Usability Test How long does usability testing take? –Important for clients and managers to know –Development does not have to stop Time factors –Complexity of the product? –How much of the product is to be tested? –Number of participants? –Test preparation? –Test evaluation/report?

Examples Microsoft –8-12 weeks –Testers work together with developers –Formal reports General Electric Information Services –4-6 weeks –3 days to determine participants and tasks –Week 1&2 test: scenario & questionnaire design –End of week 2: pilot test –Week 3&4: conducting the test w 12 participants –Week 5: test data analysis –Week 6: finished the report

Can you do it faster? 1 Week –In-house usability specialists conduct test –Each test covers only a few features –Participants background is already known –Tasks are short –Generate short report 1.5 days… –Special case

What can you do to make testing successful? Planning! Starting testing w/o thinking about: –What aspects might not be as usable as they should be? –How well do the participants represent the actual users? –What tasks will we ask them to do in the short time they have? –What information are we going to collect? –How are we going to analyze the information? –What are we going to do with the analyzed information?

The steps Define the goals and concerns that are driving the test Decide who should be participants Recruit participants Select and organize tasks to test Create task scenarios Decide how to measure usability Prepare other materials Prepare testing environment Prepare test team Conduct pilot test

Team Work Usability specialists Designers and developers Technical communicators Trainers Marketing Helpdesk or customer assistance

Usability Testing Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Defining Goals and Concerns You cannot do exhaustive testing –Make choices among goals and concerns –Move from general concerns to specific ones –Understand sources of goals and concerns

Making Choices Among Goals and Concerns Clearly define the concern that the study will answer Prioritize all the concerns

Moving From General Concerns to Specific Ones Refinement Example –General: how easy is it to use the system? –Specific: how long does this specific task take?

Understanding Source of Goals and Concerns Sources: –Task analysis and quantitative usability goals –Timely issues –A heuristic analysis –Previous tests of this or other products

Usability Testing Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Choosing Participants Steps for choosing participants: –Developing user profiles –Selecting subgroups for a test –Defining and quantifying characteristics for each subgroup –Deciding how many people to include in a test

Developing User Profiles Collaborative work with: –Marketing, usability specialists, product designers Use –Surveys, focus groups sessions, contextual interviews, current users of previous version Characteristics: –Those that all users will share –Those that might make a difference among the users

Profile factors Work experience General computer experience Specific computer experience Experience with this product Experience with similar products

Selecting Subgroups for a Test Dividing users into subgroups Adding more characteristics Selecting the most critical characteristics Gathering other information

Defining and Quantifying Characteristics of Each Subgroup “Intermediate” users Setting a minimum and a maximum for a subgroup Selecting a range of participants within each subgroup

Deciding How Many People to Include in a Test Less than half of usability problems found with 3 participants (Nielson and Molich, 1990) 80% of usability problems detected with 4-5 participants (Virzi, 1992) 90% of usability problems detected with 10 participants (Virzi, 1992)

Deciding How Many People to Include in a Test (Cont.) The number of participants to include in a test depends on: –Number of subgroups needed –Budget and time –Importance of computing statistical significance of results

Usability Testing Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Design Techniques –Video, interaction logging, questionnaire, interviews Tasks –Time to complete –Number/Type of errors per task/unit time –Number of help requests –Number of users per error/success Testing conditions

Design (Cont.) Variables and Conditions Allocation of participants –Different participants –Same participants –Matched pairs of participants Data collection and analysis

Usability Testing Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Measure user performance without testing the users GOMS (Goal, Operators, Methods, Selection rules) The Keystroke Level Model

Balance Time, Money, and Information Gain