1. Involving Users in Interface Evaluation Marti Hearst (UCB SIMS) SIMS 213, UI Design & Development April 8, 1999

2. Adapted from slide by James Landay Outline l Why do user testing? l Informal studies –collecting and analyzing process data –ethical considerations l Formal studies –chosing variables –interaction effects –special considerations for studies involving users

3. What is Usability? l The extent to which a product can be used by specified users to achieve specified goals with –effectiveness –efficiency –satisfaction in a specified context of use. [ISO9241] l Usability evaluation is a methodology for measuring these usability aspects of a user interface

4. Adapted from slide by James Landay Why do User Testing? l Can’t tell how good or bad UI is until: –people use it! l Other methods are based on evaluators: –may know too much –may not know enough (about tasks, etc.) l Summary: Hard to predict what real users will do

5. Adapted from slide by James Landay Two Main Approaches l Less formal, get a feeling for how users will use interface –participants work through task scenarios –gather process data –obtain user preference information l Formal studies –isolate the effects of particular UI components –compare competing designs –make quantitative measurements

6. Adapted from slide by James Landay Why Two Main Approaches? l Informal Study –Prcess data is easier to gather –Gives an overview of where big problems are l Formal Study –need many participants to prove your points (obtain statistical significance) –experiments that isolate effects properly often end up measuring things that are too fine- grained to really inform UI design

7. Adapted from slide by James Landay Informal Study l Select tasks l Select participant groups l Decide methodology for collecting data and what kinds of data to collect l Do the study l Analyze the results l Make recommendations for changes to the design

8. Adapted from slide by James Landay Selecting Tasks l Should reflect what real tasks will be like l Tasks from analysis & design can be used –may need to shorten if »they take too long »require background that test user won’t have l Avoid bending tasks in direction of what your design best supports l May have to simplity in order to produce usable results

9. Adapted from slide by James Landay Choosing Participants l Should be representative of eventual users in terms of –job-specific vocabulary / knowledge –tasks l If you can’t get real users, get approximation –system intended for doctors »get medical students –system intended for electrical engineers »get engineering students l Use incentives to get participants

10. Adapted from slide by James Landay Deciding on Data to Collect l Process data –observations of what users are doing & thinking –kinds of errors made –general strategies used and not used

11. Adapted from slide by James Landay The “Thinking Aloud” Method l Need to know what users are thinking, not just what they are doing l Ask users to talk while performing tasks –tell us what they are thinking –tell us what they are trying to do –tell us questions that arise as they work –tell us things they read l Make a recording or take good notes –make sure you can tell what they were doing

12. Adapted from slide by James Landay Thinking Aloud (cont.) l Prompt the user to keep talking –“tell me what you are thinking” l Only help on things you have pre- decided –keep track of anything you do give help on l Recording –use a digital watch/clock –take notes, plus if possible »record audio and video (or event logs)

13. Adapted from slide by James Landay Ethical Considerations l Sometimes tests can be distressing l You have a responsibility to alleviate this –make voluntary with informed consent form –avoid pressure to participate –let them know they can stop at any time [Gomoll] –stress that you are testing the system, not them –make collected data as anonymous as possible l Often must get official approval for use of human subjects –There is a campus exception for class projects

14. Adapted from slide by James Landay User Test Proposal l A report that contains –objective –description of system being testing –hypotheses –task environment & materials –participants –methodology –tasks –test measures l A good strategy: –Get this approved & then reuse it when writing up your results

15. Adapted from slide by James Landay Using the Test Results l Summarize the data –make a list of all critical incidents (CI) »positive & negative –include references back to original data –try to judge why each difficulty occurred l What does data tell you? –Did the UI work the way you thought it would? –Is something missing?

16. Adapted from slide by James Landay Using the Results (cont.) l Update task analysis and rethink design –rate severity & ease of fixing CIs –fix both severe problems & make the easy fixes l Will thinking aloud give the right answers? –not always –if you ask a question, people will always give an answer, even it is has nothing to do with the facts »try to avoid specific questions

17. Adapted from slide by James Landay Measuring User Preference l How much users like or dislike the system –often use Likert scales –or have them choose among statements »“best UI I’ve ever…”, “better than average”… –hard to be sure what data will mean »novelty of UI, feelings, not realistic setting, etc. –Shneiderman’s QUIS is a general example (in the reader) l If many give you low ratings -> trouble l Can get some useful data by asking –what they liked, disliked, where they had trouble, best part, worst part, etc. (redundant questions)

18. Adapted from slide by James Landay Formal Usability Studies l Situations in which these are useful –to determine time requirements for task completion –to compare two designs on measurable aspects »time required »number of errors »effectiveness for achieving very specific tasks l Do not combine with thinking-aloud –talking can affect speed & accuracy (neg. & pos.) l Require Experiment Design

19. Experiment Design l Experiment design involves determining how many experiments to run and which attributes to vary in each experiment l Goal: isolate which aspects of the interface really make a difference

20. Experiment Design l Decide on –Response variables »the outcome of the experiment »usually the system performance »aka dependent variable(s) –Factors (aka attributes)) »aka independent variables –Levels (aka values for attributes) –Replication »how often to repeat each combination of choices

21. Experiment Design l Studying a system; ignoring users l Say we want to determine how to configure the hardware for a personal workstation (from Jain 91, The art of computer systems performance analysis) –Hardware choices »which CPU (three types) »how much memory (four amounts) »how many disk drives (from 1 to 3) –Workload characteristics »administration, management, scientific

22. Experiment Design l We want to isolate the effect of each component for the given workload type. l How do we do this? –WL1CPU1 Mem1Disk1 –WL1CPU1Mem1Disk2 –WL1CPU1Mem1Disk3 –WL1CPU1Mem2Disk1 –WL1CPU1Mem2Disk2 –…–… l There are (3 CPUs)*(4 memory sizes)*(3 disk sizes)*(3 workload types) = 108 combinations!

23. Experiment Design l One strategy to reduce the number of comparisons needed: –pick just one attribute –vary it –hold the rest constant l Problems: –inefficient –might miss effects of interactions

24. Interactions among Attributes A1A2 B135 B268 A1A2 B135 B269 A1 B1 A2 A1 B2 A2 B2 Non-interactingInteracting

25. Experiment Design l Another strategy: figure out which attributes are important first l Do this by just comparing a few major attributes at a time –if an attribute has a strong effect, include it in future studies –otherwise assume it is safe to drop it l This strategy also allows you to find interactions between a few attributes

26. Special Considerations for Formal Studies with Human Participants l Studies involving human participants vs. measuring automated systems –people get tired –people get bored –people (may) get upset by some tasks –learning effects »people will learn how to do the tasks (or the answers to questions) if repeated »people will (usually) learn how to use the system over time

27. More Special Considerations l High variability among people –especially when involved in reading/comprehension tasks –especially when following hyperlinks! (can go all over the place)

28. Adapted from slide by James Landay Between Groups vs. Within Groups l Do participants see only one design or both? l Between groups –two groups of test users –each group uses only 1 of the systems l Within groups experiment –one group of test users »each person uses both systems »can’t use the same tasks (learning) –best for low-level interaction techniques l Why is this a consideration?n –People often learn during the experiment.

29. Summary l User testing is important, but takes time/effort l Use real tasks & representative participants l Be ethical & treat your participants well l Want to know what people are doing & why –collect process data –early testing can be done on mock-ups (low-fi) l More on formal studies next time.