1. High-fidelity or Low-fidelity, Paper or Computer? Choosing attributes when testing web prototypes Miriam Walker Leila Takayama Professor James Landay University of California Berkeley G r o u p f o r User Interface Research

2. 2002 October 3HFES 46th Annual Meeting2 Outline  Motivation  Experimental testing of prototypes  Results and conclusions “You can fix it now on the drafting board with an eraser, or you can fix it later with a sledgehammer” Frank Lloyd Wright

3. 2002 October 3HFES 46th Annual Meeting3 Practical prototyping Prototypes as tools for design – “Track changes” is easier on computer than paper – Designers invest less time and ego in low- fidelity prototypes Prototypes as tools for usability testing – Computer prototypes allow remote testing – Computer prototypes have more realistic interactions

4. 2002 October 3HFES 46th Annual Meeting4 Website prototyping tools Prototype functionality depends on tools Prototyping tools can be – Low-fidelity or high-fidelity –Paper medium or computer medium Research and practical considerations should drive the selection of prototyping tools

5. 2002 October 3HFES 46th Annual Meeting5 Dreamweaver: Familiar high-fidelity, computer prototyping tool

6. 2002 October 3HFES 46th Annual Meeting6 Paper: Familiar low-fidelity, paper prototyping tool Post-its Scissors Pens Tape Transparencies Rulers Cardboard Foam-core (e.g. Rettig, 1994)

7. 2002 October 3HFES 46th Annual Meeting7 DENIM: A low-fidelity, computer prototyping tool (Lin, Newman, Hong, & Landay, 2000)

8. 2002 October 3HFES 46th Annual Meeting8 Why would fidelity and medium affect user testing? Fidelity and medium change interaction – Example: text-entry is handwritten or typed – Colors in high-fidelity direct attention Fidelity and medium may alter the users’ views on: – Functionality of prototype – Causes of and solutions for usability problems – Ability of users to have an impact on design (Hong et al, 2001)

9. 2002 October 3HFES 46th Annual Meeting9 Outline  Motivation  Experimental testing of prototypes  Results and conclusions

10. 2002 October 3HFES 46th Annual Meeting10 Making early stage prototypes Paper Medium Computer Medium Low- fidelity High- fidelity Sketched with paper and pens Scanned in paper pages Printed screens Coded in HTML

11. 2002 October 3HFES 46th Annual Meeting11 Low-Fidelity Prototype

12. 2002 October 3HFES 46th Annual Meeting12 High-Fidelity Prototype

13. 2002 October 3HFES 46th Annual Meeting13 Experimental design Participants were unaware of the experimental hypotheses Each participant saw either low-fidelity or high-fidelity websites on both paper and computer Paper then ComputerComputer then Paper Low-fidelity8 users High-fidelity6 users

14. 2002 October 3HFES 46th Annual Meeting14 Testing Methods Faked prototype functionality by constraining tasks – Sign-up for online banking services – Email a checking account statement – Calculate value of foreign currency – … Asked participants to think aloud Recorded their comments, and took copious notes Gave participants minimal assistance Followed up user tests with more questions

15. 2002 October 3HFES 46th Annual Meeting15 Outline  Motivation  Experimental testing of prototypes  Results and conclusions

16. 2002 October 3HFES 46th Annual Meeting16 Analysis method: effective usability testing More problems - six comments on one issue vs. one comment on each of six issues Most severe problems All types of problems e.g. consistency, feedback Level of detail - information architecture problems, widget problem

17. 2002 October 3HFES 46th Annual Meeting17 Quantitative analysis process comment issue Ratings: Issue severity Issue heuristic category (Nielsen, 1994) Comment scope (widget, page, website) Counts: Issue Comments Quantitative statistical analysis

18. 2002 October 3HFES 46th Annual Meeting18 Issues Comments: “ I would like recurrent payments … no scheduled. I don ’ t see the point, I don ’ t see the difference between these two but um.. ” “ Payment. Oh, actually, it would be recurring. I ’ m trying to decide if it ’ s a scheduled payment or recurring payment. ” “ Oops. Recurring. Then the single payment would be like a scheduled? How would … I ’ m just trying to figure out what the difference would be between the two. ” Issue: confusion between scheduled, single, and recurring on bill payer

19. 2002 October 3HFES 46th Annual Meeting19 Quantitative Analysis of Results 1270 comments and 169 issues Low-fidelity vs High-fidelity – No significant differences in number of comments or issues Paper vs. computer – Average of 5 more comments about computer prototype (Wilcoxon signed ranks test, p = 0.015) – Issues – no significant difference

20. 2002 October 3HFES 46th Annual Meeting20 Results No differences in severity of issues found No differences in scope of issues Differences between fidelities but not media categorizing issues by Nielsen’s heuristics (Chi Squared, p<0.01) Only 10% of comments mentioned aesthetics Classifying issues using Nielsen’s Heuristics is difficult

21. 2002 October 3HFES 46th Annual Meeting21 Conclusion Fidelity and medium do not seem to affect quantity of problems found by user testing Prototyping techniques should be chosen by considering: Need for remote testing Importance of recording design process Keeping designs at a level of detail appropriate to the stage of design

22. 2002 October 3HFES 46th Annual Meeting22 Special thanks to: Corey Chandler Jason Hong, James Lin, and Francis Li Sarah Waterson Professor Rashmi Sinha Ten anonymous expert raters Miriam Walker - mwalker@cs.berkeley.edu Leila Takayama - leila@cs.berkeley.edu http://guir.berkeley.edu/projects/fidelity G r o u p f o r User Interface Research University of California Berkeley

23. 2002 October 3HFES 46th Annual Meeting23 Question Time!