1. Lecture #3.3 USABILITY TESTS *39TUR Summer 2011/2012

2. (2) Who is the “customer” of the usability tests? Who asks you to do the usability test? – –Designers – –Software engineers – –Management – –Government agencies

3. (3) How well do they specify what to do? Thorough assignment – –“Here’s a detailed overview of what situations to test. We want to test such and such users, in such and such conditions.” Partial assignment – –Specific user group: “We would like to know how well this works with {elderly | children | violists }.” … Vague assignment – –“Find out whether it works.” – –In such case it is the responsibility of the tester to refine the assignment.

4. Terminologická poznámka “Tester” – –Rozpor ve významu – –Ten, s kým se testuje (dobryweb.cz, h1.cz) – –Ten, kdo organizuje a provádí test (správná) Lepší termíny – –“Participant” – –“Účastník testu”

5. METHODS OF USABILITY TESTING

6. (6) Usability Testing Usability testing – –Main goal: Looking for design problems Acceptance testing – –Refers to user satisfaction. A subset of the usability testing. “Will users like it?” – –Make sure the system meets expected criteria from the user side “80% customers need 1—3 minutes to withdraw amount of $50.”

7. (7) Methods Non-user based (without users) – –Expert review – –Heuristic evaluations – –Cognitive walkthrough User-based (with users) – –User surveys – –Ethnographic observations – –Usability engineering

8. (8) Methods with Users Observing users in their own environment – –Their office, etc. – –Natural and therefore realistic conditions Environment plays very important role (ringing phones, traffic on the streets, …) The user is accustomed to these conditions – –Difficult to organize, time consuming – –Results can not be generalized

9. (9) Methods with Users Controlled experiments – –Controlled (laboratory) environment – –Controlled conditions Limited influence of external stimuli No unwanted distractions by other people No unwanted distractions by software running on background – –Repeatable The same procedures will give the same results – –Non-realistic conditions problem with ecological validity

10. (10) User Surveys Questionnaires with questions oriented towards expected usability problems Quantitative – –Get precise measurement of one particular aspect Qualitative – –Open-ended questions – –Check how much you agree with the following statements: – –“It is easy to enter the telephone number” – –[strongly agree] [agree] [neither] [disagree] [strongly diagree] – –“It is easy to setup the owner’s address” – –[strongly agree] [agree] [neither] [disagree] [strongly diagree] – –“The menu item ‘Games’ is easy to find” – –[strongly agree] [agree] [neither] [disagree] [strongly diagree]

11. (11) User Surveys – Quantitative Check how much you agree with the following statements: “It is easy to enter the telephone number” [strongly agree] [agree] [neither] [disagree] [strongly diagree] “It is easy to setup the owner’s address” [strongly agree] [agree] [neither] [disagree] [strongly diagree] “The menu item ‘Games’ is easy to find” [strongly agree] [agree] [neither] [disagree] [strongly diagree] Your age: ……… Your income: ……… + Easy to process +/– Limited amount of possible answers

12. (12) User Surveys – Qualitative What are the biggest problems you find while using the telephone XX-1234? ……………………………………………………… What functions do you use most frequently? ……………………………………………………… – More difficult to process +/– Intended to map topics that were not previously studied

13. (13) User Surveys Pros – –Inexpensive – –Can be done remotely “While waiting for the video to load, why don’t you fill out our little questionnaire?” – –Can provide data on the current trends Cons – –Relies only on the data provided by the users – –Recruiting the right people is a problem Self-selection bias Add some screening questions Provide incentives

14. (14) Ethnographic Observation Observing users in their environment, without any intervention – –Can be rural India, can be office of the US government. – –E.g.: “Find out how cell phones are used in rural areas of India.” – –E.g.: “Find out how / whether the elderly people spend their day. Could they use instant messaging of some sort?” (ICQ, Skype…) The observer stays with the users and watches them how they use (struggle using) some system.

15. (15) Ethnographic Observation Pros: – –Based on the reality Cons: – –Needs a thorough preparation – –Not an interactive method – –Possible ethical problems – –Expensive

16. (16) Usability Engineering Nielsen (1983) Assessing the usability of a design Observing the users using the system in a simulated environment – –Aspects of the real world are simulated – –Users work on predefined, but realistic tasks – –Users are observed (measured) while carrying out the tasks – –Identification of the problematic parts

17. (17) Usability Engineering Are the results of such a test comparable to the results obtained from the “real-world observation”? – –Selected users are being tested – –The tasks are defined by the experimenters – –Users are not in their natural “habitat” – –Different social settings Solutions – –Use the real users – –Make the environment as real as possible – –Consult the tasks with the users

18. (18) Usability Engineering Make the tasks realistic Make the tasks ecologically valid

19. LAYOUT OF A USABILITY LAB

20. (20) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator)

21. (21) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator)

22. (22) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator)

23. (23) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator) 6-12

24. (24) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator) Tasks Task 1: xx xxx xx x xxx xx xxx. Tasks Task 1: In few minutes explore the new IDE environment. Please think aloud during the exploration process. Task 2: Pleas create and run simple mobile application "Hello World", which will display "Hello World!" message on the display of the cell phone emulator. Task 3: Your colleague finished the application GoodBye. Your complete this application wit

25. (25) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator) Tasks Task 1: xx xxx xx x xxx xx xxx. Tasks Task 1: In few minutes explore the new IDE environment. Pleas think aloud during the exploration process. Task 2: Please create and run simple mobile application "Hello World", which will display "Hello World!" message on the display of the cell phone emulator. Task 3: Your colleague finished the application GoodBye. Your complete this application wit

26. (26) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator) I am trying to create new project Tasks Task 1: xx xxx xx x xxx xx xxx. Log Task 1: xx xxx xx x xxx xx xxx. Test Session Log Task 1: 12:01 start 12:07 end Task 2: 12:09 start 12:10 creates new project

27. (27) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator) I am trying to create new project Tasks Task 1: xx xxx xx x xxx xx xxx. Zápis Úloha 1: xx xxx xx x xxx xx xxx. I can't find appropriate menu item Test Session Log Task 1: 12:01 start 12:07 end Task 2: 12:09 start 12:10 creates new project 12:12 can not find appropriate menu item

28. (28) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator) I am trying to create new project Tasks Task 1: xx xxx xx x xxx xx xxx. Zápis Úloha 1: xx xxx xx x xxx xx xxx. I expected item "New Project" in menu "File". I can't find appropriate menu item Test Session Log Task 1: 12:01 start 12:07 end Task 2: 12:09 start 12:10 creates new project 12:12 can not find appropriate menu item 12:13 expects "New Project" in menu "File"

29. (29) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator) Tasks Task 1: xx xxx xx x xxx xx xxx. Zápis Úloha 1: xx xxx xx x xxx xx xxx. Test Session Log Task 1: 12:01 start 12:07 end Task 2: 12:09 start 12:10 creates new project 12:12 can not find appropriate menu item 12:13 expects "New Project" in menu "File" 12:27 end Task 3: 12:31 start

30. (30) Usability Lab – usability testing Observer room (UI designers, programmers, test organizers) Participants room (test participant, moderator) Zápis Úloha 1: xx xxx xx x xxx xx xxx. Report 1. Goal xx xxx xx x xxx xx xxx. Final report 1.Test goal 2.Summary 3.Results 3.1Issue 1 Problem description Solution proposal 3.2 Issue 2 4. Conclusion

31. (31) We do need a lab! Everything is at hand – –The software that is tested, the tools for recording – –The experts are present – –The stakeholders can witness the process A chance to observe and record non-obvious findings – –Survey: User fills out: “I can use the function X to do Y.” – –Usability engineering: User says: “I can use this function X to do Y.” Experimenter: “Show me.” User: “Sure, I just …” (struggles) “uh … actually, I can not!”