CS 321: Human-Computer Interaction Design November 12, 2015 Questionnaires Usability Questionnaires Post-Study & Post-Task Questionnaires Triangulation

Usability Questionnaires Using standardized questionnaires for usability studies offers several advantages. Objectivity Usability practitioners are able to independently verify the measurement statements of others. Replicability Studies can easily be replicated, improving their reliability. Quantification Results can be reported in finer detail and more objectivity. Communication Standardized measures facilitate communication between practitioners. Economy Developing standardized measures takes work, but reusing them is inexpensive. CS 321 November 12, 2015 Questionnaires Page 272

Post-Study Usability Questionnaires The PSSUQ is a 16-item survey that measures users’ perceived satisfaction with a product or system. The Post-Study System Usability Questionnaire (Version 3) Strongly Agree Strongly Disagree 1 2 3 4 5 6 7 NA 1. Overall, I am satisfied with how easy it is to use this system.  2. It was simple to use this system. 3. I was able to complete the tasks and scenarios quickly using this system. 4. I felt comfortable using this system. 5. It was easy to learn to use this system. 6. I believe I could become productive quickly using this system. 7. The system gave error messages that clearly told me how to fix problems. 8. Whenever I made a mistake using the system, I could recover easily and quickly. 9. The information (such as on-line help, on-screen messages, and other documentation) provided with this system was clear. 10. It was easy to find the information I needed. 11. The information was effective in helping me complete the tasks and scenarios. 12. The organization of information on the system screens was clear. 13. The interface of this system was pleasant. 14. I liked using the interface of this system. 15. This system has all the functions and capabilities I expect it to have. 16. Overall, I am satisfied with this system. An overall satisfaction score is obtained by averaging the sub-scales of System Quality (items 1-6), Information Quality (items 7-12), and Interface Quality (items 13-16). The PSSUQ is susceptible to “acquiesce bias”, the fact that people are more likely to agree with a statement than to disagree with it. CS 321 November 12, 2015 Questionnaires Page 273

Interpreting Questionnaire Results Psychometric analysis of usability questionnaires is conducted to determine their reliability, validity, and sensitivity. For example, the PSSUQ-3 norms here show that most items have means that fall below the scale midpoint of 4, indicating that the scale midpoint should not be used exclusively as a reference from which to judge participants’ perceptions on usability. PSSUQ-3 Norms (Means and 99% Confidence Intervals) Lower Limit Mean Upper Limit 1. Overall, I am satisfied with how easy it is to use this system. 2.60 2.85 3.09 2. It was simple to use this system. 2.45 2.69 2.93 3. I was able to complete the tasks and scenarios quickly using this system. 2.86 3.16 3.45 4. I felt comfortable using this system. 2.40 2.66 2.91 5. It was easy to learn to use this system. 2.07 2.27 2.48 6. I believe I could become productive quickly using this system. 2.54 3.17 7. The system gave error messages that clearly told me how to fix problems. 3.36 3.70 4.05 8. Whenever I made a mistake using the system, I could recover easily and quickly. 3.21 3.49 9. The information (such as on-line help, on-screen messages, and other documentation) provided with this system was clear. 2.65 2.96 3.27 10. It was easy to find the information I needed. 2.79 3.38 11. The information was effective in helping me complete the tasks and scenarios. 2.46 2.74 3.01 12. The organization of information on the system screens was clear. 2.41 2.92 13. The interface of this system was pleasant. 2.06 2.28 2.49 14. I liked using the interface of this system. 2.18 2.42 15. This system has all the functions and capabilities I expect it to have. 2.51 3.07 16. Overall, I am satisfied with this system. 2.55 2.82 Also note the relatively poor ratings associated with Item 7, which reflect the difficulty of providing usable error messages in a software product, as well as the overall dissatisfaction that such errors cause in users. CS 321 November 12, 2015 Questionnaires Page 274

Post-Task Usability Questionnaires While post-study surveys provide information regarding the general satisfaction of users with an interface, brief mini-surveys of user reaction to specific tasks in specific scenarios are often more useful when attempting to diagnose more focused problems. The After-Scenario Questionnaire (Version 1) Strongly Agree Strongly Disagree 1 2 3 4 5 6 7 NA 1. Overall, I am satisfied with the ease of completing the tasks in this scenario.  2. Overall, I am satisfied with the amount of time it took to complete the tasks in this scenario. 3. Overall, I am satisfied with the support information (online help, messages, documentation) when completing the tasks. Example scenarios and tasks for office software systems: Mail Scenario #1 Open a note Send reply Delete note Mail Scenario #2 Open a note Forward w/reply Save response Delete original Address Scenario Create new listing Modify old listing Delete unmodified listing File Scenario Rename file Copy file Delete file Editor Scenario Locate document Edit document Open note Copy note’s text into document Save document Print document CS 321 November 12, 2015 Questionnaires Page 275

Any given research method has advantages and limitations. Triangulation Any given research method has advantages and limitations. Lab experiments are abstract and obtrusive, and may not be representative of the real world. Field studies cannot be controlled, so it’s hard to make strong, precise claims regarding comparative usability. Self-reporting (via questionnaires) is often biased by reactivity (e.g., the subjects try to be polite or to say what they think they should say, instead of the truth). One way to deal with this problem is via triangulation, using multiple methods to tackle the same research question. If they all support your claim, then you have much stronger evidence, without as many biases. CS 321 November 12, 2015 Questionnaires Page 276