Questionnaires Judy Kay CHAI: Computer Human Adapted Interaction research group Human Centred Technology Cluster for Teaching and Research School of Information Technologies

2 Landscape of methods to understand user and work Interviews, Surveys, Focus Groups Ask people what they believe they want Many limitations: Human memory Politeness Self-awareness Many benefits: User involvement, commitment, buy-in Access to what users know, think, perspectives, contexts that matter Contextual Inquiry Observe what people actually do Many limitations: Can you be there at the right time? Observer effect Intrusiveness Many benefits: Insights into actual behaviour And contexts And stakeholder perspectives

Questionnaires Can be easy to administer – Closed – Open Likert Difficult to design them well Standard Questionnaires – SUS – SEQ

SUS (on tasks completion) SUS: A Quick and Dirty Usability ScaleSUS: A Quick and Dirty Usability Scale by John Brooke  3,000 Scholar citations  Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale by Kortum Bangor and Miller May * Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale

From usability.gov The scoring system is somewhat complex There is a temptation, when you look at the scores, since they are on a scale of 0-100, to interpret them as percentages, they are not The best way to interpret your results involves “normalizing” the scores to produce a percentile ranking SUS is not diagnostic - its use is in classifying the ease of use of the site, application or environment being tested

Please check the box that reflects your immediate response to each statement. Don’t think too long about each statement. Make sure you respond to every statement. If you don’t know how to respond, simply check box “3.” * 1.I think that I would like to use this system frequently. 2.I found the system unnecessarily complex. 3.I thought the system was easy to use. 4.I think that I would need the support of a technical person to be able to use this system. 5.I found the various functions in this system were well integrated. 6.I thought there was too much inconsistency in this system. 7.I would imagine that most people would learn to use this system very quickly. 8.I found the system very cumbersome (*awkward) to use. 9.I felt very confident using the system. 10.I needed to learn a lot of things before I could get going with this system. * System … product

Likert scales, semantic differential From SUS: Please check the box that reflects your immediate response to each statement. Don’t think too long about each statement. Make sure you respond to every statement. If you don’t know how to respond, simply check box “3.”

Semantic differential scales Granularity of the scale –Number of discrete points –Including anchors –The number of choices we allow users Copyright MKP. All rights reserved. 8

Semantic differential scales Typical labeling of a point on a scale is verbal –Often with associated numeric value –Labels can also be pictorial Example, smiley faces Helps make it language-independent Copyright MKP. All rights reserved. 9

Example, semantic differential scale To assess participant agreement with this statement –“The checkout process on this Website was easy to use.” Might have these anchors: Strongly agree and strongly disagree Copyright MKP. All rights reserved. 10

Examples In between scale : –Agree, neutral, disagree Could have associated values –+2, +1, 0, -1, and -2 What do these mean for the person?

Getting the SUS score (/100) 1.Convert each score into a number/4 1.For items 1,3,5,7,and 9 the score contribution is the scale position minus 1 2.For items 2,4,6,8 and 10, the contribution is 5 minus the scale position. 2.Multiply the sum of the scores by 2.5. Empirically – overall 68 is “average” SUS: A Quick and Dirty Usability ScaleSUS: A Quick and Dirty Usability Scale by John Brooke

More subtlety in scoring We have used this version of the SUS in almost all of the surveys we have conducted, which to date is nearly 3,500 surveys within 273 studies. It has proven to be a robust tool, having been used many times to evaluate a wide range of interfaces that include Web sites, cell phones, IVR, GUI, hardware, and TV user interfaces. In all of these cases, participants performed a representative sample of tasks for the product (usually in formative usability tests) and then, before any discussion with the moderator, completed the survey. Determining What Individual SUS Scores Mean: Adding an Adjective Rating ScaleDetermining What Individual SUS Scores Mean: Adding an Adjective Rating Scale by Kortum Bangor and Miller May

Mean ~ 70 over long term. Median score of % of surveys < 50

Class activity So SUS for your experience to-date of CUSP

More detail in SUS pragmatics Score SUS properly, accounting for errors, omissions, inconsistencies tested by internal reliability test (Cronbach's alpha) Percentile Ranks and Grades A+.. F: Compare against 500 studies by application type Compares SUS Scores Statistically Computes Sample Sizes, based on desired margin of error

Design decisions The questions to ask – Challenges of biasing the user – Asking all questions in term of a positive view eg I found the LearnWell site easy to use – Or all in terms of a negative view eg I found the LearnWell site hard to use User choices: – Note we previously used a different approach Words to describe each case (as in the learning scale for the concept inventory: never heard of it … could teach a friend) – Just give end points Even or odd? What makes sense to ask users?

Why standard questionnaires Reliability : how consistent responses are to the questions. How would you assess that? Validity : correlates with other evidence eg completion rates. Sensitivity: discriminates between good interfaces and bad interfaces. The ability to detect differences at even small sample sizes (<10) was one of the major reasons why the System Usability Scale (SUS) and the Single Ease Question (SEQ) are recommended.System Usability Scale (SUS)Single Ease Question (SEQ) Objectivity: Give independent instrument. Quantification: Fine grain of reporting and statistical analysis. Economy: Cheap to design!

SEQ Single Ease Question Sauro, J., & Dumas, J. S. (2009, April). Comparison of three one-question, post-task usability questionnaires. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp ). ACM.

SEQ

Adapting questionnaires You can modify an existing questionnaire –Choosing a subset of questions –Changing the wording in some questions –Adding questions to address specific areas of concern –Using different scale values Warning: Modifying a questionnaire can damage its validity Copyright MKP. All rights reserved. 24

AttrackDiff A very different sort of more recent questionnaire

Hedonic quality: AttrakDiff Marc Hassenzahl

“The model separates the four essential aspects: – The product quality intended by the designer. – The subjective perception of quality and subjective evaluation of quality [by the user]. – The independent pragmatic and hedonic qualities. – Behavioural and emotional consequences. “

Summary Questionnaires – what is their role in the whole process? When is it appropriate to use a questionnaire? When is it not? Why standard questionnaires? What you need to know about altering them? How you should use them for Assignment 2.