1. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 1 C yLab U sable P rivacy and S ecurity Laboratory http://cups.cs.cmu.edu/ Surveys, interviews, and focus groups Lorrie Faith Cranor September 2011

2. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 2 Surveys, interviews, and focus groups  Surveys – Ask people set list of questions (possibly with conditional questions or branching) with multiple choice or free response answers – Can be conducted in person, by postal mail, by phone, online  Interviews – Ask people open-ended questions, with follow-up questions depending on their responses – Usually conducted in person or by telephone  Focus groups – Group interview, usually with 5-10 participants – Usually conducted in person

3. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 3 What method should I use?  Survey – Allows you to collect data from a lot of people relatively quickly – Easy to collect data in standardized format, ready for analysis – May be difficult to answer “why” questions – Useful for validating hypotheses from smaller studies with a larger population  Interview – Allows you to probe mental models, what people think and why – Semi-structured interviews allow you to explore issues as they are raised – Allows you to clarify if people don’t understand a question  Focus group – Similar to interviews, but more efficient as you can interview 5-10 people at once – Less detail from any interviewee than you would get in an individual interview – Not great for testing usability because participants probably won’t all use the software – Sometimes an opinionated individual can dominate a focus group – Hard to publish paper based only on one or two focus groups

4. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 4 SURVEYS

5. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 5 Designing good survey questions  Word questions clearly, without jargon or undefined abbreviations  Avoid leading questions, ambiguous terms, or emotionally-loaded terms  Design questions to evoke truthful responses – Non-threatening, don’t bias participants to provide what they think you want, protect confidentiality  Probe one dimension at a time  Design questions such that respondents are likely to provide a range of answers

6. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 6 Multiple-choice answers  Make sure answer choices are clear, mutually exclusive, cover entire space of possible answers  Make sure answer choices are at appropriate level of granularity  Where appropriate, allow respondents to indicate they don’t know, don’t have an opinion, or the question is not applicable to them  Use consistent rating scales throughout your survey  Be aware that the rating scale can skew responses (people like to think they are normal)

7. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 7 More survey design tips  Cluster similar questions together  Do not ask respondents to perform cognitively difficult tasks (unless you are testing their ability to perform these tasks) such as ranking more than 5 items  Use a clear and attractive layout  Pilot, pilot, pilot!

8. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 8 INTERVIEWS

9. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 9 Developing an interview script  Keep your questions fairly open-ended – You can follow-up with specific probes (“What files do you have on your computer that you consider valuable?” Follow up with “Do you have valuable photos? Videos? School work? Letters?”)  Start with general questions and get more specific so you get their unbiased impressions before you direct their thinking to particular details  A semi-structured approach allows you to adjust as needed  Write-out all your questions and follow-up probes

10. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 10 Role play and hypothetical scenarios  Appropriate for some interview studies  Give participant a role to play or put them in a hypothetical scenario – Imagine you just saw this message on your computer screen…. – Imagine your friend called you and told you he saw this message and asked you what to do….

11. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 11 Preparing for an interview  The day before the interview: – Print out: Protocol – including detailed interview script Official consent form Payment sheet – Prepare: Compensation payment Audio/video recording (devices, batteries, extension cords, etc.)‏ Additional material – Send an email to the interviewee to remind him/her of the date/time/place

12. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 12 During the interview  Explain the purpose of the study (unless you need to hide that)‏  Ask to read and sign the consent form  If recording/video taping, turn it on!  Perform the study!  Debrief the person (if applicable)‏  Ask to sign the signing sheet  Give the payment  Thank the person!

13. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 13 Coding interview data  Transcribe interview data (yourself or with hired transcription service) – Depending on purpose of study, transcribing only selected quotes may be sufficient  Iteratively review transcripts and create code for concepts mentioned by participants  As new codes are added, check to see whether those concepts were mentioned in previously analyzed transcripts  Keep track of how many participants mentioned each concept to find concepts that resonate with a lot of participants  Group similar codes together into categories  Note interesting quotes

14. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 14 Analyzing interview data  Interview can be formally analyzed through a coding process  Qualitative approach – Use concepts and categories to develop theory (Grounded theory approach, does not start with hypotheses)  Quantitative approach – With large number of interviewees (~30), and questions that ask participants to provide numerical ratings, quantitative analysis may be feasible – Important to validate for coder reliability  Can be used to develop mental models

15. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 15 FOCUS GROUPS

16. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 16 Planning a focus group  Develop very detailed script to guide you  Pre-screen participants to get a good mix of people who meet your criteria  Setup audio and video recorders, but don’t make people feel under surveillance  Helpful to have at least 2 people, a moderator and a note taker  Give people name tags with their first name only  Plan to do multiple focus groups to mitigate effects of dominant participant steering conversation

17. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 17 Conducting a focus group  Make the session fun, informal, relaxed feel – Provide drinks and snacks  Promote a free flowing conversation that engages all participants – Ask open ended questions – Show people multiple things and ask them to compare – Give demos or show videos to start-off discussion – Give people handouts and ask them to circle things they like/don’t like, or jot down first impressions before discussing with the group

18. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 18 Using videos in focus groups  Videos can provide concrete scenarios for people to discuss

19. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 19 Cylab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/