1. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 1 C yLab U sable P rivacy and S ecurity Laboratory http://cups.cs.cmu.edu/ Surveys, interviews, and focus groups Lorrie Faith Cranor September 2009

2. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 2 Surveys, interviews, and focus groups  Surveys – Ask people set list of questions (possibly with conditional questions or branching) with multiple choice or free response answers – Can be conducted in person, by postal mail, by phone, online  Interviews – Ask people open-ended questions, with follow-up questions depending on their responses – Usually conducted in person or by telephone  Focus groups – Group interview, usually with 5-10 participants – Usually conducted in person

3. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 3 What method should I use?  Survey – Allows you to collect data from a lot of people relatively quickly – Easy to collect data in standardized format, ready for analysis – But, not very good for answering “why” questions unless you have already developed hypothesis through other studies – Useful for validating hypotheses from smaller studies with a larger population  Interview – Allows you to probe mental models, what people think and why – Semi-structured interviews allow you to explore issues as they are raised – Allows you to clarify if people don’t understand a question  Focus group – Many of the advantages of interviews, but more efficient as you can interview 5-10 people all at once – Less detail from any interviewee than you would get in an individual interview – Not that great for testing usability because participants probably won’t all get hands on experience with software – Sometimes an opinionated individual can dominate a focus group – Hard to publish papers unless focus groups are used along with other methods or you report on a large number of focus groups together

4. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 4 SURVEYS

5. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 5 Designing good survey questions  Word questions clearly, without jargon or undefined abbreviations  Avoid leading questions, ambiguous terms, or emotionally-loaded terms  Design questions to evoke truthful responses – Non-threatening, don’t bias participants to provide what they think you want, protect confidentiality  Probe one dimension at a time  Design questions such that respondents are likely to provide a range of answers

6. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 6 Multiple-choice answers  Make sure answer choices are clear, mutually exclusive, cover entire space of possible answers  Make sure answer choices are at appropriate level of granularity  Where appropriate, allow respondents to indicate they don’t know, don’t have an opinion, or the question is not applicable to them  Use consistent rating scales throughout your survey  Be aware that the rating scale can skew responses (people like to think they are normal)

7. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 7 More survey design tips  Cluster similar questions together  Do not ask respondents to perform cognitively difficult tasks (unless you are testing their ability to perform these tasks) such as ranking more than 5 items  Use a clear and attractive layout  Pilot, pilot, pilot!

8. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 8 INTERVIEWS

9. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 9 Developing an interview script  Keep your questions fairly open-ended – You can follow-up with specific probes (“What files do you have on your computer that you consider valuable?” Follow up with “Do you have valuable photos? Videos? School work? Letters?”)  Start with general questions and get more specific so you get their unbiased impressions before you direct their thinking to particular details  A semi-structured approach allows you to adjust as needed  Write-out all your questions and follow-up probes

10. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 10 Preparing for an interview  The day before the interview: – Print out: Protocol – including detailed interview script Official consent form Payment sheet – Prepare: Compensation payment Audio/video recording (devices, batteries, extension cords, etc.)‏ Additional material – Send an email to the interviewee to remind him/her of the date/time/place

11. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 11 During the interview  Explain the purpose of the study (unless you need to hide that)‏  Ask to read and sign the consent form  If recording/video taping, turn it on!  Perform the study!  Debrief the person (if applicable)‏  Ask to sign the signing sheet  Give the payment  Thank the person!

12. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 12 More interview recommendations  Schedule meetings at least one week in advance  Be mindful of your subject/interviewee time  Keep a careful record of your interviews and keep it secure!

13. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 13

14. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 14

15. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 15 Analyzing interview data  Interview can be formally analyzed through a coding process – Transcribe interview data (yourself or with hired transcription service) – Iteratively review transcripts and create code for concepts mentioned by participants – As new codes are added, check to see whether those concepts were mentioned in previously analyzed transcripts – Keep track of how many participants mentioned each concept to find concepts that resonate with a lot of participants – Group similar codes together into categories – Note interesting quotes  Qualitative approach – Use concepts and categories to develop theory (Grounded theory approach, does not start with hypotheses)  Quantitative approach – With large number of interviewees (~30), and questions that ask participants to provide numerical ratings, quantitative analysis may be feasible – Important to validate for coder reliability  Can be used to develop mental models

16. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 16 FOCUS GROUPS

17. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 17 Advice on conducting focus groups  Develop detailed script to guide you  Pre-screen participants to get a good mix of people who meet your criteria  Record everything, but don’t have cameras pointed in people’s faces so they feel like they are under surveillance  Helpful to have at least 2 people, a moderator and a note taker  Give people name tags with their first name only  Make the session fun, informal, relaxed feel – Provide drinks and snacks  Promote a free flowing conversation that engages all participants – Ask open ended questions – Show people multiple things and ask them to compare – Give demos or show videos to start-off discussion – Give people handouts and ask them to circle things they like/don’t like, or jot down first impressions before discussing with the group  Plan to do multiple focus groups to mitigate effects of dominant participant steering conversation

18. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 18 Using videos in focus groups  Videos can provide concrete scenarios for people to discuss  Videos from L. Little, E. Sillence and P. Briggs. Ubiquitous Systems and the Family: Thoughts about the Networked Home. SOUPS 2009 Videos Ubiquitous Systems and the Family: Thoughts about the Networked Home. SOUPS 2009

19. CyLab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ 19 Cylab Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/