CyLab Usable Privacy and Security Laboratory 1 C yLab U sable P rivacy and S ecurity Laboratory Surveys,

Slides:



Advertisements
Similar presentations
Exploratory Research and Qualitative Analysis
Advertisements

4.12 U NDERSTAND DATA - COLLECTION METHODS TO EVALUATE THEIR APPROPRIATENESS FOR THE RESEARCH PROBLEM / ISSUE Understand promotion and intermediate.
What is a Survey? A scientific social research method that involves
Chapter 7 Data Gathering 1.
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
Conducting Interviews
+ Questions & Interviews What you should know. + Types of Questions 6 Basic 2.
Chapter 11: Collecting Data by Communication. Key Issues for Collecting Information by Communication.
An Introduction to Focus Groups Peter Harper. Focus Groups are Group Interviews Focus group interviews are characterised by: having a clear focus involving.
CyLab Usable Privacy and Security Laboratory 1 C yLab U sable P rivacy and S ecurity Laboratory Designing.
CyLab Usable Privacy and Security Laboratory 1 C yLab U sable P rivacy and S ecurity Laboratory Laboratory.
Data gathering.
Research Methods Lab In-Depth Interviews. Why Interviews? A major advantage of the interview is its adaptability A skillful interviewer can follow up.
Usable Privacy and Security Carnegie Mellon University Spring 2008 Lorrie Cranor 1 Designing user studies February.
Focus Groups for the Health Workforce Retention Study.
Developing a Questionnaire. Goals Discuss asking the right questions in the right way as part of an epidemiologic study. Review the steps for creating.
Chapter 7 GATHERING DATA.
FOCUS GROUPS & INTERVIEWS
Sabrina M Zooey M Data Collection. I. The use of data To know what certain people are thinking or doing by asking them or observing them.
How To Write A questionnaire
Interviews.
STRUCTURED INTERVIEWS AND INSTRUMENT DESIGN PART I Lecture 7.
Chapter 9 Descriptive Research. Overview of Descriptive Research Focused towards the present –Gathering information and describing the current situation.
SURVEYS AND QUESTIONNAIRES See Robson Chapter 8. Typical Survey Features  Use of a fixed quantitative design  Collection of a small amount of data in.
Development of Questionnaire By Dr Naveed Sultana.
4.12 & 4.13 UNDERSTAND DATA-COLLECTION METHODS TO EVALUATE THEIR APPROPRIATENESS FOR THE RESEARCH PROBLEM/ISSUE Understand promotion and intermediate.
Choosing and Implementing a Research Design Lauren Garcia-DuPlain The University of Akron English Composition 112.
Business and Management Research
Qualitative Research Methods
PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Chapter 8 Survey Research.
Human Computer Interface
The Marketing Survey How to construct a survey
CyLab Usable Privacy and Security Laboratory 1 C yLab U sable P rivacy and S ecurity Laboratory Surveys,
Questionnaires and Interviews
By: Christopher Prewitt & Deirdre Huston.  When doing any project it is important to know as much information about the project and the views of everyone.
A survey is a set of questions created to find out more information about an issue, usually using a wide range of people. Surveys are examples of quantitative.
Making Sense of the Social World 4th Edition
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
Creating Questionnaires. Learning outcomes Upon completion, students will be able to: Identify the difference between quantitative and qualitative data.
Gathering User Data IS 588 Dr. Dania Bilal Spring 2008.
Evaluating a Research Report
Data Collection Methods
Educational Action Research Todd Twyman Summer 2011 Week 2.
©2001 Southern Illinois University, Edwardsville All rights reserved. Today Interview Techniques (Hand-in partner preferences) Thursday In-class Interviewing.
Quality Assessment July 31, 2006 Informing Practice.
1 ISE 412 Usability Testing Purpose of usability testing:  evaluate users’ experience with the interface  identify specific problems in the interface.
COMP5047 Pervasive Computing: 2012 Think-aloud usability experiments or concurrent verbal accounts Judy Kay CHAI: Computer human adapted interaction research.
Slide 1 Requirements Determination Chapter 5. Slide 2 Objectives ■ Understand how to create a requirements definition. ■ Become familiar with requirements.
Focus Groups. What Are Focus Groups? “A group of individuals selected and assembled by researchers to discuss and comment on, from personal experience,
1 Learning Objectives: 1.Understand data collection principles and practices. 2.Describe the differences between collecting qualitative and quantitative.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Interviewing for Dissertation Research But these ideas apply to many types of interviewing.
Survey Methodology Survey Instruments (1) EPID 626 Lecture 7.
Marketing Research Approaches. Research Approaches Observational Research Ethnographic Research Survey Research Experimental Research.
AVI/Psych 358/IE 340: Human Factors Data Gathering October 6, 2008.
Do now! Complete the keywords test on the sheet. Remember that you are not writing 4 mark answers – just concise definitions (like in the book). You need.
AVI/Psych 358/IE 340: Human Factors Data Gathering October 3, 2008.
Sridhar Rajappan 1.  Key Terms  Review of unit 6 project  Buddy Work: suggestions for questions.
Asking Questions Dr. Guerette. Appropriate Topics Counting Crime Counting Crime Asking respondents about their victimization or offenders about their.
Fashion MARKETING TID1131. Types of research Quantitative research Information relating to numbers – quantity. Method - surveys Qualitative research To.
May 24, / Questionnaires... a. are a quick and easy way to gather information b. are only good for collecting qualitative data c. are.
Qualitative Research Methods Interviews Alexandra Bousiou (School of Public Administration)
Week 2: Interviews. Definition and Types  What is an interview? Conversation with a purpose  Types of interviews 1. Unstructured 2. Structured 3. Focus.
Some general tips for the interviews. Interviews Data collection method used to discover opinions held by users Works better for qualitative research.
Selecting a method of data collection. Qualitative and Quantitative Research Qualitative research explores attitudes, behavior and experience through.
School of Engineering and Information and Communication Technology KIT305/607 Mobile Application Development Week 7: Usability (think-alouds) Dr. Rainer.
Lecture3 Data Gathering 1.
Developing a Methodology
Oral History Resources
Business and Management Research
Presentation transcript:

CyLab Usable Privacy and Security Laboratory 1 C yLab U sable P rivacy and S ecurity Laboratory Surveys, interviews, and focus groups Lorrie Faith Cranor September 2011

CyLab Usable Privacy and Security Laboratory 2 Surveys, interviews, and focus groups  Surveys – Ask people set list of questions (possibly with conditional questions or branching) with multiple choice or free response answers – Can be conducted in person, by postal mail, by phone, online  Interviews – Ask people open-ended questions, with follow-up questions depending on their responses – Usually conducted in person or by telephone  Focus groups – Group interview, usually with 5-10 participants – Usually conducted in person

CyLab Usable Privacy and Security Laboratory 3 What method should I use?  Survey – Allows you to collect data from a lot of people relatively quickly – Easy to collect data in standardized format, ready for analysis – May be difficult to answer “why” questions – Useful for validating hypotheses from smaller studies with a larger population  Interview – Allows you to probe mental models, what people think and why – Semi-structured interviews allow you to explore issues as they are raised – Allows you to clarify if people don’t understand a question  Focus group – Similar to interviews, but more efficient as you can interview 5-10 people at once – Less detail from any interviewee than you would get in an individual interview – Not great for testing usability because participants probably won’t all use the software – Sometimes an opinionated individual can dominate a focus group – Hard to publish paper based only on one or two focus groups

CyLab Usable Privacy and Security Laboratory 4 SURVEYS

CyLab Usable Privacy and Security Laboratory 5 Designing good survey questions  Word questions clearly, without jargon or undefined abbreviations  Avoid leading questions, ambiguous terms, or emotionally-loaded terms  Design questions to evoke truthful responses – Non-threatening, don’t bias participants to provide what they think you want, protect confidentiality  Probe one dimension at a time  Design questions such that respondents are likely to provide a range of answers

CyLab Usable Privacy and Security Laboratory 6 Multiple-choice answers  Make sure answer choices are clear, mutually exclusive, cover entire space of possible answers  Make sure answer choices are at appropriate level of granularity  Where appropriate, allow respondents to indicate they don’t know, don’t have an opinion, or the question is not applicable to them  Use consistent rating scales throughout your survey  Be aware that the rating scale can skew responses (people like to think they are normal)

CyLab Usable Privacy and Security Laboratory 7 More survey design tips  Cluster similar questions together  Do not ask respondents to perform cognitively difficult tasks (unless you are testing their ability to perform these tasks) such as ranking more than 5 items  Use a clear and attractive layout  Pilot, pilot, pilot!

CyLab Usable Privacy and Security Laboratory 8 INTERVIEWS

CyLab Usable Privacy and Security Laboratory 9 Developing an interview script  Keep your questions fairly open-ended – You can follow-up with specific probes (“What files do you have on your computer that you consider valuable?” Follow up with “Do you have valuable photos? Videos? School work? Letters?”)  Start with general questions and get more specific so you get their unbiased impressions before you direct their thinking to particular details  A semi-structured approach allows you to adjust as needed  Write-out all your questions and follow-up probes

CyLab Usable Privacy and Security Laboratory 10 Role play and hypothetical scenarios  Appropriate for some interview studies  Give participant a role to play or put them in a hypothetical scenario – Imagine you just saw this message on your computer screen…. – Imagine your friend called you and told you he saw this message and asked you what to do….

CyLab Usable Privacy and Security Laboratory 11 Preparing for an interview  The day before the interview: – Print out: Protocol – including detailed interview script Official consent form Payment sheet – Prepare: Compensation payment Audio/video recording (devices, batteries, extension cords, etc.)‏ Additional material – Send an to the interviewee to remind him/her of the date/time/place

CyLab Usable Privacy and Security Laboratory 12 During the interview  Explain the purpose of the study (unless you need to hide that)‏  Ask to read and sign the consent form  If recording/video taping, turn it on!  Perform the study!  Debrief the person (if applicable)‏  Ask to sign the signing sheet  Give the payment  Thank the person!

CyLab Usable Privacy and Security Laboratory 13 Coding interview data  Transcribe interview data (yourself or with hired transcription service) – Depending on purpose of study, transcribing only selected quotes may be sufficient  Iteratively review transcripts and create code for concepts mentioned by participants  As new codes are added, check to see whether those concepts were mentioned in previously analyzed transcripts  Keep track of how many participants mentioned each concept to find concepts that resonate with a lot of participants  Group similar codes together into categories  Note interesting quotes

CyLab Usable Privacy and Security Laboratory 14 Analyzing interview data  Interview can be formally analyzed through a coding process  Qualitative approach – Use concepts and categories to develop theory (Grounded theory approach, does not start with hypotheses)  Quantitative approach – With large number of interviewees (~30), and questions that ask participants to provide numerical ratings, quantitative analysis may be feasible – Important to validate for coder reliability  Can be used to develop mental models

CyLab Usable Privacy and Security Laboratory 15 FOCUS GROUPS

CyLab Usable Privacy and Security Laboratory 16 Planning a focus group  Develop very detailed script to guide you  Pre-screen participants to get a good mix of people who meet your criteria  Setup audio and video recorders, but don’t make people feel under surveillance  Helpful to have at least 2 people, a moderator and a note taker  Give people name tags with their first name only  Plan to do multiple focus groups to mitigate effects of dominant participant steering conversation

CyLab Usable Privacy and Security Laboratory 17 Conducting a focus group  Make the session fun, informal, relaxed feel – Provide drinks and snacks  Promote a free flowing conversation that engages all participants – Ask open ended questions – Show people multiple things and ask them to compare – Give demos or show videos to start-off discussion – Give people handouts and ask them to circle things they like/don’t like, or jot down first impressions before discussing with the group

CyLab Usable Privacy and Security Laboratory 18 Using videos in focus groups  Videos can provide concrete scenarios for people to discuss

CyLab Usable Privacy and Security Laboratory 19 Cylab Usable Privacy and Security Laboratory