Interviews & Focus Groups KSE966/986 Seminar - Fall 2012/Spring 2013 May 10, 2013 Uichin Lee

Contents 1. Pros and cons of interviews 2. Applications of interview research 3. Who to interview 4. Interview strategies 5. Interviews vs focus groups 6. Types of questions 7. Conducting an interview 8. Electronically mediated interviews 9. Analyzing interview data

What is interview? Gathering direct feedback from interested individuals Survey: reaching out a large number of people, but often restrictive ◦ Open-ended questions receive short (or no) answers Alternative: direct discussions with concerned participations ◦ Interviews with individuals ◦ Focus groups involving multiple users at one time

1. Pros and cons of interviews Pros ◦ Go deep ◦ Flexible Cons ◦ Skill to manage (unbounded discussion? Setting up meetings, listening/taking notes) ◦ Time and resource intensive (both interviewing and analyzing content) ◦ Recall problems (similar to surveys)

2. Applications of interview research Initial exploration Requirements elicitation Evaluation and Subjective Reactions

2. Applications of interview research Initial exploration Starting from clean slate, try to understand users’ goals/needs, to probe new possibilities, etc. by asking broader questions Requirements elicitation Evaluation and Subjective Reactions

Breaking the Disposable Technology Paradigm: Opportunities for Sustainable Interaction Design for Mobile Phones (CHI 2008) Goals: ◦ How people understand the lifespan of their phones ◦ What factors, such as style, service contracts, and functionality, affect how they attribute value to their phones, and their awareness and actions regarding mobile phone sustainability Methods: ◦ a 34-question web survey (n=79) ◦ a series of in-depth, semi-structured phone interviews with participants selected based on survey responses (who suggested an interesting history or experience) (n=10)

How do people organize their desks? Implications for the design of office information systems (ACM TOIS 1983) Goals: ◦ In the early 1980s, desktop info sys is relatively new; understanding how people organize info was incomplete ◦ Goal is to understand how people organize info Methods: Interviewed 10 workers in their offices ◦ Workers are asked to describe the layout of the office (indicating where info is stored and why)  Only clarification questions are asked  Some are asked to find some documents.. ◦ Also asked about information practices:  How well organize (Likert scale)  Biggest problems?  Do you keep lists of things to do?  Do you keep a calendar of appointments?  How often are you unable to do… Results: files (structured) vs. piles (unorganized, but also acting as reminders)

2. Applications of interview research Initial exploration Requirements elicitation ◦ Try to understand needs/concerns/goals during the early stage of design Evaluation and subjective reactions ◦ Try to get summative evaluations of completed products (e.g., usability surveys) (say for planning design revision)

3. Who to interview Beyond users – Stakeholders ◦ Anyone who may be affected by the use of a system ◦ There could be multiple types of users (or related people)  E.g., hospital info systems; patients and their family members Interview representatives of different groups from different perspectives Key informants who are knowledgeable about how work is done ◦ Particularly useful and informative individuals ◦ Must be selected carefully to avoid bias When selecting, respect the participants (e.g., reaching out homeless people on the street or people with disabilities)

4. Interview strategies: Types of Interviews Fully Structured Semi-Structured Unstructured Less structure: greater difficulty in conducting and interpreting interview But More opportunity for insight

4. Interview strategies: Types of Interviews Fully structured – Orally-presented survey ◦ Stick with the script. ◦ Good for comparison across individuals Semi-structured – pre-specified questions serve as starting point for discussion ◦ Digression is okay Unstructured – initial question, possible list of topics, but no pre-defined script ◦ Less structure good for open ended exploration

4. Interview strategies: Focused & Contextual Interviews Go beyond asking questions Ask participant to ◦ Demonstrate use of technology (instead of explaining how to do) ◦ Rely on artifacts (papers, photos, etc.) and context ◦ Use external aids to elicit feedback or reaction (known as “probes” e.g., props or prototypes) Example: ◦ Asking a user to show the office and tell how to store things ◦ Asking a user to show how tech is used at home in detail ◦ Asking to use a functional prototype and complete some tasks

5. Interviews vs focus groups Interviews take time ◦ 1 hour or more/response ◦ Several hours for analyzing notes Focus groups ◦ More people in less time ◦ Up to 8-12 people at once.

5. Interviews vs focus groups Focus Groups: Pros & Cons Pros ◦ Broad range of viewpoints and insights ◦ Each group will likely have at least one person who will stimulate others to talk Cons ◦ Hard to manage group dynamics ◦ Generally can't be fully structured ◦ May need to ask fewer questions ◦ Selection can be challenging

6. Types of questions Closed-ended Questions ◦ “On a scale of 1-10, 10 being best, how did you like the web page?” ◦ Easy to analyze, but may not be informative Yes/no questions ◦ Remember, the goal is to get interviewees to give you useful insight ◦ Simple questions get simple answers ◦ “Did you like the home page?” You'll get a one-word answer Open-ended questions ◦ Invite elaboration, discussion: e.g., What did you think about the web page?” ◦ Ask users to complete a sentence: “My favorite web browser feature is...” ◦ Conceptual mapping: draw pictures or layouts to describe understanding of a situation or problem

6. Types of questions Simple questions – no jargon Avoid compound questions with multiple parts ◦ Not "“What were the strengths and weaknesses of the menu layout and the toolbar?” ◦ Ask two separate questions instead Avoid judgmental phrasing or tone ◦ Not “Don’t you think this is hard to use?” ◦ Possible bias, instead ask “What did you think?”

6. Types of questions Questions in un- or semi-structured interviews ◦ Often, questions are generated in response to participant comments ◦ Can be hard to do this well ◦ Start with structured interviews  Get a few under your belt before moving to unstructured

7. Conducting an interview Preparation Recording responses During the interview ◦ Rapport ◦ Introduction ◦ Getting down to business ◦ Promoting discussion ◦ Debriefing

7. Conducting an interview Preparation Pilot test – w/ colleagues and participants ◦ Good for logistics and for confidence Write an interview guide listing what to do and when ◦ Particularly good if multiple researchers are involved Logistical backups ◦ Batteries for laptop, audio recorder, extra paper, etc.

7. Conducting an interview Recording responses: notes Audio and video recordings are fine, but Paper notes are still important – Record insights, non-verbal responses, etc. – Try to record what you can, but – Don't do so at the expense of listening Summarize written notes as soon as possible after the interview before you forget...

7. Conducting an interview Recording responses: audio/videos Complete, but expensive (transcribing and interpreting responses) Video is tricky, but gets useful information Consider audio + still pictures Respect privacy and anonymity

7. Conducting an interview During the Interview You're the Host: Build Rapport First few minutes are crucial Be professional, friendly, respectful, nonjudgmental If so, interviewees are inclined to trust you and provide honest and useful feedback Listen carefully Providing refreshments is a good idea (e.g., water, snacks) Avoid loud, crunchy food though Outline Briefly introduce research goals (use script if possible to be consistent) Tell that you’re recording the session and how Complete paperwork (informed consent) Simple questions first, hard questions later

7. Conducting an interview During the Interview Be flexible If your interview is not fully structured But, try to keep things on track Explain why you are asking each question Define terms, avoid jargon Ask for clarification

7. Conducting an interview During the Interview Read between the lines… ◦ Is the interviewee telling you what they think you want to hear? (hoping to please researchers) ◦ If so, make a note of it ◦ Might want to downplay in interpretation

7. Conducting an interview During the Interview Challenges of focus groups ◦ Manage the room. Be prepared to deal with  Digressions  Arguments ◦ Give everyone a chance to talk  Address them directly  “Joan, what do you think about...?”

7. Conducting an interview During the Interview Promoting discussion ◦ What if they won't talk? ◦ Fully-structured – not much to do ◦ Otherwise  Rephrase questions  Dig deeper into specifics ◦ Use props and probes to stimulate feedback ◦ Focus groups – ask for dissenting or concurring feedback

7. Conducting an interview Closing it out: Debriefing Ask for any final comments ◦ Interviewees might make additional useful comments Provide more details about research goals Brief summary of findings Say “thanks”! After thanking the users, spend some time to reflect and summarize notes immediately Consider turning off recording devices

8. Electronically mediated interviews Phone, web chat, , conference calls ◦ Pros  Easy, inexpensive  Reach out more people with less effort ◦ Cons  Lack of face-to-face contact  Fewer non-verbal cues  Pacing can be harder

9. Analyzing interview data Do it as soon as possible Avoid “cherry-picking” Fully-structured, closed-ended: tabulate answers Open-ended questions require coding (audio transcribe data, written notes) ◦ Organizing responses to various categories  e.g., affinity diagrams: hierarchical groupings of structures and themes, built from the bottom up Critical incidence analysis: if interviews are used for examining stories (e.g., a notable failure of an excising system)

Summary 1. Pros and cons of interviews 2. Applications of interview research 3. Who to interview 4. Interview strategies 5. Interviews vs focus groups 6. Types of questions 7. Conducting an interview 8. Electronically mediated interviews 9. Analyzing interview data