1. Introductions
Name
Institution/Department
Specific project (if one)
Familiarity with qualitative research

2. Qualitative Research Methods: Data Collection
Bridget O’Brien, PhD
Associate Professor of Medicine
Researcher in the Center for Faculty Educators

3. Creative Commons License
Attribution-NonCommercial-Share Alike 3.0 Unported
You are free:
to Share — to copy, distribute and transmit the work
to Remix — to adapt the work
Under the following conditions:
Attribution. You must give the original authors credit (but not in any way that suggests that they endorse you or your use of the work).
Noncommercial. You may not use this work for commercial purposes.
Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.
See for full license. 3

4. Overview of Workshop Exercise 1 – Article discussion
Definition and characteristics of qualitative methods
Review common types of data collection methods
Exercise 2 – Instrument development
Exercise 3 – Conducting an interview
Aligning questions with methods
Discuss rigor in qualitative research
Exercise 4 – Convincing a colleague
Middle section is didactic heavy – lets aim to make it more interactive - please interrupt with questions
chose 4 common types because no one discussed wanting to know about picture or photo analysis

5. Exercise 1: Article Discussion
Why was this study done? Was the significance of the problem with the relevant theory stated?
What was the research approach used?
What were the researcher characteristics?
How was the sample selected?
Were all the data collection details included?
Who were the participants?
How are the key elements of a research article treated similarly / differently in writing up qualitative research?
What other methods could have been used to study this topic? Would those methods have given rise to the same results?

6. Qualitative Research
Research on lives, behavior, organizational functioning, interactions, belief systems
Exploratory
Open-ended
Data = Words
-Research about perceptions, beliefs, behavior, social interactions and relationships, organizational processes
-often exploratory or descriptive
-Data=descriptions, narrative accounts, observed behaviors
-Analytic procedures are interpretive and interact with data collection (moving back and forth between development of concepts or frameworks and inquiry through data collection)
-Data collection is complete when saturation is achieved – no new themes, no new areas of inquiry

7. Example: Quantitative Data
Rate the overall effectiveness of this rotation:
(Mean: 4.8, SD: 0.51)

8. Example: Qualitative Data
Great opportunity to develop meaningful relationships with patients, attendings, and residents within this longitudinal rotation over 6 months. Lots of opportunities to take care of my own patients. Patient population is almost 100% low-income, so I learned a lot about the needs and assets of the underserved communities in the Central Valley, which is very different from the Bay Area. Really enjoyed doing my project and felt that I received a lot of support and guidance from faculty and the dept. Most residents and attendings are friendly.

9. Qualitative Research Analytic procedures are interpretivist
Analysis is iterative (i.e. concurrent with and post data collection)

10. Qualitative Methods Purposes
To describe and understand
To explore complex phenomena through collection/analysis of qualitative data
To supplement quantitative data—to validate, explain, illuminate, interpret
To understand meaning in everyday activities
To discover framework or hypothesis
To condense extensive and varied raw data (comments, transcripts, notes) into a brief summary or key points

11. Qualitative Methods Uses
Research
Evaluation
Needs assessment for curriculum development
Instrument development

12. Qualitative Research Questions
What helps individual clinicians learn from and adapt after making a harmful mistake?
What are the qualities of an ideal mentor?
How are attending physicians prepared and trained to perform the tasks and duties after transitioning to their new positions?
What are the different ways clinical teachers understand what constitutes a good teacher and a good clinical supervisor and what similarities and differences do they report between them?

14. Qualitative Paradigms
Sets of beliefs that guide action
Qualitative research takes place within a vast spectrum of paradigms:
Positivism
Post-positivism
Interpretivism / Constructivism
Critical Theory
*Ginsburg article mentions a constructivist grounded theory approach.
*NOTE: Often authors are not explicit about this and may not even be mindful of the paradigm. But it is important to be mindful of ontology (what is reality), epistemology (what is the nature of knowledge), and methodology (research approach). For our purposes, the distinction btwn post-positivist and interpretive / constructive are most relevant. PP is hypothesis testing – assumes there is one truth / reality, but we can’t know it completely so we make hypotheses and measure as precisely as possible to approximate truth. Strive for objectivity. Constructivist assumes multiple realities and truths, knowledge is subjective, tries to capture multiple different perspectives and interpretations; inductive approach, purpose is understanding.
See: Bunniss & Kelly. Research paradigms in medical education. Medical Education, 2010, 44:

15. Qualitative Approaches
Acts as a pivotal frame that shapes the research question, the method of data collection, and how data are analyzed:
General Inductive Approach*
Ethnography*
Grounded theory*
Case study*
Phenomenology / Phenomenography*
Hermeneutics
Narrative research
Action research*
NOTES on general inductive approach: exploring themes coming out of data//Often used to analyze evaluation data i.e. responses to open ended questions on evaluation forms; notes or transcripts from focus groups
No pre-existing framework or hypothesis, though may be organized around a set of questions
Purpose is typically to condense extensive and varied raw data (comments, transcripts, notes) into a brief summary or key points
Involves detailed review of raw data for frequent, dominant, or significant themes
(REF: Thomas, 2006)
Ethnography – purpose is to understand the nature of the culture (or certain cultural elements such as notions, beliefs, representations) of a group based on the point of view of different members, on observations of their actions and how they function, and/or on analysis of relevant documents.
Phenomenology – purpose is to understand the meaning or significance of a phenomenon based on the experience of those who go through it (e.g. continuity with a patient, reflective writing, witnessing or contributing to a medical error)
Grounded Theory – purpose is to understand a process or interaction (e.g. clinical teaching, formation of professional identity, development of a new skill or competency) – no theory or hypothesis guides the work.

16. Sampling
“Participants are not recruited on a representative basis, but rather because of their expert knowledge of the phenomenon under inquiry” (Green & Thorogood, 2005)
Purposeful
Seeking the best sources of information about the phenomenon of interest
Theoretical
Seeking the best sources of information to confirm / disconfirm your developing explanatory model
Ginsburg article mentions: Sample size was estimated to be sufficient based on the principle of theoretical saturation and our previous experience with this methodology (i.e. relatively homogenous population). ALSO note: invited all students in the class and took first 15 volunteers (not sure this is really theoretical sampling? I guess because there is no theory to expect differences?)
Green J, Thorogood N. Qualitative methods for health research. London: Sage; 2005.

17. Considerations in Sample Selection
The choices of who/how many to study are directly related to the research question
Might select all members of a group
Might randomly stratify within a group based on a common variable
Usually small numbers (compared to quantitative approaches)

18. Examples of Sampling Maximum variation Snowball Extremes Typical case
Negative case

19. Qualitative Data Collection
Observation
Interview
Focus groups
Text / documents
Artifacts (pictures, photos)

20. Observation
Spend time with a group of people as they carry out their daily activities, to understand their way of life and how they make sense of the world in which they live.
To understand experiences of group members, their activities, interactions, discussions
To understand how contextual factors influence the activities and decisions of group members
Example: How do students participate in ambulatory clinic? What influences students’ level of participation? -

21. Observation Types of observation data Field Notes Sketches
Relation diagrams
Role of Researcher
Researcher is instrument
Continuum of roles that range from complete participation to complete observation
Type of role adopted depends on research question
To understand process of change
To generate hypotheses
Roles can change over time in research setting

22. Observation
Observer effect – impact of observer presence on participant behavior
Spend time for participants to acclimate
Conceal specific focus

23. Observation Pluses and Minuses
Overcome discrepancy between what people say and what they do
Uncover behaviors unknown to participants
Minuses:
Getting in
“Going native”
Selective recall/viewing
Time consuming

24. Interviews
The interactional exchange of dialogue in face-to-face or other contexts
A thematic or topic-centered approach where the researcher has topics, themes or issues they wish to cover, but with a fluid and flexible structure.
A perspective regarding knowledge as situated and contextual,
requiring the researcher to ensure that relevant contexts are brought into focus so that the situated knowledge can be produced.
Meanings and understandings are created in an interaction

25. Interviews Types of Interviews
Structured: verbal questionnaire, scripted questions
Semi-structured: outline of topics guides questions
Informal: conversational, questions arise in context
Retrospective: focuses on past events
Participants express their understandings and experiences in their own terms
Ginsburg – pretty scripted, response to video prompts.
Extent to which wording and sequencing ought to be predetermined depends on purpose of interview
To understand daily experiences and the meanings of those experiences

26. Interviews Pluses and Minuses
People like to talk
Interviewer can follow leads during the interview
Flexible
Can be implemented according to the interviewee schedule
Minuses:
Depends on interviewer skill
Interviewer requires training
Interviews need to be monitored for bias

27. Focus Groups
Group discussions organized to explore a specific set of themes, topics or issues
The group is focused in that it involves some kind of collective activity
Data include spoken words AND group interaction

28. Focus Groups Small groups with a moderator
Concentrated data on topic of interest in short time span
Group interview with insight into interactions on topic (compared to interviews)
Important that topic of interest would be easy for participants to discuss in a group

29. Focus Groups Pluses and Minuses
Always learn something
Does not discriminate against people who cannot read or write
Encourages reluctant participants
Investigators can learn new words/issues/ideas
Minuses
Focus group moderator training
Silence
Shy/dominating participants
Confidentiality
Staying on task/becoming a support group

30. Text/Document Analysis
Existing documents/text/records which include information of interest
Open ended questions or prompts that ask participants to reflect on a topic, theme, or issue

31. Text/Document Analysis
Provides insights into content and processes
Define sample
Sources
Years
Collate and ‘neutralize’
Define abstraction processes and rules

32. Text/Document Pluses and Minuses
Efficiencies in data collection
Minuses:
Influences of external cues
Reading is not the same as hearing
Cannot probe
Document analysis generally requires fewer resources than observations, interviews, and focus groups. This is particular true for electronic searches for documents. The thematic analysis of the documents requires as much effort as other qualitative methods. The sample can be clearly defined for document analysis, for example, the professionalism curricula from all medical schools.
Documents may have identifying cues, for example, unique course names, that might identify the course of the documents. Identifying cues might influence coding or analysis. One of the other limitations of document analysis is that the investigators only have the written document. If you are analyzing narratives of personal experiences, then you don’t have the non-verbal cues that you would have if you were interviewing subjects.

33. Exercise 2: Designing Instruments
Each group will be charged with developing a qualitative instrument to answer the following research question:
How do learners’ roles in patient care change over the course of the clerkship year?
Group 1: Observation tool
Group 2: Interview protocol
Group 3: Text/Document analysis

34. Exercise 2: Debrief
Groups share overall description and sample items from instrument
Discuss questions, challenges and suggestions in relation to each type of data collection method or specific instrument

35. Example: How does participation in a teamwork & communication skills curriculum improve small group process in Year 1 of medical school?
Observation of small groups pre & post
Interviews with small group leaders (retrospective comparison)
In-depth interviews with students
Focus group with students
Written evaluations of group process by students and faculty

36. Exercise 3: Interviewing Skills
We will role play 2 interviews between a qualitative researcher and participant.
After each interview, discuss:
How you feel the interview went
Thoughts about the questions asked and the responses elicited
What you think was done particularly well and/or what could be improved

37. Aligning Research Question & Method
Rule of thumb: Explain your choices (i.e. make your decisions transparent). This includes:
Why your choice of approach?
Why your data collection method and not another?
Modification of instrument in response to data collected
Discuss your choice for sampling

38. Rigor in Data Collection
Role and relationships with participants
Hierarchical relationships
Prolonged engagement
Establish rapport, trust
Script – pilot tested
Questions appropriate to capture perspectives
Non-leading, non-judgmental, open
Comportment
Participant Reactivity– observer effect, when participants may act differently
Reflection process
Data security
the HPE community use the phrase ‘participant reactivity’ when considering the participant, observer and research question triad; that researchers invest in interpersonal relationships at their study site to mitigate the effects of altered behaviour; and that researchers use theory to make sense of participants’ altered behaviour and use it as a window into the social world. The term ‘participant reactivity’ better reflects current scientific understandings of the research process and highlights the cognitive work required of participants to alter their behaviour when observed.

39. Combining Qualitative and Quantitative (Mixed) Methods
Development: to inform the development of one method from another, using the methods sequentially for the purposes of increasing construct validity
Complementing: to explore areas of overlap and uniqueness within a phenomenon to elaborate, illustrate or clarify results
Triangulation: to cross-check and corroborate results
Expansion: to increase the scope of inquiry by appropriately matching the methodology to various components of the question of interest
Initiation: to specifically discover inconsistencies and new perspectives that may be uncovered

40. Exercise 4 - Convincing a Colleague
“You are about to present the results of a survey evaluating your program to colleagues in your department. The results include both quantitative results (descriptive statistics for responses to structured questions) and an analysis of responses to open-ended questions. One of your colleagues says: “Let’s just look at the numbers, the hard data, not the anecdotes.” You would say (or do) . .

41. Summary Definition and characteristics of qualitative methods
Review 4 common types of data collection methods
Acquire skills to begin to develop questions, select methods and plan details for a qualitative project
Aligning questions with methods
Discuss rigor in qualitative research

42. Qualitative Rip Out Series http://www.jgme.org/page/ripouts
Balmer DF, Rama JA, Martimianakis MA, et al. Using data from program evaluations for qualitative research. J Grad Med Educ. 2016;8(5):773–774.
Baker L, Phelan S, Snelgrove R, et al. Recognizing and responding to ethically important moments in qualitative research. J Grad Med Educ. 2016;8(4):607–608.
Nimmon L, Paradis E, Schrewe B, et al. Integrating theory into qualitative medical education research. J Grad Med Educ. 2016;8(3):437–438.
Paradis E, O'Brien B, Nimmon L, et al. Design: selection of data collection methods. J Grad Med Educ. 2016;8(2):263–264.
Wright S, O'Brien B, Nimmon L, et al. Research design considerations. J Grad Med Educ. 2016;8(1):97–98.
Teherani A, Martimianakis T, Stenfors-Hayes T, et al. Choosing a qualitative research approach. J Grad Med Educ. 2015:7(4):669–670.

43. Teaching Material References
Bunniss & Kelly. Research paradigms in medical education. Medical Education, 2010, 44:
Lofland & Lofland. Analyzing social settings.
Rubin, H.J. & Rubin, I.S. (1995). Qualitative interviewing: The art of hearing data. Thousand Oaks, CA: Sage Publications
Strauss, A. & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park: Sage Publications
Cote, L., & Turgeon, J. (2005). Appraising qualitative research articles in medicine and medical education. Medical Teacher, 27(1),
Green J, Thorogood N. Qualitative methods for health research. London: Sage; 2005.
Thomas D. A general inductive approach for analyzing qualitative evaluation data. Am J of Eval. 2006; 27:

44. Exercise 1: Manuscript Discussion
Why was this study done? What questions or problems does the study examine?
Do you have a clear sense of the context of the study and the role of the researcher(s)?
How well do you feel the methodology used in this study captured the research question? Explain some of the strengths and/or limitations.
What other methods could have been used to study this topic?