AARON W. BERNARD, MD JONATHAN FISHER, MD Qualitative Methods in Educational Research

Objectives Define qualitative research Contrast with quantitative research Describe how to conduct qualitative research by using examples in the literature

Quantitative Research Test well-specified hypothesis concerning some predetermined variables Answer questions like;  Whether……..  How much….

Quantitative Educational Research Attempts to study links between teaching factors and learning outcomes Improved Medical Student Satisfaction and Test Performance with a Simulation-based Emergency Medicine Curriculum: A Randomized Controlled Trial. Ten Eyck. Ann Emerg Med. 2009;54:

Qualitative Research Qualitative research questions tend not to ask whether or how much but rather;  What  Why  How Qualitative research offers insight into social, emotional, and experiential phenomena. Qualitative reports do not typically generate answers but rather generate narrative accounts, explanations, conceptual frameworks.

Qualitative Research Qualitative research is inquiry-guided, inductive, and descriptive. Similar to an anthropologic or sociologic descriptive study.

Data Collection Methods Field observations Interviews  Acad Med Document analysis  Teach Learn Med

Data Analysis Grounded theory Coding Iteration Theoretical saturation

Qualitative Educational Research  Acad Emerg Med 2005;12:

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med Background  Effective teaching practices in the ambulatory setting have been documented  The ED is a different type of learning environment  A qualitative study to determine what EM teachers consider important has been done 1 What do EM learners consider important to effective EM teaching?  Quantitative methods vs.. qualitative methods Bandiera. Ann Emerg Med 2005; 45:

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med Methods  Focus groups at Five EM programs in Ontario  Multiple sites adds validity  Medical student, EM Resident, Off-service resident, Junior FRCP, Senior FRCP  Purposeful sampling  28 participants total

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med Methods  Semi-structure interviews by a trained moderator  Tape-recorded and transcribed by a third party  Participants used terms like “junior” to identify themselves to allow the data to be matched to type of participant

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med Methods/Data Analysis  Transcripts were reviewed by two independent assessors and a coding framework was derived using grounded theory methods  Novel ideas were assigned a new teaching code or a new qualifier as they emerged  When a new code was added the entire transcript was reviewed  The two codes that emerged were compared and consensus was reached  Agreement on 41 coded items and disagreement on 9  5/9 were subthemes

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med 2005.

Data analysis  The transcripts were then reviewed against the final code by one investigator to check for frequency of codes  A second investigator reviewed 20 percent of the transcript to allow for an estimation of interrater reliability  Computer software exists to help with this!

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med Summary  Qualitative methods were most appropriate for this investigation  Focus group interviews are a common method to obtain data  Purposeful sampling (low numbers, key players)  Grounded theory to create code  Frequency of themes to assess importance of themes to subject

Qualitative Educational Research  Academic Medicine 2010; 85:

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Background  The informal and hidden curriculum drives professionalism development  There is a lack of evidence regarding what students experience. Objective  To understand students experiences as they relate to an understanding of professionalism

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Methods  Students on IM required to post 2 narratives over 2 months  Retrospective review of narratives June – November 2007  272 narratives from 135 students

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Methods/Data Analysis  Thematic content analysis  Immersion/crystallization method  Reading and rereading narratives until themes emerge  Code was created with the first 50 narratives  A narrative could be coded more then once

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Reliability 1. One author created the code based on the first 50 narratives then two others independently reviewed the work. 2. Created focus groups of students to review themes

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Two major domains 14 themes 63.4% positive, 29.1% negative, 7.5 % hybrid

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

Direct representative quotes Smith AK, Fisher J. Ann Emerg Med

Direct representative quotes

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Summary  Qualitative methods were most appropriate for this investigation  Document analysis is a common method to obtain data  Methodological steps to improve validity is important  The use of direct representative quotes

Qualitative Educational Research  Teach Learn Med. 2010;22(1):28-32

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 Background  Disaster training is relevant to medical students who as physicians may be asked for guidance during training or real events.  Medical school offer little formal curriculum Objective  Provide 4 th year medical students with a didactic and clinically experience in disaster medicine  Evaluate the effectiveness through lecture evaluations and a qualitative analysis of focus groups

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 Methods  43 students  4 hour didactic session the evening before the drill  Training in the use of a standardized assessment tool for disaster medicine drills Evaluation and Focus Group  5 point likert scale surveys regarding the didactics  2 Focus groups after the drill Semi-structured interviews

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 Methods/Data Analysis  Median, means, SD for the survey data  Notes were taken in real time  One researcher performed the qualitative analysis and created the thematic domains of the code

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med of the 42 students participated in both the didactic and the drill 17 agreed to participate in the focus group Themes  Changes in self-perceived attitude toward DM  Changes in student’s ability to apply this knowledge in a simulated setting.

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 Effective use of qualitative research to create a descriptive study of an educational curriculum that is publishable. Combination of qualitative and quantitative methods

Final Conclusions Qualitative research is a tool that can be used for educational investigations. Qualitative research is inquiry-guided, inductive, and descriptive. Data usually comes from field observations, interviews, or document analysis. Grounded theory is used to review the data to create themes and subthemes. This is known as coding. Frequency of themes is often used as a marker of importance of themes to subjects studied Various techniques can be used to increase reliability of the analysis

References Tehaerani A, O’Brien B, Masters DE, et al. Burden, responsibility, and reward: preceptor experiences with the continuity of teaching in a longitudinal integrated clerkship. Acad Med. 2009;84(10 Supp):s50-s53. Alford CL, Currie DM. Introducing first-year medical students to clinical practice by having them “shadow” third-year clerks. Teach Learn Med. 2004;16(3): Thurgur L, Bandiera G, Lee S, et al. What do emergency medicine learners want from their teachers? A multicenter focus group analysis. Acad Emerg Med 2005; 12: Karnieli-Miller O, Vu R, Holtman M, et al. Medical students’ professionalism narratives: a window on the informal and hidden curriculum. Acad Med. 2010;85: Giacomini MK, Cook DJ, et al. Users’ guides to the medical literature XXIII. Qualitative research in health care A. Are the results of the study valid? JAMA. 2000;284: Smith AK, Fisher J, Schonberg MA, et al. Am I doing the right thing? Provider perspectives on improving palliative care in the Emergency Department. An Emerg Med. 2009;54: Kaji AH, Coates WC, Fung C-C. Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med. 2010;22(1):28-32