When All You Have Are Words: Qualitative Data Analysis Techniques Tasha R. Wyatt, PhD 6/27/16 Educational Innovation Institute (EII)

Words… Reflect, Shape, Reveal Beliefs Background Experience Conversations Intentions Actions Perspective Policies Experience Identity

Reveal what is both in plain sight and hidden.

Qualitative Analysis Evaluation Data Research Data What is working? What is not working? How can we make improvements? How does this theory explain participants’ perspectives? How do these perspectives contribute to the development of a new theory?

Inductive Inquiry* SIMPLE Purpose: Allow research findings to emerge from the frequent, dominant or significant themes inherent in the raw data SIMPLE “Goal free” – describes actual program effects, not just planned effects *Commonly used in health and social science research and evaluation

How do you do it…? 3. Get to Know Your Data: Do a close reading of the text. Be familiar with the themes and events in the text. 4. Create Categories: Upper level categories are derived from the evaluation aims. Lower level categories are derived from multiple readings. Step by Step 5. Revision & Refinement of Categories: Within each category, look for subtopics, including contradictory points of view and new insights. Select quotes that convey core themes. 1. Clean your data! 2. Get Clarity: Data analysis is guided by evaluation objectives. These provide a focus or domain of relevance.

Can we trust your work? Peer Debriefing: Discuss findings with peers. Does this interpretation make sense? Do you see alternative explanations? Checks for Inter-Rater Reliability: Independent Parallel Coding: 1st coder conducts analysis; 2nd coder repeats Categories are compared and merged Member Checks: Check with participants, funding agencies, and providers who comment on categories and interpretations. Clarity of Categories: 1st coder conducts analysis; 2nd coder is given raw data and asked to use 1st coder’s categories

Writing Findings When writing up your findings: Identify a label for the category Include the authors’ description of the meaning of the category Provide a quote from the raw text to elaborate on the category

Analyzing Data for Research What does it mean to analyze content? What are the various ways to analyze content? How do you choose among different techniques?

What Does It Mean to Analyze Content? Examination of large amounts of text The purpose is to organize text in a meaningful way: Types of text: Interviews, focus groups, meeting notes, open-response comments, lesson plans, syllabi, websites, evaluations, social media, student projects, etc. Brings understanding to specific phenomena Used to analyze explicit or inferred communication

“how” and “why” How do medical students feel about moving to entrustable professional activities (EPAs) at MCG? How are EPAs shifting the way students prepare for residency? Why are some medical students pushing back?

Technique 1: Emergent Content Analysis Describe a phenomena where there is little published research Hmm…what is going on here? I see a pattern. What does it mean? I’ve never seen anyone respond this way! I wonder why I am seeing this. The goal is to develop a concept or a model

When you feel tipsy… let the data speak.

Step by Step Step 1: Immersion with all the data Step 2: Code data directly from the text Step 3: Sort into categories/meaningful clusters Step 4: Define each category, subcategory, code Step 5: Results contribute to knowledge development

Developing Trustworthiness Build Trust with others Peer debriefing Discussion with a disinterested peer Prolonged engagement Adequate time with participants Persistent observation Themes, information begins to repeat itself Triangulation Linking data sources Negative Case Analysis Looking for examples of the opposite Member checks Verifying respondent answers

Technique 2: Directed Content Analysis Develop, extend or validate theory and theoretical frameworks Existing theory or research helps shape codes and relationship between codes Step 1: Code using pre-determined codes. Data that doesn’t fit gets a new code Step 2: Identify sub-codes for granulation

Apriori Coding… Shaping the data to fit your codes You define the codes (from theory) and then overlay them on the data.

Developing Trustworthiness Use an audit trail or audit process An auditor should review definitions of codes before the study Increases the accuracy of pre-determined codes

Technique 3: Summative Content Analysis Quantify certain words or content in a text to understand their contextual factors Objective: Explore usage and range of meaning in various contexts/speakers Educators vs. Clinicians Medical Education Textbooks Online forums

When are these phrases used? “Transitioning” “Crossing Over” “Meeting the Angels” “Passing Away” “Moving-On” “Dying” “Going to the Other Side” “Going to the Great Choir in the Sky”

Summative Content Analysis Step 1: Search for identified words Step 2: Identify word frequency for given word Step 3: Analyze the context and/or speaker for use When is it used? By whom? What were they trying to say/not say? What are the implications for this usage?

Developing Trustworthiness You must demonstrate credibility Use experts for definitions The goal is to demonstrate internal consistency (show evidence is consistent with interpretation) Member checking to check intended meaning

Choosing a Technique You must first understand what you are trying to achieve.

Questions to Ask… Am I trying to create a theoretical model to better understand this area? Emergent Content Analysis Do I want to see how this theory explains students actions/beliefs/identity? Directed Content Analysis Do I want to better understand the language currently being used to describe a shifting construct? Summative Content Analysis

Don’t Analyze On Your Own NVIVO Atlas Ti Dedoose Excel

Qualitative Data Analysis Software NVIVO http://www.qsrinternational.com/product Dedoose http://www.dedoose.com/ Atlas http://atlasti.com/ Excel You know where to find this!

Questions? Consider a consultation with the EII: edi@augusta.edu Qualitative Data Analysis Research Design Advice on Manuscript Prep