1. Qualitative Methods for Health Program Evaluation
CHSC 433
Module 5/Chapter 12
L. Michele Issel, PhD
UIC School of Public Health

2. “Different kinds of problems require different types of data.”
Patton, 1997

3. Objectives List the major qualitative designs
List at least one pro and con for each of the major qualitative designs
Provide an outline of how qualitative data analyses are done

4. Beyond the Paradigm Debate
History of science favored quantitative (empiricism), deductive hypothesis testing, logical postivism
Current science favors understanding based on rigorous methods

5. Use Qualitative when
Want to minimize research manipulation by studying natural field setting.
Program aims at individual outcomes (so when program aims at common outcomes across individuals, use quantitative methods).

6. Key Characteristics
The use of non-numeric data, such as narratives, pictures, music
The use of subjective, experiential, naturalistic inquiry to explain phenomena
Use of inductive, iterative analysis
Holistic and contextual concerns
Pays attention to individual’s uniqueness

7. Functions of Qualitative Methods (Adapted from Green & Lewis ‘86)
1. Develop and delineate program elements
2. Booster power of quantitative designs
3. Broad the observational field
4. Analyze processes and cases to understand why or how the program worked
5. Generate a program or intervention theory
6. Use instead of quantitative methods

8. Underlying Perspectives
Phenomenology- experiences and meanings
Ethnography- culture
Critical analysis- communication and power
Grounded Theory- discovery of theory
Content Analysis-manifest meanings in the written word

9. Perspective --> Question
What does it mean to the person?
What are the norms and values (culture)?
How has power shaped it ?
What are the relationships (theory)?
What themes are in the text?
Phenomenology
Ethnography
Critical analysis
Grounded theory
Content analysis

10. Major Types of Qualitative Methods
Participant observation
Case studies
In-depth Interviews
Focus groups
Open-ended survey questions

11. Participant Observation
Acting as a member of a group, collect data
Make narrative notes and memos about processes, events, people observed
Use key informants to verify data analysis

12. Case Studies Define what is a case (organization, program, person)
Use variety of types of raw data generated by or about the case: memoranda, observations, surveys, interviews, etc

13. Case Study
Key Benefits for use in planning and evaluation
Key Challenges for use in planning and evaluation
Allows for understanding of context as influence on program or participant

14. In-depth Interviews
Use open-ended questions with key individuals (participants, key informants)
Use probes to clarify and explore issues or topics alluded to by the respondent or earlier data analysis
Use tape recorder and transcripts of the interviews

15. In-depth Interviews
Key Benefits for use in planning and evaluation
Key Challenges for use in planning and evaluation
Provides rich insights into personal thoughts, values, meanings, and attributions

16. Focus Groups
Carefully selected group of individuals who participate in guided discussion about a specific topic
Use a facilitator and a recorder

17. Focus Groups
Key Benefits for use in planning and evaluation
Key Challenges for use in planning and evaluation
Inexpensive given the amount and type of data, get collective views rather than individual views

18. Observations
Non-participatory and Participatory techniques can be used
Need training on what will be observed and how will record the observation
Data collection methods vary:
Audio-visuals recording
Field notes
Logs

19. Observations
Key Benefits for use in planning and evaluation
Key Challenges for use in planning and evaluation
Can identify sequence of causes and effects, may identify new behaviors or events

20. Open-ended survey questions
Use open-ended question placed at end of quantitative survey
Unable to use probes for clarification
Handwriting and spelling can make interpretation difficult

21. Sampling for Naturalistic Inquiry
Small purposive samples
Select for a specific characteristic
Theoretical sampling
Select based on what “ought” to matter
Sample for category saturation
Select until no new information is gained from participants

22. Data Analysis Coding and interpreting the data
To count or not to count

23. Coding Terminology Category- classification of concepts in the data
Dimension- implies continuum
Property- attributes or characteristics of a category
Constant comparison- process to develop categories, involves comparing new with existing categories
Codable- unit of data to be categorized

24. Analysis Procedures Identify codable units of data
Understand the meaning
Discover categories
Name categories
Discover properties and dimensions of the categories
Generate explanation

25. Scientific Rigor = Trustworthiness (Lincoln & Guba, 1985)
Credibility ~ Internal validity
Transferability ~ External validity
Dependability ~ Reliability
Confirmability ~ Objectivity

26. Credibility Have confidence in the truth of the findings by:
Invest sufficient time, triangulate
Use outsiders for insights (peer debriefing)
Refine working hypotheses with negative cases
Check findings against raw data
Use participant feedback

27. Transferability Applicability to other contexts and respondents
Provide thick (detailed, comprehensive) descriptions for others to assess possibility of transferability

28. Dependability Find same results if repeated the study
Leave a trail that can be followed so that others can see the findings are supported by the data

29. Confirmability Findings are from the respondents not the researcher
Leave an Audit Trail (same as for Dependability)

30. To count or not to count
Number of participants who mentioned a category
Number of times category mentioned throughout the study
Issues…. Neither help with interpretation of meanings, both can misrepresent the scope of the sentiment

31. From Data to Description
Categories as typologies are rudiments of a theory
Category dimensions and properties as essential
Linkages between categories form theory

32. Data Presentation Use descriptions of context to show transferability
Use tables showing category development to show dependability and confirmation
Use participants’ words to show confirmation
Use diagrams of relationships among categories

33. Realities of Data Analysis
Messy, confusing, repetitive
Iterative category development
Overwhelming quantities of data
Conflicting interpretations of data by peers and participants
Manifest versus implied meanings cloud data analysis
Investigator biases

34. Cost of Data Collection
Interview time
Travel to interviewee
Reading and listening
Transcription time
1 hour interview: 3 hours transcribing
Data analysis time

35. Evaluation Caveats Integrate with quantitative data
Use of participant feedback (credibility) as key to acceptance of findings
Stories are more powerful than numbers and make the numbers more human

36. Qualitative Methods Across the Pyramid
Each qualitative method has potential usefulness for programs at each level of the Pyramid.