1. Evaluating Qualitative Research
INFO 272. Qualitative Research Methods
12 November 2009
Evaluating Qualitative Research

2. Typical Reactions is not generalizable / is “anecdotal”
The sample is too small to say anything / is not a random sample / not representative
What is the hypothesis you are testing?
Great stories, but can you show me some data that supports your claims?
is subjective, the researcher’s presence in the setting biases the data
lacks rigor, procedure is unsystematic

3. Becker – epistemology of qual research
Quantitative Tradition
Qualitative Tradition
Reliability – reproducing the findings through the same procedures, same findings from multiple observers
Accuracy – based on close observation not remote indicators
Validity – the degree to which one measured the phenomenon one claims to be dealing with
Precision – captures the outline and dimensions of the thing discussed
Breadth – knowledge of a broad range of matters that touch on the topic

4. Criteria for Quant Research
The “Holy Trinity”
Reliability
Validity
Representativeness and Generalizability
Internal
External
Content
Criterion
Construct

5. Functional Equivalence
Criteria for evaluating quantitative research is not directly applicable to qualitative research
Can we draw out some abstract, general standards and then respecify for qualitative research
Kvale on epistemology
Abandoning a correspondence theory of truth
Defensible (rather than absolute) knowledge claims requiring argumentation

6. Functional Equivalence
Quantitative Tradition
Qualitative Tradition
Reliability of measures (c)
Confidence (c)
Relevance (r)
Triangulation and reflexivity (c)
Internal validity (c)
Transparency and procedural clarity (c)
Sample size (c)
Corpus construction (c, r)
Representative sampling (r)
Thick description (c, r)
External validity (r)
Local surprise (r)
Validity of measures (r)
Communicative validation (r)

7. Triangulation and Reflexivity (c)
In situ verification process
i.e. interviews about Internet use supplemented by observation

8. Transparency (c)

9. Corpus Construction (c, r)
Maximizing the diversity of unknown representations and mapping those representations
Representativeness and ‘external validity’ is a matter of argumentation

10. Thick Description (c, r)
‘high-fidelity’ reportage: verbatim quotes – demonstrating the provenance of a claim
Footnotes and sources
But also, do you get a whole picture of the social world, its elements, and how they are interlinked? Especially the meaning of the social phenomenon.

11. Local Surprise (r) Surprise in relation to a common-sense view
Surprise in relation to theoretical expectation
Solely confirming evidence (just as totally consistent evidence) should raise suspicion

12. Communicative Validation (r)
Gaining feedback from research participants (and others?)
Remember interviewing technique of ‘interpreting’ on the fly to get confirmation from interviewees

13. Ethnography in organizational settings (Jordan and Dalal)
Focus groups…what people say is not what they do
Not generalizable…rapidly expanding corpus of individual and research community experience
Not a science…a discovery science for handling the complex social world that does not function well as a laboratory
Show me results…creative presentation modes

14. The Future of Evaluation
Websites and digital archives that make qualitative data accessible to the public

15. Summary Make your methods visible Make your data (ideally) available
Continual verification in situ (as part of your iterative process)
Closeness to the social phenomenon and openness to surprises, the counter-intuitive
Re-read Becker on the “epistemology of qualitative research” for further suggestions