Evaluating Qualitative Research INFO 272. Qualitative Research Methods 17 April 2008 Evaluating Qualitative Research

Criteria for Quant Research The “Holy Trinity” Reliability Validity Representativeness and Generalizability Internal External Content Criterion Construct

Functional Equivalence Criteria for evaluating quantitative research is not directly applicable to qualitative research Kvale on epistemology Abandoning a correspondence theory of truth Defensible (rather than absolute) knowledge claims requiring argumentation

Functional Equivalence We must abstract existing quantitative standards to be respecified for qualitative research

Functional Equivalence Quantitative Tradition Qualitative Tradition Reliability of measures (c) Confidence (c) Relevance (r) Triangulation and reflexivity (c) Internal validity (c) Transparency and procedural clarity (c) Sample size (c) Corpus construction (c, r) Representative sampling (r) Thick description (c, r) External validity (r) Local surprise (r) Validity of measures (r) Communicative validation (r)

Triangulation and Reflexivity (c) i.e. interviews about Internet use supplemented by observation

Transparency (c)

Corpus Construction (c, r) Maximizing the diversity of unknown representations and mapping those representations Representativeness and ‘external validity’ is a matter of argumentation

Thick Description (c, r) ‘high-fidelity’ reportage: verbatim quotes – demonstrating the provenance of a claim Footnotes and sources But also, do you get a whole picture of the social world, its elements, and how they are interlinked?

Local Surprise (r) Surprise in relation to a common-sense view Surprise in relation to theoretical expectation Solely confirming evidence (just as totally consistent evidence) should raise suspicion

Communicative Validation (r) Gaining feedback from research participants (and others?) Remember interviewing technique of ‘interpreting’ on the fly to get confirmation from interviewees

Additional thoughts on validity Kvale: Checking, questioning findings through an iterative approach Pragmatic validity

Kvale on generalizability Statistical – formal and explicit Analytical – generalization through assertional logic

Defending Ethnography Focus groups…what people say is not what they do Not generalizable…rapidly expanding corpus of individual and research community experience Not a science…a discovery science for handling the complex social world that does not function well as a laboratory Show me results…creative presentation modes