COM 633: Content Analysis Reliability Kimberly A. Neuendorf, Ph.D. Cleveland State University Fall 2010

Reliability Generally—the extent to which a measuring procedure yields the same results on repeated trials (Carmines & Zeller, 1979) Types: Test-retest: Same people, different times. Intracoder reliability. . . Alternative-forms: Different people, same time, different measures. Internal consistency: Multiple measures, same construct. Inter-rater/Intercoder: Different people, same measures.

Index/Scale Construction Similar to survey or experimental work e.g., Bond analysis—Harm to female, sexual activity Need to check internal consistency reliability (e.g., Cronbach’s alpha)

Intercoder Reliability Defined: The level of agreement or correspondence on a measured variable among two or more coders What contributes to good reliability? careful unitizing, codebook construction, coder training (training, training!)

Reliability Subsamples Pilot and Final reliability subsamples Because of drift, fatigue, experience Selection of subsamples Random, representative subsample “Rich Range” subsample Useful for “rare event” measures Reliability/variance relationship

Reliability Statistics - 1 Types Agreement Agreement beyond chance Covariation Core assumptions of coefficients “More scholarship is needed”—these coefficients have not been assessed!

Reliability Statistics - 2 My recommendations Do NOT use percent agreement ALONE Nominal/Ordinal: Kappa (Cohen’s, Fleiss’) Interval/Ratio: Lin’s concordance Calculated via PRAM Reliability analyses as diagnostics, e.g., Problematic variables, coders (“rogues”?), variable/coder interactions Confusion matrixes (categories that tend to be confused)

PRAM: Program for Reliability Analysis with Multiple Coders Written by rocket scientists! Trial version available from Dr. N!