1. Content Analysis: Measurement and Validity Kimberly A. Neuendorf, Ph.D. Cleveland State University Fall 2011

2. Measurement  “Assignment of numerals to objects or events according to rules” (Stevens, 1951)

3. Operationalization (Measurement)  Categories that are exhaustive and mutually exclusive  High and appropriate level of measurement Ratio Interval Ordinal Nominal

4. Evaluating Measures: Comparing Reliability, Accuracy (lack of Bias), and Precision  Targets…also see p. 114 of book.  See also: Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Newbury Park, CA: Sage Publications.

6. High Reliability High Accuracy High Precision

8. High Reliability Low Accuracy High Precision

10. Low Reliability Moderate Accuracy High Precision

12. Reliability? High Accuracy Low Precision

13. Validity  Internal Validity Conceptual definition =matches= Operationalization The standard of having a “good” measure—”Are we hitting the target we want?”  External Validity Generalizability  Sample  Ecological validity  Replicability

14. Validity  “Face” Validity  Criterion Validity Relate to established standard or behavior  Concurrent  Predictive e.g., Oliver & Kalyanaraman (2002) study of sexual and violent content in trailers and films... Criterion is the MPAA rating

15. Validity  Content Validity Construct = ???

16. Validity  Content Validity

17. Validity  Content Validity

18. Validity  Content Validity High internal consistency reliability...

19. Validity  Construct Validity Related to other measures (constructs) in ways consistent with theory  Convergent—relationships as expected  Discriminant—lack of relationships, as expected

20. Construct Validity  Smith volume (1992) on “thematic content analysis”—psychographic applications—has examples Trust-mistrust coding system for TAT stories construct validated against:  Other thematic coding systems (e.g., Affiliation; Intimacy)  Demographics (e.g., Mothers’ feeding habits during infancy (more or less principled), moderated by gender & age)  Self-reported behaviors (e.g., Reported positive relationships with others; Divorce, family fights in last year; Drug use)  Personality measures (e.g., Impulsivity; Locus of Control)

21. Validity Assessment & CATA  Short et al. (2010) provides a model for validity and CATA, using tests of a new dictionary for “entrepreneurial orientation” Content Validity  Deductive  Inductive External Validity Reliability Dimensionality (a type of Construct Validity--Discriminant) Predictive Validity (seems to really be Construct Validity— Convergent???)

22. Validity Assessment & CATA  A Slight Adaptation of Short et al. (2010), converging with Carmines and Zeller (1979): Face Validity Content Validity  Deductive  Inductive External Validity Construct Validity--Dimensionality (Discriminant Validity) Construct Validity--Predictive (Convergent Validity) Criterion Validity (Concurrent or Predictive) [Reliability—separate, but without it, validity is also compromised]

23. Apply to the iCarly Case? (Disrespectful Behaviors)  A Slight Adaptation of Short et al. (2010), converging with Carmines and Zeller (1979): Face Validity Content Validity  Deductive  Inductive External Validity Construct Validity--Dimensionality (Discriminant Validity) Construct Validity--Predictive (Convergent Validity) Criterion Validity (Concurrent or Predictive) [Reliability—separate, but without it, validity is also compromised]

24.  pause