QUALITATIVE RESEARCH METHODS CONTENT-ANALYSIS

CONTENT ANALYSIS  I. OVERVIEW  A. A method for analyzing the nature, functions, & effects of texts. 1. Originated in 18th century Sweden when a group of scholars & clergy analyzed a collection of 90 non-orthodox hymns. 2. Counted religious symbols in hymns to see if they were heretical. 3. Modern biblical exegesis of texts still uses content analysis as a method (along with other critical methodologies).

CONTENT ANALYSIS  4. Also used in mass media research. a. 1st applied to content of leading newspapers in 1890s (Wilcox) b. In 1930s, applied to radio & public speeches c. In WWII, invaluable in examining German radio transmissions d. More recently, used to examine television programs  B. Sometimes seen as descriptive work

CONTENT ANALYSIS  C. Defining content-analysis is confusing & in dispute. a. Some define broadly as any technique that objectively analyzes messages in a “systematic, rule-governed & rigorous way” ( Hocking, Stacks & McDermott, 2003, p. 171). b. This definition permits more qualitative content-analysis. c. Others, however, would not include purely qualitative studies as “true” content analysis (e.g. Budd, Thorpe, & Donohew, 1967).

CONTENT ANALYSIS  3. Frey, Botan & Kreps(2000):“textual analysis”is a continuum of quantitative & qualitative approaches.  4. Standard definition of content analysis: A method of studying & analyzing messages in a systematic, objective, & quantitative manner for the purpose of measuring variables.  5. Qualitative content analysis (from Bernard & Ryan, 2010): “a set of methods for systematically coding & analyzing qualitative data” (p. 287)  a. Explores meanings in texts  b. Different from “grounded theory” or critical analysis because usually quantitatively analyzed

CONTENT ANALYSIS  5. Traditional quantitative content analysis aims for objective rigor like any social science.  a. Operationalization of concepts.  b. Classification criteria must be both explicit & comprehensive.  c. Coders essential for more objectivity & reliability.  d. Study should be able to be replicated by another researcher with similar results.

CONTENT ANALYSIS  e. Requires random sampling for generalization  f. Requires equal treatment of all content (through a uniform coding process).  6. Main goal--the accurate & precise description of messages.  a. Usually done on many texts, not just one.  b. Quantification aids in making descriptions.  c. Quantification aids in comparisons among categories & sets of data.  d. Seek generalizable results from sample to population (called a universe).

CONTENT ANALYSIS  D. Functions & uses of content analysis  1. Description of message characteristics--what, how, & to whom something is stated or occurs in a text. a. To compare different data sets. b. To compare content to the “real world.” c. To assess images of particular groups in society. d. To describe cause & effect.

CONTENT ANALYSIS  2. Explanation of message characteristics:  a. Examining a dependent variable.  b. Measuring an independent variable.  c. Testing hypotheses--relate certain characteristics of the source of a given message to its message characteristics. 1) Make inferences about causes--why something is stated or occurs in a text. 2) Make inferences about effects--what happens when something is stated or occurs in a text.

CONTENT ANALYSIS  II. PROCEDURES FOR CONDUCTING A CONTENT ANALYSIS  A. Setting up the study: 1. Identify the research problem. 2. Develop research questions (RQs) based on theory and/or prior research 3. Sometimes develop research hypotheses (RHs), testing IV/DV relationships

CONTENT ANALYSIS 3. Determine a suitable data base to test questions or hypotheses. a. Select textual artifacts directly pertinent to the research problems. b. If can’t do a census of all relevant texts must take a representative sample.  B. Obtaining a representative sample.  1. Operationalize “population of interest” or universe of texts  2. After defining population boundaries, specify the smallest unit or elements to be studied.

CONTENT ANALYSIS  3. Construct a sampling frame.  4. Take a sample of texts a. Random sampling--key component for quantitative content data (to be able to generalize results) b. Can use non-random sampling if not testing a hypothesis  C. Collect data from sample, along with contextual information:  1. Relevant data about extrinsic matters, such as source data, historical background, documented effects, etc.

CONTENT ANALYSIS  2. Contextual information can emerge from comprehensive literature review.  3. Can also emerge through audience surveys, or interviews with the source(s), looking at historical data, etc.  4. Helps justify inferences made from the intrinsic data analysis.  D. Develop a measurement system  1. Unitizing--identifying the units of analysis (or message elements).

CONTENT ANALYSIS  2. Five common units: a. Physical--actual kinds of texts, speeches, shows, etc. b. Syntactical--symbols, metaphors, words, etc. c. Referential --what the text is about. d. Propositional--specific positions taken in the text. e. Thematic--specific topics or themes.

CONTENT ANALYSIS  3. Count number of times the relevant units appear in a text.  4. Place units into appropriate categories:  a. Pre-formulated--categories drawn from research.  b. Emergent--categories emerge from the data itself.  5. Define category boundaries with maximum detail (helps with reliability).

CONTENT ANALYSIS  6. Criteria for categorizing:  a. Mutually exclusive--every unit fits into only one category.  b. Equivalent--all categories are similar in type.  c. Exhaustive--every unit fits into one of the categories.  d. Any “other” category shouldn’t contain more than 10% of the units.

CONTENT ANALYSIS  7. Categories also must be valid:  a. Face validity  b. Semantic validity 1) Confirmation that words chosen for both units & categories mean what you say they mean. 2) Panel of informed judges can provide confirmation.  c. Criterion-related validity  d. Construct validity

CONTENT ANALYSIS  E. Coding the data  1. Place units into categories (called coding) a. Codes related to variables or themes in the research question/hypotheses b. Best if pretest on a small sample of the texts to fix any problems in the codes  2. Researchers seldom (or rarely) code units themselves a. Introduce bias or error into the analysis b. Independent coders permit reliability to be assessed  3. Use at least two or more independent coders (also known as raters, or judges); or can use a computer to do the coding

CONTENT ANALYSIS  3. Coders must be trained, but ideally are “naïve”  4. Do pilot study to obtain an interrater (intercoder, interobserver) reliability coefficient  a. Assess level of agreement among raters  b. If closer to 1.00, the greater the degree of intersubjective agreement (hence reliability)  5. Revise coding scheme  6. Recode the data

CONTENT ANALYSIS  5. As apply the codes to the texts, keep checking for coder reliability a. Manifest coding usually highly reliable (e.g. looking for words or phrases) b. Latent coding permits freedom to find themes, but less reliable  6. Can develop a case-by-variable matrix from the text & codes (e.g. different paragraphs would be the rows, while the themes or units—such as gender or some other category--would be the columns)

CONTENT ANALYSIS  7. Using coding schemes  a. Can code or classify questions (closed or open- ended)  b. Code or classify observations  c. Closed schemes use checklists & predetermined categories (used most in experimental, content & interaction analysis)  d. Open schemes permit patterns to emerge (more used in qualitative content-analysis, ethnography, grounded theory, & some critical work)

CONTENT ANALYSIS  F. Analyzing & interpreting the data.  1. Can numerically code units, then apply statistical procedures to the data a. Usually nominal or ordinal level data 1) Nominal--discrete categories of data (instead of degree or amount), such as male-female 2) Need at least 2 different categories which are mutually exclusive, equivalent, & exhaustive b. Ordinal data--discrete categories rank ordered along some type of dimension (e.g. ratings, judging categories, etc.)

CONTENT ANALYSIS  2. Can use a Chi-Square statistic to assess differences or relationships  3. Can also determine means, modes, medians, etc.  4. If have interval level data (equi-distance ranking, as in a Likert-type scale), can also use inferential statistics: 1) Use a t-test for comparisons 2) Use a frequency distribution  5. Can also do qualitative analysis, focusing on themes & patterns

CONTENT-ANALYSIS  2. From analysis, interpret the data a. Discuss relationship of the variables, depending on statistical tests run b. If descriptive, need to justify interpretations 1) Can link to trends (e.g. sex-role stereotyping in media) 2) Can consider linkages among categories  c. Basically answering your questions or supporting hypotheses that build theory

III. Assessing Content-Analysis  A. Advantages 1. Unobtrusive technique 2. Accepts unstructured material 3. Can handle large amounts of data 4. Studies communication in context 5. Can study changes in messages over time 6. Can be used to support theory 7. Easy to triangulate with both quantitative & qualitative methods

ASSESSING CONTENT ANALYSIS  B. Disadvantages  1. Not relevant to all research (e.g. a few texts)  2. Cannot determine the truth of an assertion  3. Cannot evaluate the aesthetic qualities of a message  4. Establishing cause is tricky  5. Interpretations of meaning can be biased  6. Requires trained coders, so is time-consuming  7. If don’t use trained coders, reliability (and thus validity) is suspect