Threats to Internal Validity One of the criteria for making causal inference is to rule out alternative explanations. Threats to internal validity are confounds that serve as plausible alternative explanations for a research finding. There are eight general classes of confounds referred to as “threats to internal validity”: (1)History (2)Maturation (3)Testing (4)Instrumentation (5)Regression (6)Selection (7)Subject attrition (8)Additive effects with selection Selection + Maturation Selection + History Selection + Instrumentation

Threats to Internal Validity History Events that occur at the same time as treatment (X) which can change participants’ behavior Does an AIDS awareness/safe sex campaign on campus influence condom sales in campus vending machines? History threat: Suppose at Week 4, Magic Johnson announces he is HIV+. Can you conclude the campaign was effective?

Threats to Internal Validity Maturation Participants mature over time which may explain changes in participants Does a new school reading program improve 2nd graders’ reading comprehension? Maturation threat: Reading comprehension improves as children naturally mature over the year. Can you conclude that the reading program was effective?

Threats to Internal Validity Testing Taking a test generally affects subsequent testing Does showing students a new problem solving technique influence their ability to solve problems quickly? Testing threat: If similar problems are used in the pretest and posttest, faster problem solving may be due to familiarity with the test. Can we conclude that teaching the new technique improves problem solving?

Threats to Internal Validity Instrumentation Instruments used to measure participants’ performance may change over time Observers may become bored, tired, altered expectations Modification in the testing setup, mechanical, electronics, room location Operational definition of dependent variable Suppose a police protection program is implemented to decrease the incidence of rape as recorded in crime reports. Instrumentation threat: At the same time the program is implemented, reporting laws are changed such that what constitutes rape is more inclusive. Can we conclude that the program is effective (or ineffective)?

Threats to Internal Validity Regression (to the mean): if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement Participants placed into a group because of past performance For example: have done poorly on writing assignments Placed into a program to improve writing Participants sometimes perform very well or very poorly because of chance factors (not luck, intangibles). Such as mood, items on a test, motivation chance factors are not likely to be the same in a second testing scores will not be so extreme, the scores “regress to the mean” so in this they go up because of change of the intangibles

Threats to Internal Validity Regression Suppose students were selected for an accelerated enrichment program because of their very high scores on a brief test. Regression threat: Scores on the first test could be inflated from chance factors so we can expect their scores to regress to the mean during the second testing. Can we conclude the program is effective?

Threats to Internal Validity Subject Attrition Participants are lost from the study (attrition) group equivalence formed at the start of the study may be destroyed differences between treatment and control groups at the end of the study may be due to differences in those who remained in each group rather than to the effects of treatment. Suppose an exercise program is offered to employees who would like to lose weight. At Time 1, 50 participants sign up (M weight = 225 pounds). At Time 2, 25 participants remain (25 dropped out). Suppose the 25 who persisted in the program weighed, on average, 150 pounds at Time 2. Did the exercise program help people to lose weight?

Threats to Internal Validity Selection Differences exist between individuals in treatment and control groups at the start of the study A community recycling program is evaluated for effectiveness in reducing amount of garbage going into landfill. Individuals who are interested in recycling are encouraged to participate. The evaluation consists of weighing the garbage of participants in the recycling program and the garbage of those not participating in the new program. Can we tell if the new recycling program is effective?

Threats to Internal Validity Additive Effects with Selection When comparing two groups, experimental and control. Changes over time in one group but not the other. Selection + History: when one group of participants responds differently to an external event Section + Maturation: when one group of participants matures differently Selection + Instrumentation: when one group of participants is measured more sensitively by a test (instrumentation)

Additive Effects with Selection Section + History Does an AIDS awareness/safe sex campaign at School A influence condom sales in campus vending machines? History threat: Suppose at Week 4, Magic Johnson announces he is HIV+. Can you conclude the campaign at School A was effective? Yes, both groups should have experienced the same history threat equally.

Additive Effects with Selection Section + History Does an AIDS awareness/safe sex campaign at School A influence condom sales in campus vending machines? Additive Effect of Selection and History: Suppose at Week 4, the student newspaper at School A reports about students who are HIV+. Can you conclude the campaign at School A was effective?

Threats to Internal Validity Important Points to Remember: When there is no comparison group in the study, the following threats to internal validity must be considered: History Maturation Testing Instrumentation Regression Subject attrition Selection When a comparison group is added, the following threats to internal validity must be considered: Additive effects with selection

Problems That Even True Experiments May Not Control Threats to internal validity that true experiments may not eliminate: Experimenter expectancy effects Contamination communication of information about the experiment between groups of participants Novelty effects when an innovation such as new work environment is introduced as the treatment Hawthorne effect: changes in people’s behavior brought about by the interest that “significant others” show in them. Because of these problems researchers may have difficulty concluding whether a treatment was effective.

Problems That Even True Experiments May Not Control Expectancy Effects: This occurs when an experimenter unintentionally influences the results of an experiment. Experimenters can make systematic errors in their interpretation of participants’ performance based on their expectations. Experimenters can make errors in recording data based on their expectations for participants’ performance. For example, observer bias

Problems That Even True Experiments May Not Control Contamination: This occurs when there is communication about the experiment between groups of participants. Three possible outcomes of contamination: resentment: some participants’ performance may worsen because they resent being in a less desirable condition; rivalry: participants in a less desirable condition may boost their performance so they don’t look bad; and diffusion of treatments: control participants learn about a treatment and apply it to themselves.

Problems That Even True Experiments May Not Control Novelty Effects: This refers to changes in people’s behavior simply because an innovation (e.g., a treatment) produces excitement, energy, and enthusiasm A special case of novelty effects is the Hawthorne effect: performance changes when people know “significant others” (e.g., researchers, company bosses) are interested in them or care about their living or work conditions. Or disruption effects when the change disrupts the workplace

Quasi-Experiments Quasi- (“resembling”) experiments provide an important alternative when true experiments are not possible. Quasi-experiments lack the degree of control found in true experiments. Generally do not have random assignment of participants Researchers must seek additional evidence to eliminate threats to internal validity in a quasi-experiment. Take into account possible threats to internal validity for their particular study

The One-Group Pretest-Posttest Design This is a “bad experiment” and is sometimes referred to as a “pre-experimental design.” Not quasi-experimental design An intact group is selected for a treatment (e.g., a classroom of children, a group of employees). A pretest measure is used to record participants’ performance before treatment (O1— or “Observation 1”) The treatment (X) is implemented. A posttest measure is used to record performance following the treatment (O2). Observation 1 -- Treatment -- Observation 2 O1 X O2

The One-Group Pretest-Posttest Design The one-group pretest-posttest design is a bad experiment because none of the threats to internal validity are controlled. Any change between pretest (O1) and posttest (O2) scores may be due to the treatment (X) or due to: History (some other event that coincided with treatment), Testing (the effects of repeated testing), Maturation (natural changes in participants over time), or due to the other threats to internal validity

Quasi-Experimental Designs Three different designs Nonequivalent Control Group Design Simple Interrupted Time-Series Design Time Series with Nonequivalent Control Group Design

Quasi-Experimental Designs Nonequivalent Control Group Design: a group similar to the treatment group serves as a comparison group researchers obtain pretest and posttest measures for individuals in both groups. random assignment to groups is not used pretest scores are used to determine whether the groups are equivalent

FIGURE 10.3 Langer and Rodin (1976) used a nonequivalent control group design to study the effect of two different types of responsibility instructions on the behavior of nursing home residents. Because a “true experiment” was not conducted, the researchers examined features of the study to determine if any threats to internal validity were present.

Nonequivalent Control Group Langer and Rodin (1976) Lack of opportunity to make personal decision contributes to psychological debilitation amongst nursing home residents. Comparison Groups (1) Treatment group: communication stressing independent decision making were also given a small plant as a gift (if they decided to accept it) living on one floor of the nursing home (2) Nonequivalent control group: communication stressing staff responsibility for them also received a plant as a gift (whether they chose to have one or not) living on another floor of the nursing home Questionnaires about control and happiness One week before treatment Three weeks after treatment

Nonequivalent Control Group Suppose group differences are observed at a posttest. Rule out threats to internal validity: By adding a comparison group, researchers can rule out threats due to history, maturation, testing, instrumentation, and regression. We assume that these threats happen the same to both groups, therefore, these threats can’t be used to explain posttest differences.