Introduction to Research Design Threats to Internal Validity Two or More Groups Social Threats

Selection Comparison groups are selected, or subjects selected into group, such that the groups differ on the criterion variable prior to administration of the treatment. This is a common threat in “quasi- experiments, where –An IV is manipulated, but –Subjects are not randomly assigned to groups

Static-Group Comparison Design N (X) O N O Find two existing groups –one has experienced the treatment –the other has not Without a pretest, cannot tell if the groups differed prior to the treatment Independent t or nonparametic equivalent

Selection: Example You want to evaluate the effectiveness of a computer tutorial. Subjects are allowed to self-select into the comparison groups. Posttest comparison shows that the groups differ. Why do they differ?

Pretest-Posttest Nonequivalent Groups Design N O X O N O O You have pretest and posttest, and The IV is manipulated, but Subjects were not randomly assigned to groups This is also a “quasi-experiment.”

Interaction with Selection You want to compare one educational program with another. One is offered at Suburban High, the other at Central High. You have both pretest and posttest data on the criterion variable, scores on an achievement test.

Suburban shows greater gains during the year than does Central. Why?

Selection x Maturation Subjects in one group are maturing more quickly than those in the other group. Students at Suburban are maturing, intellectually, at a faster pace than those at Central and would do so regardless of the treatment.

Selection x History Extraneous events, between pre- and post-tests, were quite different in the two groups. At Central there was a student riot and a teachers’ strike. No such disruption at Suburban.

Selection x Testing The effect of the pretest is different for one group than for the other group. At Suburban the teachers paid attention to what was on the pretest and made an effort to teach that to the students. At Central no such special effort was made.

Selection x Mortality (Differential Attrition) Subjects with certain characteristics are more likely to drop out of one group than the other group. Drill Sergeant Stedanko is running the program at Suburban, 60 of 80 students drop out of the program. At Central only 10 of 80 students drop out.

Students at Central made greater gains. Why?

Selection x Instrumentation: Example 1 The characteristics of the instrument are different for the one group than for the other group. Max score on the achievement test = 60. Most of the students at Suburban were already near the max, no room to show improvement with this test. This is called a Ceiling Effect.

Students at Central made greater gains. Why?

Selection x Instrumentation: Example 2 No ceiling effect problem, but At Central, the examiners misunderstood the instructions and gave the students only half the allotted time on the pretest. They gave them the correct time on the posttest. No such problems at Suburban.

Selection x Regression A regression artifact is greater in the one group than in the other group. At Central, the teachers made sure that all of their students with poor achievement test scores were enrolled in the special program. No such selection at Suburban. Greater regression up towards the mean at Central.

Social Threats to Internal Validity Participants know that they are involved in research. May know there is another group that is treated differently. They, or others, may feel that this is not fair. Leading to …………………………. 

Diffusion/Imitation of Treatment Students in the experimental treatment are taught using some new techniques. The students share these new ideas with their friends in the control group. Or the teacher in the control class imitates the experimental teacher.

Compensatory Rivalry Teacher in control class feels threatened by the special class. Control teacher works harder than usual to be sure that her students do well.

Resentful Demoralization Students in control class don’t believe assignment to treatments was random. We aren’t getting the special treatment because we are not worthy.

Compensatory Equalization of Treatment You learn your child was assigned to the control group. You want the special experimental benefit for your child. You call up the principal and demand. Principal moves your child or provides some special compensatory treatment to the “control” class.