1. Quasi-Experiments – Outline
True Experiments
Characteristics
Threats to validity controlled by experiments
Threats not controlled by experiments
Obstacles to true experiments in the field
Quasi-experiments
The logic of quasi-experiments
Non-equivalent control group design
Example – Langer & Rudin (1976)
Interrupted time-series design
Example – Campbell (1969)
Quasi

2. True Experiments - Characteristics
True experiments are characterized by:
A manipulation
A high degree of control
An appropriate comparison (the major goal of exerting control)
Manipulation in the presence of control gives you an appropriate comparison.
Quasi

3. Threats to validity controlled by true experiments
History
occurrence of an event other than the treatment
(Campbell & Stanley, 1966)
Quasi

4. Threats to validity controlled by true experiments
History
Maturation
participants always change as a function of time. Is change in behavior due to something else?
Quasi

5. Threats to validity controlled by true experiments
History
Maturation
Testing
improvement due to practice on a test (familiarity with procedure, or with testers expectations)
Quasi

6. Threats to validity controlled by true experiments
History
Maturation
Testing
Instrumentation
especially if humans are used to assess behavior (fatigue, practice)
Quasi

7. Threats to validity controlled by true experiments
History
Maturation
Testing
Instrumentation
Regression
when first observation is extreme, next one is likely to be closer to the mean.
Interactions with selection – one or more groups respond differently to effects of history, maturation, or instrumentation.
Quasi

8. Threats to validity controlled by true experiments
History
Maturation
Testing
Instrumentation
Regression
Selection
if differences between groups exist from the outset of a study
Interactions with selection – one or more groups respond differently to effects of history, maturation, or instrumentation.
Quasi

9. Threats to validity controlled by true experiments
History
Maturation
Testing
Instrumentation
Regression
Selection
Mortality
if exit from a study is not random, groups may end up very different
Interactions with selection – one or more groups respond differently to effects of history, maturation, or instrumentation.
Quasi

10. Threats to validity controlled by true experiments
History
Maturation
Testing
Instrumentation
Regression
Selection
Mortality
Interactions of selection…
with History
with Maturation
with Instrumentation (ceiling effects)
Interactions with selection – one or more groups respond differently to effects of history, maturation, or instrumentation.
Quasi

11. Note difference between these threats:
Maturation
One group; performance better on post-test than on pre-test
Interaction of Maturation & Selection
Two or more groups
Performance difference larger on post-test than on pre-test
Interaction: threat to validity is that the performance difference may occur because the two groups matured at different rates (e.g., selection effect produces difference between the two groups at beginning of study, and different rates of maturation increase this difference).
A true experiment controls for both kinds of threats.
Quasi

12. Threats to validity not controlled by experiments
Contamination
communication of information about the experiment between groups of subjects
Cook & Campbell (1979):
resentment
‘compensatory rivalry’
diffusion of treatment: control subjects use information given to others to change their own behavior.
Contamination could lead to lowered performance in a control group if they ‘lose heart’ because they are not getting a desirable treatment; it may also lead to a spirit of competition.
Quasi

13. Contamination – an example
Craven, Marsh, Debus, & Jayasinghe (2001)
Journal of Educational Psychology
Teachers trained to improve students’ academic self-concept through praise
Internal control
External control
Quasi

14. Contamination – an example
Craven, Marsh, Debus, & Jayasinghe (2001)
Next slide shows T2 (post-test) academic self-concept scores as a function of T1 scores for control children only.
Quasi

15. T2 acad self concept
1.0
0.5
0.0
-0.5
-1.0
No diffusion
Resentful demoralization?
Overzealous cooperation?
Low focus group consistently higher than external control
Diffusion
Low
Medium
High T1 acad self concept
External control
Internal control
Internal high focus
Internal low focus

16. Threats to validity not controlled by experiments
Contamination
Threats to external validity
best way to deal with this is replication
Quasi

17. Threats to validity not controlled by experiments
Contamination
Threats to external validity
Hawthorne effects
changes in a person’s behavior due to being studied rather than the manipulation.
a special kind of reactivity.
name stems from studies of productivity at Western Electric Company, Hawthorne, Illinois,
Quasi

18. Hawthorne effects Demand characteristics
cues communicated by researcher
subject’s under-standing of their role
Quasi

19. Hawthorne effects Role of “research subject”
Is subject behaving the way he thinks a person in that role should behave?
(E.g., hypnotized person)
Subjects in psychological studies are sensitive and accommodating.
Quasi

20. Hawthorne effects Orne (1962)
‘good subjects’ think they are contributing to science by complying with researcher’s demands
Orne tried to find a task that subjects would either refuse to do or do for only a short time. He asked subjects to ad thousands of rows of 2-digit numbers. Five and a half hours after subjects began, the experimenter gave up. Subjects even kept going when instructed to tear each worksheet into a minimum of 32 pieces before going on to the next one.
Quasi

21. Hawthorne effects What to do about Hawthorne effects?
Orne (1962): Use quasi-control subjects as “co-investigators”
They do your task, reflect on demand characteristics of the experiment.
Quasi

22. Obstacles to true experiments in the field
Sometimes, we cannot bring the phenomenon we want to study into the lab, so we have to work in the field.
Can we do experiments in the field?
Quasi

23. Obstacles to true experiments in the field
Can’t get permission from individuals in authority?
Your study may involve some time and effort on their part. But what’s in it for them?
In schools, parents also have to agree.
Quasi

24. Obstacles to true experiments in the field
Can’t get permission from individuals in authority?
Can’t assign subjects to groups randomly?
have to work with intact groups (e.g., classes in a school)
fairness issue: who wants to be in control group?
give treatment to control group afterwards
Quasi

25. Quasi-Experiments Quasi-experiments resemble true experiments…
usually include a manipulation, and provide a comparison.
…but they are not true experiments.
lack high degree of control that is characteristic of true experiments.
Quasi

26. Quasi-Experiments Quasi-Experiments are compromises
They allow the researcher some control when full control is not possible.
Quasi

27. Quasi-Experiments
Because full control is not possible, there may be several “rival hypotheses” competing as accounts of any change in behavior observed.
How do we convince others that our hypothesis is the right one?
Quasi

28. The Logic of Quasi-Experiments
Eliminate any threats you can
Show how each threat to validity on list given above is dealt with in your study.
Argue that others don’t apply.
using evidence or logic
Quasi

29. Two kinds of quasi-experiments
Non-equivalent control group
“non-equivalent” because not randomly assigned
Quasi

30. Two kinds of quasi-experiments
Non-equivalent control group
Interrupted time-series design
a series of observations over time, interrupted by some treatment
Quasi

31. Non-equivalent Control Group design
Control group is “like” the treatment group.
Chosen from same population
Pre- and post-test measures obtained for both groups, so similarity can be assessed.
Quasi

32. Non-equivalent Control Group design
Control group is not equivalent
subjects are not randomly-assigned to control & treatment groups
so best you can do is argue that comparison is appropriate.
Quasi

33. Non-equivalent Control Group design
If the groups are comparable to begin with, this design potentially eliminates threats to internal validity due to:
History
Maturation
Testing
Instrumentation
Regression
Quasi

34. Problems with the NECG design
Threats to validity due to interactions with selection may not be eliminated using the NECG design.
Selection and maturation
Most likely when treatment group is self-selected (as in psychotherapy cases – people who sought help).
When treatment group is self-selected, then comparison group may be from a different population.
Quasi

35. Problems with the NECG design
Selection and maturation
Selection and history
Does one group experience some event that has a positive or negative effect (e.g., teacher of one class leaves)?
Quasi

36. Problems with the NECG design
Selection and maturation
Selection and history
Selection and instrumentation
Does one group show ceiling or floor effects?
Quasi

37. Problems with the NECG design
Selection and maturation
Selection and history
Selection and instrumentation
Regression to the mean
Are one group’s pretest scores more extreme than the other group’s?
Quasi

38. Possible NECG study outcomes
both experimental and control groups show improve-ment from pretest to posttest
appears not to be any effect of the treatment
Pretest
Posttest
Control group
Quasi

39. Possible NECG study outcomes
Looks like a treatment effect, but there may be a threat due to
selection and maturation,
selection and history
Pretest
Posttest
Control group
Selection and maturation may be a threat – control group did not mature.
Selection and history could be if Program group experienced something Comparison group didn’t (or vice-versa).
Selection and regression probably not a threat here.
Quasi

40. Possible NECG study outcomes
Selection and maturation could be a threat
Or interaction of selection and
history
testing
instrumentation
or mortality.
Pretest
Posttest
Control group
Selection and maturation could certainly be a threat – the 2 groups are different to begin with, which may reflect different maturation rates.
Could be interaction of selection and history, testing, instrumentation, or mortality.
Selection and regression probably not a threat here.
Quasi

41. Possible NECG study outcomes
Interaction of selection and regression looks like a serious threat here
Selection and maturation probably not a threat here.
Pretest
Posttest
Quasi

42. Possible NECG study outcomes
Crossover effect
Clearest evidence for an effect of the program of any of these graphs.
Selection and instrumentation not a problem – no ceiling or floor effects
Pretest
Posttest
Quasi

43. Quasi-experiment example
Langer & Rudin (1976)
Research conducted in retirement home.
Residents on one floor given more control over their daily lives
Residents of another floor given same interaction with staff, but no increased control.
Treatment group given control over things such as rising time, bedtime, choice of movie shown in the evening.
Quasi

44. Langer & Rudin (1976) – Measures
Ratings
Self-report of feeling of control from residents
Staff assessments of mental & physical well-being, by ‘blind’ assessors
Objective measures
record of movie attendance
participation in “Guess how many jelly-beans” contest on each floor
Quasi

45. L & R (1976) – limits on control
L & R had no control over
who entered the home
who was assigned to either floor.
no control over staff hiring or firing / resigning.
Only whole floors could be assigned to conditions.
Quasi

46. L & R (1976) – Possible Problems
Interaction of Selection and Maturation
even if groups have similar pretest scores, they may differ on things pretest didn’t measure
probably not a problem here – people on both floors had similar SES
assigned to floors randomly, not by health status.
Quasi

47. L & R (1976) – Possible Problems
Selection and history
suppose a popular (or unpopular) nurse left one of the floors during the study. That might influence well-being.
L & R did not address this issue.
Quasi

48. L & R (1976) – Possible Problems
Selection and history
Selection and instrumentation
did one group show ceiling or floor effects?
L & R say, no.
Quasi

49. L & R (1976) – Possible Problems
Selection and history
Selection and instrumentation
Regression
were one group’s pretest scores more extreme than the others?
L & R say, no.
Quasi

50. L & R (1976) – Possible Problems
Selection and history
Selection and instrumentation
Regression
Observer bias and Contamination
observers in the L & R study were not aware of the hypothesis.
L & R reported there was little communication between floors.
Quasi

51. L & R (1976) – Possible Problems
Selection and history
Selection and instrumentation
Regression
Observer bias and Contamination
Hawthorne Effect
cannot be ruled out, but L & R took care to give both floors same attention.
Message varied between floors, but “face time” was the same.
Quasi

52. L & R (1976) – Possible Problems
Selection and history
Selection and instrumentation
Regression
Observer bias and Contamination
Hawthorne Effect
External Validity
might be an issue.
home involved was rated “one of the finest” in the state
subjects may have been atypical in their desire for control
If residents were relatively wealthier than residents of other homes, thus more independent, the move into the home might have been harder on them
Then treatment (increasing control) might have been more important to them than it would have been to residents of other homes.
Quasi

53. Two kinds of quasi-experiments
Non-equivalent control group
Interrupted time-series design
a series of observations over time, interrupted by some treatment
Quasi

54. Time-Series Designs
In T-S designs, performance is measured both before and after a treatment.
If there is an abrupt change in performance at time of treatment, we conclude that treatment worked.
Major issues – are there history effects? Instrumentation effects?
Quasi

55. Time-series designs example
Campbell (1969)
Effect of speed limit reduction on traffic fatalities in Connecticut
incidence of traffic fatalities in years before and after the speed limit reduction,
conclusion: speed limit change had a modest effect.
Neighboring states would have had roughly the same climate, roads of about the same quality, automobile fleet of about the same age.
Quasi

56. Campbell (1969) Any threat to internal validity?
other explanations for any change in traffic fatality incidence:
Changes in car safety
Weather
Record keeping
Quasi

57. Campbell (1969) Any threat to internal validity?
Such effects should be similar in neighboring states
Campbell found no change in fatality incidence in those states.
Quasi

58. Campbell (1969) Any threat to external validity?
E.g., would treatment have same effect in other states, or are people in Connecticut more law-abiding?
Quasi

59. Campbell (1969)
Time-series design eliminates most other threats to validity – e.g., maturation, testing, regression.
For example, maturation would probably not produce a sudden change in performance of the kind found in Time-Series Designs.
Quasi