1. Research Problems

2. Hawthorne Effect
Western Electric’s Hawthorne Plant 1939 study of light intensity
productivity went…up
Potential Solutions:
run experiment for a longer period
use a control group

3. John Henry Effect legend of black railway worker
control group overcompensates
Potential Solutions:
don’t do threatening experiments
don’t set up obviously competitive situations
don’t tell control group that they are control group
conduct in another school somewhere else
unfortunately, produces new variable of different school, neighbourhood, etc.!

4. Placebo Effect
introduce placebo in attempt to make conditions in treatment and control groups identical
placebo effects are reactions that (after taking the placebo) cannot be explained by the chemical or medical effects of the placebo.
psychological factors

5. Placebo Effect Potential Solutions
double-blind experiment
secrecy
but then violate principle of informed consent
screen out or balance number of placebo reactors in treatment & control groups

6. the Pygmalion effect Rosenthal and Jacobson, 1968
self-fulfilling prophecy
Potential solution:
do not tell subjects what experiment is about
but what happens to principle of informed consent?

7. Demand Characteristics
rumour
setting
instructions
status and personality of researcher
unintentional cues from experimenter
experimental procedure itself

8. Demand Characteristics Potential Solutions
reduce clarity of demand characteristics
generate alternative demand characteristics
reduce subjects’ motivation to respond to demand characteristics

9. Validity Internal Validity External Validity
control experiment to eliminate extraneous variables
External Validity
outcome can be generalized to other populations in other settings

10. Threats to Internal Validity
History
change producing events in addition to the experimental treatment
Maturation
subjects grow older, learn more, can do more

11. Threats to Internal Validity
Testing
in pre-test/post-test designs — subjects may remember questions from pre-test
Instrumentation
changes in measurement
changes in observer
changes in subjects (testing effects)

12. Threats to Internal Validity
Differential Selection
Differences between treatment and control group
E.g., volunteers vs non-volunteers
Experimental Mortality (or Attrition)
subjects that drop out of experiment may differ from others in important ways

13. Threats to Internal Validity
Testing & Experimental Treatment Interaction
In pre-test/post-test design, pretest may interact with experimental treatment to exaggerate result
Statistical Regression

14. Threats to Internal Validity
Not Paying Attention to What’s Really Going On

15. External Validity 1. Population Validity 2. Ecological validity
can results be generalized from specific sample to the population from which sample was drawn
2. Ecological validity
can results be generalized from contrived conditions created by experimenter to another set of environmental conditions (i.e., real world)

16. Experimental Design

17. Experimental Design
Purpose is to design an experiment that controls for as many extraneous variables as possible
Campbell and Cook, classic 1968 paper categorized experimental designs by how many variables they controlled

18. Weak Designs
Single Group Designs

19. The One-Shot Case Design

20. The One-Shot Case Design
hardly experiment at all
can’t be sure result was result of treatment, not history, maturation, etc.
no control over group selection
could have problem with subject mortality (e.g., transient student population)
since only tested once, can’t even measure gain — maybe students knew it before we started
consequently, results are largely meaningless

21. One Group Pre-test/Post-test Design

22. One Group Pre-test/Post-test Design
Uncontrolled Extraneous Factors (all of them!)
history e.g., Hawthorne effect
maturation
testing
Pre-test may have contributed to higher scores on post-test due to greater familiarity with types of questions or focus on certain topics
instrumentation
pre-test and post-test the same? Observer the same?
selection: could be an atypical group
mortality
interaction of testing & experimental treatment
statistical regression

23. One Group Pre-test/Post-test Design
useful for studying stable dependent variable
justified when
extraneous factors can be estimated with a high degree of certainty or
can be safely assumed to be nonexistent
E.g., not maturation because change is too dramatic for maturation to explain it

24. Control Group Designs

25. Posttest-Only Nonequivalent Groups Design

26. Posttest-Only Nonequivalent Groups Design
also called “static group comparisons”
still a relatively weak design
advantage over single one shot design that one can compare, so cancel out maturation, etc.
Problems:
Differential selection
experimental mortality

27. Non-Equivalent Control Group Design

28. Non-Equivalent Control Group Design
stronger design (weakest of the strong designs)
common design in education because usually can’t randomize assignment of students to classes
pre-test measures whether initial groups are similar on tested variable
could also match subjects based on pre-test
but may miss other factors which could impact results (e.g., better teacher in one group)

29. Non-Equivalent Control Group Design
Factors controlled by inclusion of a control group
history
maturation
testing
instrumentation (assuming same for both groups)
statistical regression
Factors still NOT controlled
differential selection
experimental mortality (if any)
testing and experimental treatment interaction

30. Pre-test/Post-test Control Group Design

31. Pre-test/Post-test Control Group Design
still two uncontrolled factors
Intersession History
events that are specific to one group and not the other which occur during the group sessions
To control for interssion history, need to balance (control) such factors as
experimenters
time of day
day of week
by randomly assigning to groups of subjects

32. Pre-test/Post-test Control Group Design
Interaction of testing and treatment
Cannot be avoided when there is a pretest
eliminating the pre-test would therefore improve design

33. The Post-test Only Control Group Design

34. The Post-test Only Control Group Design
do not need pre-test with random selection
disadvantages
random assignment may not be fully successful in eliminating initial differences between control and experimental groups
cannot form subgroups (i.e., high, medium, & low) to determine whether the experimental treatment has a different effect on subjects at different levels of the variable as measured by the pretest (because no pretest)

35. Solomon Four-Group Design

36. Solomon Four-Group Design
basically, a pretest and non-pretest experiment at the same time
controls for everything, except differential mortality

37. Time Series Design

38. Time Series Design
Similar to one-group pre-test /post-test design (weak #2) but additional pre- and post- measurements add power
additional measurements enable researcher to rule out maturation
testing effects as sources of influence shift from pretest to posttest