1. Research design I: Experimental design and quasi-experimental research

2. Today’s Agenda
Types of studies
Meta-analysis
Experimental design

3. Meta-analysis Combining the results from multiple studies.
Goal: identify patterns among studies, find disagreements, find other relationships that only appear when examining multiple studies.

4. Meta-analysis and the file drawer problem
Many studies are conducted but never reported.
Strong bias towards publishing significant findings.
Bias against publishing negative findings.
Study in 1979 of 293 registered clinical trials found that significant findings were 12 times more likely to be published.

5. Experimental and quasi-experimental designs
Share many features
We’ll start with the common features and end with the differences.
Spoiler alert: major difference is that quasi-experimental designs do not use random selection.
…while we go through this, let’s discuss issues of random vs. non-random assignment

6. Variance Variance has both conceptual and statistical meanings
We’re only talking about the conceptual piece here

7. Two types of variance we care about
Systematic variance
Variance
Variability due to the treatment/grouping/condition/independent variable.
Variability that is predicted by the IV
(could be condition, etc.)
Also called between groups variance.
Error
variance
Variability between groups/participants not accounted for by systematic differences between those groups/people.

8. Where does error variance come from?
Individual differences – pre-existing differences between people; this is the most common source of error variance
Transient states – at the time of the experiment, participants differ in how they feel (mood, health, fatigue, interest, etc.)

9. Sources of Error Variance
Environmental factors – differences in the conditions under which the study is conducted (noise, time of day, temperature, etc.)
Differential treatment –treating different participants in slightly different ways
Measurement error – unreliable measures contribute to error variance

10. Variances are used to calculate ratios
Systematic
Signal
Error
Noise
More signal is better
If the independent variable has an effect on behavior, we should see systematic differences between the scores in the various experimental conditions.

11. How can we reduce error variance?
Error variance = unknowns
The more you know, the easier it is to reduce variance
Control variables

12. Let me complicate things a bit…
Systematic variance
Systematic variance
Primary variance
(Treatment variance)
Variability due to the treatment/grouping/condition/independent variable.
Secondary variance
(Confound variance)
Variance due to a confound that can be corrected (i.e.., through control variables)
Error
variance
We want to reduce confounds!!

13. Why are confounds bad? Confounds are a threat to internal validity.
Internal validity is the degree to which a researcher draws accurate conclusions about the effects of the independent variable on a dependent variable.

14. Threats to Internal Validity
Biased assignment of participants to conditions
Effects are due to initially nonequivalent groups rather than to the independent variable; when random assignment fails.
2. Differential attrition
Participants drop out of experimental conditions at different rates, making the experimental groups no longer equivalent.
3. Pretest sensitization/ order effects
Completing a pretest leads participants to react differently than they would had they not been pretested.

15. How do you avoid threats to internal validity?
Design a good study…

16. Simple Random Assignment
Simple random assignment – participants are placed in experimental conditions in such a way that every participant has an equal probability of being in any condition.
Random assignment is used to make the conditions roughly equivalent at the start of the study.

17. Match Random Assignment
Matched random assignment – participants are matched into homogenous blocks, and then participants within each block are assigned randomly to conditions
Helps to ensure that the conditions will be similar along some specific dimension, such as age or intelligence.
Important to decide which characteristics people should be matched on.

18. Repeated Measures Design
Repeated measures design – an experimental design in which each participant serves in all conditions of the experiment; also called a within-subjects design
Repeated measure designs eliminate the need for random assignment because every participant is tested at every level of the independent variable.
Designs in which each participant serves in only one experimental condition are called between-subjects designs.

19. Order Effects
When the effects of an experimental condition are contaminated by its order in the sequence of experimental conditions in which participants are tested.
Practice effects – participants’ responses are affected by completing the dependent variable many times
Fatigue effects – participants become tired or bored as the experiment progresses
Sensitization – participants gradually become suspicious of the hypothesis as the experiment progresses

20. Counterbalancing
To protect against order effects, use counterbalancing.
Counterbalancing involves presenting the levels of the independent variables in different orders to different participants.

21. See you on Thursday! Next class: More research design
Next Tuesday: Developing proposals