1. Making your experiment more valid and more efficient
Experimental Design
Making your experiment more valid and more efficient

2. So, the basic experimental procedure in social science is:
Gathering of a subject pool and then random assignment to conditions
Note: the sample is not random, but once you have a subject pool you flip a coin, etc. to determine which subjects receive which condition
Depending upon the experimental design, a measure of the dependent variable may be taken from the groups prior to the manipulation

3. The data are then analyzed to test the research hypothesis
Subjects are exposed to the ‘treatment’ assigned to their group
May be some level of the factor or a zero-level (control) treatment
For example, the ‘’experimental group” may play a video game with lots of violence while the control group plays a non-violent game
Both groups are then measured on the dependent variable of interest
In this case, some measure of aggressiveness
The data are then analyzed to test the research hypothesis

4. Basic design (After-only, control group design)X O R O
R=randomization, X=manipulation, O=observation

5. What’s the effect?
The difference between group scores on the dependent variable is assumed to be due to the manipulation
But what if the groups were different at the start?
Randomization should have taken care of that
However, pure chance can sometimes lead to differences at the outset
So, measure them before the manipulation as well as after

6. Before-after, control group designX O1 O2 R O2 O1
R=randomization, X=manipulation, O=observation

7. Before-after, control group design
Now there’s a new problem:
Measuring the subjects and then measuring them again later may cause them to respond differently to the second measure than they otherwise would have (true for both experimental and control subjects)
Measuring the experimental subjects before exposing them to the treatment may change their response to the manipulation

8. Problems with the basic design
The difference between control and experimental conditions includes more than violence
Experimenters may have control group play a nonviolent video game
Many other variables may affect the outcome
For example, gender
Measure and include in statistical analyses
Use factorial design

9. So: There will always be some problems with any procedure
The goal is to minimize the threats to validity as best your circumstances (including budget) allow
Especially internal validity

10. How do we do that? Experimental design Statistical analyses
Careful application of procedures

11. Experimental design Two facets
Choices in identifying the independent and dependent variables to be measured and/or manipulated
Choices in developing the procedures for manipulation, measurement, environment of testing, etc.

12. Choosing the variables
Theory determines variables manipulated and measured
Experimental designs where the research manipulates more than one independent variable are known as factorial designs
Mediating concepts often are included

13. Choosing the measures
The measures are chosen to best represent the concepts in the original theory
Measurement validity
Must work in an experimental setting
Lengthy questionnaires unlikely to work
Physical measures more easily applied in a laboratory setting

14. Factorial designs Factorial designs are useful because they:
Are efficient
With two independent variables we can test three hypotheses instead of just one
Don’t need three times as many subjects
However, there is a practical limit to independent variables
As the number rises, the required number of subjects increases quickly (5 per cell)
Help us avoid coming to invalid conclusions
They allow us to control for potentially confounding variables and interactions
We can test the impact of third variables
Factorial designs can be more difficult to administer

15. An example We may want to test the following hypothesis:
“Exposure to Dancing with the Stars will lead to greater liking for the contestants”
We might, however, expect that men and women will be affected differently by the show
So, we might add gender as another independent variable in the design

16. See Dancing with the Stars Don’t see Dancing with the Stars
Factorial design
Women
See Dancing with the Stars
Don’t see Dancing with the Stars
Men
All groups are measured on liking for contestants

17. What we can test “Main effect” of exposure to Dancing with the Stars
“Main effect” of gender
“Interaction effect” of exposure by gender
Remember: the dependent variable is liking for contestants

18. Results (% who like contestants)

19. Conclusions Main effect of gender Main effect of exposure to DWTS
Yes: Gender affects liking for celebs
Main effect of exposure to DWTS
Yes: Exposure increases liking
Interaction effect of exposure by gender:
Yes: Exposure has a clear positive effect for women, little or even negative effect for men

20. Procedural choices How many times will you measure the subjects?
You can account for chance fluctuation by increasing the number of times the groups are measured
You increase the likelihood of testing reactivity, though

21. R O1 O2 X O3 O4 R O1 O2 O3 O4

22. Additional design concerns
How many times/how long will the manipulation be presented?
Testing a single exposure to DWTS may not be valid
Viewers may watch many shows during a season

23. R O1 X1 O2 X2 O3 R O1 O2 O3

24. Removal of the treatmentX O2 -X O3 R O1 O2 O3

25. How many levels of the independent variable will be presented?
May want to test high v. medium v. low levels of violence, etc.

26. R O
X(high) R O
X(medium) R O
X(low) R O

27. Factorial nomenclature
The number of treatments on each factor are crossed
2X3 factorial design has 2 treatment levels on one factor and 3 treatment levels on another factor
If we tested the effects of soft, medium and loud music and gender upon ease in a social situation, we would have a 2X3 factorial study
Factor 1: music volume 3 levels: soft, medium and loud
Factor 2: gender 2 levels: male, female
Dependent variable: ease in a social situation

28. Additional design concerns
Artificiality of the testing environment
Labs allow for the greatest control over third variables
Volume, distance from screen, prevent other people from walking by, etc.
Lab experience can sometimes be made less artificial
Some commercial-testing services provide the subjects with a simulated living room for exposure
Loss of control over third variables weighed against increase in generalizability (external validity)

29. Statistical analyses
If you measure potential confounds you can use statistical procedures to remove their impact on the dependent variable

30. Careful application of the design
Unplanned variations among treatments can easily occur when conducting experiments in the real world
The person you wanted to greet everyone and provide instructions may have to leave for an emergency
Subjects may talk to each other outside the lab before they have all completed their part of the experiment
And so on

31. Remember:
Experiments are the most effective quantitative research method for testing hypothesized causal relationships
Experiments emphasize internal validity but are often weak with regard to external validity
Many different experimental designs exist, each having a set of advantages and disadvantages
Your goal is to provide the most valid test of theory your circumstances allow you