1. Experimental Design

2. Experimental Design
Experimental design is the part of statistics that happens before you carry out an experiment
Proper planning can save many headaches
You should design your experiments with a particular statistical test in mind

3. Why do experiments? Contrast: observational study vs. experiments
Example:
Observational studies show a positive association between ice cream sales and levels of violent crime
What does this mean?

4. Why do experiments? Contrast: observational study vs. experiments
Example:
Observational studies show a positive association between ice cream sales and levels of violent crime
What does this mean?

5. Alternative explanation
Ice cream
Violent
crime
Hot weather

6. Alternative explanation
Ice cream
Correlation is not causation
Violent
crime
Hot weather

7. Why do experiments?
Observational studies are prone to confounding variables: Variables that mask or distort the association between measured variables in a study
Example: hot weather
In an experiment, you can use random assignments of treatments to individuals to avoid confounding variables

8. Goals of Experimental Design
Avoid experimental artifacts
Eliminate bias
Use a simultaneous control group
Randomization
Blinding
Reduce sampling error
Replication
Balance
Blocking

9. Goals of Experimental Design
Avoid experimental artifacts
Eliminate bias
Use a simultaneous control group
Randomization
Blinding
Reduce sampling error
Replication
Balance
Blocking

10. Experimental Artifacts
Experimental artifacts: a bias in a measurement produced by unintended consequences of experimental procedures
Conduct your experiments under as natural of conditions as possible to avoid artifacts

11. Goals of Experimental Design
Avoid experimental artifacts
Eliminate bias
Use a simultaneous control group
Randomization
Blinding
Reduce sampling error
Replication
Balance
Blocking

12. Control Group
A control group is a group of subjects left untreated for the treatment of interest but otherwise experiencing the same conditions as the treated subjects
Example: one group of patients is given an inert placebo

13. The Placebo Effect
Patients treated with placebos, including sugar pills, often report improvement
Example: up to 40% of patients with chronic back pain report improvement when treated with a placebo
Even “sham surgeries” can have a positive effect
This is why you need a control group!

14. Randomization
Randomization is the random assignment of treatments to units in an experimental study
Breaks the association between potential confounding variables and the explanatory variables

15. Experimental units
Confounding variable

16. Experimental units
Treatments
Confounding variable

17. Experimental units
Treatments
Without randomization, the confounding variable differs among treatments
Confounding variable

18. Experimental units
Treatments
Confounding variable

19. Experimental units
Treatments
With randomization, the confounding variable does not differ among treatments
Confounding variable

20. Blinding
Blinding is the concealment of information from the participants and/or researchers about which subjects are receiving which treatments
Single blind: subjects are unaware of treatments
Double blind: subjects and researchers are unaware of treatments

21. Goals of Experimental Design
Avoid experimental artifacts
Eliminate bias
Use a simultaneous control group
Randomization
Blinding
Reduce sampling error
Replication
Balance
Blocking

22. Replication
Experimental unit: the individual unit to which treatments are assigned
Experiment 1
Experiment 2
Tank 1
Tank 2
Experiment 3
All separate tanks

23. Replication
Experimental unit: the individual unit to which treatments are assigned
2 Experimental
Units
Experiment 1
2 Experimental
Units
Experiment 2
Tank 1
Tank 2
8 Experimental
Units
Experiment 3
All separate tanks

24. Replication Pseudoreplication
Experimental unit: the individual unit to which treatments are assigned
2 Experimental
Units
Experiment 1
2 Experimental
Units
Pseudoreplication
Experiment 2
Tank 1
Tank 2
8 Experimental
Units
Experiment 3
All separate tanks

25. Why is pseudoreplication bad?
Experiment 2
Tank 1
Tank 2
problem with confounding and replication!
Imagine that something strange happened, by chance, to tank 2 but not to tank 1
Example: light burns out
All four lizards in tank 2 would be smaller
You might then think that the difference was due to the treatment, but it’s actually just random chance

26. Why is replication good?
Consider the formula for standard error of the mean:
Larger n
Smaller SE

27. Balance
In a balanced experimental design, all treatments have equal sample size
Better than
Balanced
Unbalanced

28. Balance
In a balanced experimental design, all treatments have equal sample size
This maximizes power
Also makes tests more robust to violating assumptions

29. Blocking
Blocking is the grouping of experimental units that have similar properties
Within each block, treatments are randomly assigned to experimental treatments
Randomized block design

30. Randomized Block Design

31. Randomized Block Design
Example: tanks in a field

33. Very sunny
Not So Sunny

34. Block 1
Block 2
Block 3
Block 4

35. What good is blocking?
Blocking allows you to remove extraneous variation from the data
Like replicating the whole experiment multiple times, once in each block
Paired design is an example of blocking

36. Experiments with 2 Factors
Factorial design – investigates all treatment combinations of two or more variables
Factorial design allows us to test for interactions between treatment variables

37. Factorial Design pH
5.5
6.5
7.5 25
n=2 30 35 40
Temperature

38. Interaction Effects
An interaction between two (or more) explanatory variables means that the effect of one variable depends upon the state of the other variable

39. Interpretations of 2-way ANOVA Terms
Effect of pH and Temperature,
No interaction

40. Interpretations of 2-way ANOVA Terms
Effect of pH and Temperature,
with interaction

41. Goals of Experimental Design
Avoid experimental artifacts
Eliminate bias
Use a simultaneous control group
Randomization
Blinding
Reduce sampling error
Replication
Balance
Blocking

42. What if you can’t do experiments?
Sometimes you can’t do experiments
One strategy:
Matching
Every individual in the treatment group is matched to a control individual having the same or closely similar values for known confounding variables

43. What if you can’t do experiments?
Example: Do species on islands change their body size compared to species in mainland habitats?
For each island species, identify a closely related species living on a nearby mainland area

44. Power Analysis
Before carrying out an experiment you must choose a sample size
Too small: no chance to detect treatment effect
Too large: too expensive
We can use power analysis to choose our sample size