1. Experimental Design

2. Experimental Design and the struggle to control threats to validity

3. LOW
NATURALISTIC
CASE-STUDY
INCREASINGLY
CONSTRAINED
CORRELATIONAL
DIFFERENTIAL
EXPERIMENTAL
HIGH

4. Experimental design is a planned interference in the natural order of events by the researcher.

5. What is Experimental Design?
Experimental research allows us to test hypotheses and infer causality under controlled conditions designed to maximize internal validity. The high control and internal validity often mean a reduction of external validity. That is, the more precise, constrained, and artificial we become in the experiment, the less natural are the procedures and findings. The result is that we sometimes have difficulty generalizing experimentation to the natural environment, or a larger population.

6. Experimental Design
Comparisons are made under different and controlled conditions.
Subjects are assigned to each type of condition in an unbiased manner, usually matched or random.
Although causality can often be inferred, results may not be applicable outside of the experimental setting
Causality: If X, then Y
AND
If not X, then not Y

7. Experimental TIME 0: PRE-TEST. Collect baseline data. CONTROL
TREATMENT #1
TREATMENT #2
TREATMENT #3
Experimental

8. TIME 1: TREATMENT GIVEN
CONTROL
JOLT
COKE
COFFEE
Experimental

9. TIME 3: POST-TEST. Collect data on the effects of the treatment and compare to pretest, and to each treatment.
CONTROL
JOLT
COKE
COFFEE
Experimental

10. Hypotheses
NULL (Statistical) mean JOLT= mean COKE = mean COFFEE = mean CONTROL
RESEARCH (Causal) Caffeine causes people to grow tall and nervous.
CONFOUNDING (Rival) The differences are due to the amount of citric acid in the drinks.

11. A simple 2-group, posttest-only design
Outfitters given low- impact training
Outfitters given NO low-impact training
Measure impacts caused by their clients
Compare Scores

12. Hypotheses NULL (Statistical) mean Trained= mean Un-trained
RESEARCH (Causal) The clients of outfitters trained in low impact methods will cause fewer impacts than clients of outfitters who did not receive such training.
CONFOUNDING (Rival) Prior knowledge may have caused the observed differences in response.

13. Threats to Validity

14. Reliability
Repeated use of the measure with identical subjects yields identical and consistent results. It is improved by:
Clear conceptualization
Precise measurement
Multiple indicators
Pilot-testing

15. Validity
Internal Validity– design and measurement concerns that reduces chances for internal errors.
External Validity– describes our ability and intent to generalize to subjects beyond our study sample. Largely an issue of design and sampling.

16. Validity and Reliability
Reliable, NOT valid
Valid, NOT Reliable
Valid AND Reliable
NOT Valid, NOT Reliable
NOT Valid but Reliable
Valid but UNReliable
NOT Valid and UNReliable
Valid and Reliable

17. Internal Validity Specifically, measurement validity
Measures are valid for a single purpose
Three types of validity:
Face—as judged by others or by logic
Content—captures the entire meaning of the experience
Criterion—agrees with a validates, reliable external source:
Concurrent, agrees with a preexisting measure
Predictive, agrees with a future behavior or outcome

18. Validity
Because of the confounding Hypothesis we are not 100% sure that our conclusions are valid.
Did we indeed measure the effects of new knowledge?
Independent -- Dependent?
Other Variable/s -- Dependent?

19. Road map for research success
Anticipate all threats to validity.
Take plans to eliminate them.
Statistical validity
Construct validity
External validity
Internal validity.

20. Statistical validity
The variations in the dependent variable are not due to variation in the subjects, but due to variations in the measuring instrument. The instrument is unreliable.
Some statistical assumptions have been violated (e.g., non-normal data treated with parametric statistics; means of ordinal data, etc.).

21. Construct validity
How well the observed data support the theory, and not a rival theory.

22. External validity
The degree to which the findings of the study are valid for subjects outside the present study. The degree to which they are generalizable.
Unbiased, complex sampling procedures; many studies, mid-constraint approaches help strengthen external validity.
< external = > internal

23. Internal Validity Threats to causality (our ability to infer).
Did the independent variable cause the dependent to change (were they related), or did some confounding variable intervene?
Maturation, history, testing, instrumentation, regression to the mean, selection, attrition, diffusion and sequencing effects.

24. Maturation
If the time between pre- and posttest is great enough to allow the subjects to mature, they will!
Subjects may change over the course of the study or between repeated measures of the dependent variable due to the passage of time per se. Some of these changes are permanent (e.g., biological growth), while others are temporary (e.g., fatigue).

25. History
Outside events may influence subjects in the course of the experiment or between repeated measures of the dependent variable.
Eg., a dependent variable is measured twice for a group of subjects, once at Time A and again at Time B, and that the independent variable (treatment) is introduced between the two measurements.
Suppose also that Event X occurs between Time A and Time B. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to Event X.

26. Testing
Subjects gain proficiency through repeated exposure to one kind of testing. Scores will naturally increase with repeated testing.
If you take the same test (identical or not) 2 times in a row, over a short period of time, you increase your chances of improving your score.

27. Instrumentation
Changes in the dependent variable are not due to changes in the independent variable, but to changes in the instrument (human or otherwise).
Measurement instruments and protocols must remain constant and be calibrated.
Human observers become better, mechanical instruments become worse!

28. Regression to the mean
If you select people based on extreme scores (High or low), in subsequent testing they will have scores closer to the mean (they would have regressed to the center).

29. Selection
When random assignment or selection is not possible the two groups are not equivalent in terms of the independent variable/s.
For example, males=treatment; females=control.
Highest threats in naturalistic, case study and differential approaches.

30. Attrition When subjects are lost from the study.
If random it may be OK.
Confounding attrition is when the loss is in one group or because of the effects of the independent variable. (Jolt killed off 2 people!)

31. Diffusion of treatment
When subjects communicate with each other (within and between groups) about the treatment) they diffuse the effects of the independent variable.

32. Sequencing effects
The effects caused by the order in which you apply the treatment.
A B C
A C B
B A C, etc.

33. Subject effects
Subjects “know” what is expected of them, and behave accordingly (second guessing).
Social desirability effect.
Placebo effect. A placebo is a dummy independent effect. Some people react to it.

34. Experimenter effects Forcing the study to produce the desired outcome.
Expectations of an outcome by persons running an experiment may significantly influence that outcome.

35. Single- and double-blind procedures
Single blind –subjects don’t know which is treatment, which is not.
Double blind—experimenter is also blind.

36. Designs One shot: One Group Pre-Post: Static Group: G1 T------O2
G1 O T------O2
Static Group:
G1 T------O2
G2 (C) O2
T=Treatment
O=Observation (measurement)
C=Control

37. More designs Random Group: Pretest-Posttest, Randomized Group:
RG1 T O
RG2 (C) O
Pretest-Posttest, Randomized Group:
RG1 O T------O2
RG2 (C) O O2
R=Random

38. Yet another design: Solomon four-group: RG1 O1------T------O2
RG2 O O2
RG3 T------O2
RG4 O2

39. One last one ! Latin Square: RG1 O T1 O T2 O T3 O RG2 O T2 O T3 O T1 O