Experimental Research Attempts to Establish Cause and Effect Relationships

Cause and Affect Cannot exist unless two variables are correlated i.e. evaluate if two variables are correlated before trying to manipulate one to change the other.

The Logic: Cause and Effect can only be established if there is no other reasonable explanation for the changes in the dependent variable except the manipulation of the independent variable

Important Factors: Good theoretical framework Appropriate experimental design Correct statistical model & Design Proper selection & Control of Independent variable Appropriate selection & Measurement of dependent variable Use of appropriate subjects Correct interpretation of results

Requirement without which study is uninterpretable. Internal Validity: The extent to which the results of the study can be attributed to the treatments used in the study. Requirement without which study is uninterpretable.

Generalizability of the study results. External Validity Generalizability of the study results.

Strong internal validity may be at odds to strong external validity In real-world settings everything is not controlled and may not operate in the same way as the laboratory.

The Dilemma!!! Is it more important to ascertain that the manipulation of the independent variable caused the observed changes in the dependent variable... Or, is it more important to generalize the results to other populations?

Threats to Internal Validity: Campbell & Stanley (1963)

History: Events occurring during the experiment that are not a part of the treatment. i.e. Evaluating effects of semester-long fitness program on 8th graders. 60% of students also participated in recreational soccer will affect results.

Maturation: Processes within the subjects that operate as a result of time passing. Occurs most frequently when one group is tested several times over a long period of time. i.e. Fatigue, again, hunger. Fitness testing in fall and spring--students do better in spring.

Testing: The effects of one test on subsequent administrations of the same test. i.e. Giving test knowledge about Performance enhancing drugs to students today and re-testing two days later.

Instrumentation: Changes in the instrument calibration, including lack of agreement within and between observers. Same observer may systematically vary rating across time or subjects (observer drift) or different observers may not rate the same performance the same way. (i.e. Skin Fold Study)

Statistical Regression: The fact that Groups selected on the basis of extreme scores are not as extreme on subsequent testing. May occur when groups are not randomly formed but selected on some extreme score on a measure. (i.e. observing active and inactive behaviors. In other settings may regress toward the mean.)

Selection Bias: Identification of comparison groups in other than a random manner. Occurs when groups are not randomly assigned. Did the study use volunteers? Are subjects extremely non-representative of the population?

Experimental Mortality: Loss of subjects from comparison groups due to nonrandom reasons. Loss of subjects from the treatment groups.

Selection-Maturation Interaction: Specific to non-equivalent group designs where the passage of time might affect one group but not the other. i.e. Difference between expert and novice tennis payers. Experts might have higher socio-economic status, access to country clubs, and better instruction.

Additional Threat: Expectancy: Researchers anticipating certain subjects will perform better. i.e. Rating subjects labeled as “skilled” better than those labeled as “unskilled” regardless of the treatment.

Threats to External Validity: Campbell & Stanley (1963)

1) Reactive or Interactive Effects of Testing: Fact that the pretest may make the subject more aware of or sensitive to the upcoming treatment. Results in treatment’s not being as effective without the pretest.

Interaction of Selection Biases and the Experimental Treatment: When a group is selected on some characteristic, the treatment may work only on groups possessing that characteristic.

Reactive Effects of Experimental Arrangements: Treatments that are effective in very constrained situations (labs) may not be effective in less constrained (real-world) settings.

Multiple-Treatment Interference: When subjects receive more than one treatment, the effects of previous treatments may influence subsequent ones.

Controlling Threats:

Internal Validity: Randomization: Controls For: 1) History (up until experiment) 2) Maturation 3) Statistical Regression 4) selection Biases 5) Selection Maturation Interaction

Test Reliability: Instrumental cannot be controlled or evaluated by any design; only experimenter can control. Psychometrics: Establishing appropriate instrumentation.

Subject Retention: Cannot be controlled by experimental design Subject Retention: Cannot be controlled by experimental design. Careful discussions about importance of study with subjects.

Placebos, Blind, Double-Blind Setups: Placebo: is treatment effect real or due to psychological effects? Blind Study: Subject does not know whether receiving treatment or control. Double-Blind: Neither subject or Researcher knows who is receiving treatment or control.

External Validity: Random selection is the key. Random selection of subjects, treatments, and experimental situations.