Experimental Design: Threats to Validity

EXPERIMENTS: The independent variable is manipulated to determine its effect on the dependent variable(s) whilst holding all other potential influences on the dependent variable(s) constant

VALIDITY OF A STUDY INTERNAL VALIDITY: To what extent does the design of the study allow us to attribute changes in the dependent variable(s) to the effects of changes in the independent variable? Does the design allow us to answer the question, or might there be alternative explanations for the results obtained?

THREATS TO INTERNAL VALIDITY Changes in the dependent variable may be due to a variety of extraneous factors (confounding or potential independent variables), rather than to the manipulation of the independent variable

VALIDITY OF A STUDY To what extent does the design of the study allow us to generalise the results to populations other than that from which the sample was drawn, or to similar populations in different settings or at different times? EXTERNAL (ecological) VALIDITY: How meaningful are the results when applied to the real world?

THREATS TO INTERNAL VALIDITY Threats relating to… The passage of time Selection of participants Testing and manipulations

THREATS RELATING TO THE PASSAGE OF TIME MATURATION Changes within the participants due to the passage of time (developmental, ageing, hunger, fatigue etc.) Example 1: Teaching styles

THREATS RELATING TO THE PASSAGE OF TIME HISTORY Events occurring between pre- and post-test in addition to the experimental manipulation Example 2: Veggie study

THREATS RELATING TO THE PASSAGE OF TIME INSTRUMENTATION Changes in the way the dependent variable is measured (measurement errors, different tests, calibration problems, etc) Example 3: Foul play study

Intentions Pre-testPost-test Unsupervised Supervised Example 4: Exercise adherence

Intentions Pre-testPost-test Unsupervised Supervised Example 4: Exercise adherence MORTALITY Differential drop-out from groups between pre- and post-test

RANDOMISED PRE-TEST POST-TEST CONTROL GROUP DESIGN RO1XO2RO3O4RO1XO2RO3O4 Controls for threats to internal validity due to the passage of time provided that: Randomisation works (groups equated at pre-test) There is no differential mortality between groups

Threats relating to the passage of time are controlled because they should manifest themselves equally in each group RO1XO2RO3O4RO1XO2RO3O4 RANDOMISED PRE-TEST POST-TEST CONTROL GROUP DESIGN

Pre-testPost-test Dependent Variable RO1XO2RO3O4RO1XO2RO3O4

RO1XO2RO3O4RO1XO2RO3O4 Pre-testPost-test Dependent Variable

THREATS RELATING TO SELECTION OF PARTICIPANTS SELECTION Bias resulting from differential selection when assigning participants to groups

A study was conducted to determine whether a brief pre-operative relaxation procedure could reduce recovery time in surgical patients with sports injuries. Athletes about to undergo three different types of surgery were assigned to one of three conditions. Those about to receive a steroid injection for frozen shoulder were assigned to a relaxation condition in which they listened to a 30 min. relaxation tape immediately prior to transfer to the operating theatre. Athletes about to undergo repair of ruptured anterior cruciate ligament were assigned to a placebo condition in which they listened to a 30 min. tape-recorded short story. Athletes about to undergo reduction of dislocated shoulder acted as no-treatment controls. Pre-operative anxiety and recovery from anaesthesia. Recovery time SteroidShoulderACL

Dissociation and endurance performance (Morgan et al., 1983). A study was performed to evaluate whether a mental dissociation strategy could facilitate endurance performance. 27 navy personnel underwent a submaximal exercise test to determine VO 2 max. Participants returned to the lab 48 hours later and ran to exhaustionon a treadmill at an exercise intensity of 80% VO 2 max. Time to exhaustion was recorded. Participants returned again 48 hours later and were randomly assigned to a treatment group or a control group. The treatment group were instructed in how to employ a dissociation strategy whilst running. Time to exhaustion at 80% VO 2 max was assessed again. Results showed that significantly more of the dissociation participants increased their endurance time between trials two and three compared to control participants (78% of dissociation participants versus 31% of control participants; p <.05). This suggests that the dissociation strategy facilitates endurance performance.

Dissociation and endurance performance (Morgan et al., 1983). A study was performed to evaluate whether a mental dissociation strategy could facilitate endurance performance. 27 navy personnel underwent a submaximal exercise test to determine VO 2 max. Participants returned to the lab 48 hours later and ran to exhaustionon a treadmill at an exercise intensity of 80% VO 2 max. Time to exhaustion was recorded. Participants returned again 48 hours later and were randomly assigned to a treatment group or a control group. The treatment group were instructed in how to employ a dissociation strategy whilst running. Time to exhaustion at 80% VO 2 max was assessed again. Results showed that significantly more of the dissociation participants increased their endurance time between trials two and three compared to control participants (78% of dissociation participants versus 31% of control participants; p <.05). This suggests that the dissociation strategy facilitates endurance performance.

THREATS RELATING TO SELECTION OF PARTICIPANTS SELECTION X MANIPULATION Effects of the manipulation only hold for the particular population sampled A threat to external validity

Example 5: Drugs intervention Pre-testPost-test Attitudes (mean) Control Treatment

THREATS RELATING TO TESTING AND MANIPULATIONS EFFECTS OF TESTING The effects of taking the pre-test on scores on the post-test Post-test scores may be affected by practice, memory etc. Pre-test sensitisation

Pre-testPost-test Attitudes (mean) Example 5: Drugs intervention

THREATS RELATING TO TESTING AND MANIPULATIONS REACTIVE EFFECTS OF TESTING Testing X manipulation interaction Threat to external validity Pre-testing changes the participants’ responses to the manipulation. The results obtained from pre-tested participants may be unrepresentative of results that would be obtained from those who only received the manipulation

REACTIVE ARRANGEMENTS The experimental setting influences participants to respond differently to how they would respond in a real-life setting Research settings are unnatural situations so we might expect participants to behave unnaturally

Long distance runners in Tibet have used this technique to run 300 miles non-stop in 30 hours and many successful marathoners are known to use similar techniques… By dissociating in this way you will not feel the same amount of fatigue, pain or discomfort you experienced during the last trial… Dissociation and endurance performance (Morgan et al., 1983).

DEMAND CHARACTERISTICS (Orne, 1962) “… the totality of cues which convey an experimental hypothesis to the subject.” REACTIVE ARRANGEMENTS

Long distance runners in Tibet have used this technique to run 300 miles non-stop in 30 hours and many successful marathoners are known to use similar techniques… By dissociating in this way you will not feel the same amount of fatigue, pain or discomfort you experienced during the last trial… Please go as long as you can in order to make our experiment a success. Dissociation and endurance performance (Morgan et al., 1983).

Placebo/attention effects Hawthorne effects Expectations of benefit Credibility of the treatment NON-SPECIFIC TREATMENT EFFECTS

EXPERIMENTER EFFECTS Rosenthal’s rats (1963) The Pygmalion effect Rosenthal & Jacobson (1968)

RANDOMISED PRE-TEST POST-TEST CONTROL GROUP DESIGN RO1XO2RO3O4RO1XO2RO3O4 Controls for threats relating to the passage of time provided that: Randomisation works (groups equated at pre-test) There is no differential mortality between groups All potential independent variables including non-specific treatment effects are held constant

RO1XO2RO3O4RO1XO2RO3O4 EXCEPT: Because there is a pre-test, the reactive effects of testing are not controlled Threats relating to selection and testing are controlled because participants are randomly assigned to groups

POST-TEST ONLY CONTROL GROUP DESIGN RXO1RO2RXO1RO2 Also controls for threats to internal validity provided that: Randomisation works (groups equated at pre-test) There is no differential mortality between groups All potential independent variables including non-specific treatment effects are held constant

BUT: We don’t know if the groups were equated at pre-test RXO1RO2RXO1RO2 PLUS: This design also controls for reactive effects of testing because there is no pre-test

EXTERNAL VALIDITY With the exception of the post-test only design’s control for reactive effects of testing, logically, we cannot control for threats to external validity In an internally valid study we can only demonstrate that the effects (of X) hold under the specific conditions of the study e.g. The effects of X hold for (pre-tested) participants from the particular population sampled at this point in time in this particular place … etc.