1. Experiments II: Validity and Design Considerations
Lecture 6
Experiments II: Validity and Design Considerations

2. Example: Trust-Building Study
Question: Do increased risk-taking behaviors over time increase interpersonal trust?

3. Trust-Building Study Independent Variable
Experiment Condition (2 conditions):
Fixed partner on every trial, cannot control amount to entrust to partner
Fixed partner on every trial, can control amount to entrust to partner
Assignment
Random assignment of participants to one of the 2 conditions.
Same experiment conducted in Japan and US, and comparisons made between the two studies.
Dependent Variable
Cooperation rate (i.e., whether they returned the coins to the partner or not)

4. Measurement and Design Validity
Measurement Concerns
Construct Validity
Design Concerns
Internal Validity
External Validity
Ecological Validity
Construct validity: measuring our intended concept?
Internal validity:
External:
Ecological:

5. Construct Validity
How do we know that our independent variable is reflecting the intended causal construct and nothing else?
“Face” validity deals with subjective judgement of appropriate operationalization
“Content” validity is a more direct check against relevant content domain for the given construct.
Examples from operationalization exercise?

6. Internal Validity
Internal Validity deals with questions about whether changes in the dependent variable were caused by the treatment.

7. ? ?
Cause
Effect ? ?

8. Threats to Internal Validity
History
additional I.V. that occurs between pre-test and post-test
Maturation
Subjects simply get older and change during experiment
Testing
Subjects “get used” to being tested
Regression to the Mean
Issue with studies of extremes on some variable

9. Contamination and Internal Validity
Demand Characteristics
Anything in the experiment that could guide subjects to expected outcome
Experimenter Expectancy
Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy)

10. General Demand Characteristics
Evaluation Apprehension
Solutions
Double-blind experiments
Experiments in natural setting (i.e., subjects do not know they are in an experiment)
Cover stories
Hidden measurements
Subjects know that they are being evaluated and this changes their behavior

11. Reducing the role of the experimenter: solving expectancy effects
Naïve experimenter
Those conducting study are not aware of theory or hypotheses in the experiment
Blind
Researcher is unaware of the experiment condition that he/she is administering
Standardization
Experimenter follows a script, and only the script
“Canned” Experimenter
Audio/Video/Print material gives instructions

12. And More! Selection Bias Mortality Diffusion, Sharing of Treatments
Issue with non-random selection of subjects
Mortality
Departure of subjects in the experiment
Diffusion, Sharing of Treatments
Control group unexpectedly obtains treatment
Other ‘social’ threats?
Compensatory rivalry, resentful demoralization, etc.

13. Three threats to external validity (generalizability) in experiments
External Validity– How far does the given experiment generalize to similar groups, individuals, etc?
Setting
Population
History
Setting:Physical and social context of the experiment
Population:Is there something specific about the sample that interacts with the treatment?
Histor: Is there something about the time that interacts with the treatment?

14. Ecological Validity
Approximation of ‘real-life’ situations

15. The Validity Tradeoff: Truth and Myth
Internal
Validity
External & Ecological
Validity
Balance is important between the types of validity, but internal validity is usually (if not always) the more important factor.

16. Pro’s and Con’s of Experiments
Gives researcher tight control over independent factors
Allows researcher to test key relationships with as few confounding factors as possible
Allows for direct causal testing
Con’s
Usually a smaller N than surveys
Sometimes give up large amounts of external validity in favor of construct validity and direct causal analysis
Require a large amount of planning, training, and time– sometimes to test relationship between only 2 factors!

17. Additional considerations before using experiments
Cost and Effort
Is the effort worth it to test the concepts you are interested in?
Manipulation and Control
Will you actually be able to manipulate the key concept(s)?
Importance of Generalizability
Are you testing theory, or trying to establish a real-world test?