Experiments II: Validity and Design Considerations Lecture 6 Experiments II: Validity and Design Considerations

Example: Trust-Building Study Question: Do increased risk-taking behaviors over time increase interpersonal trust?

Trust-Building Study Independent Variable Experiment Condition (2 conditions): Fixed partner on every trial, cannot control amount to entrust to partner Fixed partner on every trial, can control amount to entrust to partner Assignment Random assignment of participants to one of the 2 conditions. Same experiment conducted in Japan and US, and comparisons made between the two studies. Dependent Variable Cooperation rate (i.e., whether they returned the coins to the partner or not)

Measurement and Design Validity Measurement Concerns Construct Validity Design Concerns Internal Validity External Validity Ecological Validity Construct validity: measuring our intended concept? Internal validity: External: Ecological:

Construct Validity How do we know that our independent variable is reflecting the intended causal construct and nothing else? “Face” validity deals with subjective judgement of appropriate operationalization “Content” validity is a more direct check against relevant content domain for the given construct. Examples from operationalization exercise?

Internal Validity Internal Validity deals with questions about whether changes in the dependent variable were caused by the treatment.

? ? Cause Effect ? ?

Threats to Internal Validity History additional I.V. that occurs between pre-test and post-test Maturation Subjects simply get older and change during experiment Testing Subjects “get used” to being tested Regression to the Mean Issue with studies of extremes on some variable

Contamination and Internal Validity Demand Characteristics Anything in the experiment that could guide subjects to expected outcome Experimenter Expectancy Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy)

General Demand Characteristics Evaluation Apprehension Solutions Double-blind experiments Experiments in natural setting (i.e., subjects do not know they are in an experiment) Cover stories Hidden measurements Subjects know that they are being evaluated and this changes their behavior

Reducing the role of the experimenter: solving expectancy effects Naïve experimenter Those conducting study are not aware of theory or hypotheses in the experiment Blind Researcher is unaware of the experiment condition that he/she is administering Standardization Experimenter follows a script, and only the script “Canned” Experimenter Audio/Video/Print material gives instructions

And More! Selection Bias Mortality Diffusion, Sharing of Treatments Issue with non-random selection of subjects Mortality Departure of subjects in the experiment Diffusion, Sharing of Treatments Control group unexpectedly obtains treatment Other ‘social’ threats? Compensatory rivalry, resentful demoralization, etc.

Three threats to external validity (generalizability) in experiments External Validity– How far does the given experiment generalize to similar groups, individuals, etc? Setting Population History Setting:Physical and social context of the experiment Population:Is there something specific about the sample that interacts with the treatment? Histor: Is there something about the time that interacts with the treatment?

Ecological Validity Approximation of ‘real-life’ situations

The Validity Tradeoff: Truth and Myth Internal Validity External & Ecological Validity Balance is important between the types of validity, but internal validity is usually (if not always) the more important factor.

Pro’s and Con’s of Experiments Gives researcher tight control over independent factors Allows researcher to test key relationships with as few confounding factors as possible Allows for direct causal testing Con’s Usually a smaller N than surveys Sometimes give up large amounts of external validity in favor of construct validity and direct causal analysis Require a large amount of planning, training, and time– sometimes to test relationship between only 2 factors!

Additional considerations before using experiments Cost and Effort Is the effort worth it to test the concepts you are interested in? Manipulation and Control Will you actually be able to manipulate the key concept(s)? Importance of Generalizability Are you testing theory, or trying to establish a real-world test?