Experiments

 Labs (update and questions)  STATA Introduction  Intro to Experiments and Experimental Design 2

 Experiment is a study in which a treatment is introduced  The type of experiment (such as a true, randomized experiment) depends on several factors  Non-Experiments include responses from natural groups (e.g., most surveys) 3

Tuskegee Study Milgrim Obedience Study Stanford Prison Experiment 4

 (1) Independent Variables must be manipulated (usually by experimenter, sometimes by context)  (2) Participants must be assigned randomly to various conditions or groups 5 Courtesy

Active independent variable(s):  The I.V. is “given” to the participants, usually for some specified time period. It is often manipulated and controlled by the investigator. Attribute independent variable(s):  A predictive, defining characteristic of individuals. Cannot be manipulated. 6

 Random Sample  System for choosing participants from a population  Generally, the larger the sampling population the better your generalizability becomes.  Random Assignment  Method for assigning participants randomly to experiment conditions 7

 True experiments protect against both time and group threats to internal validity by randomly assigning subjects to treatment and control groups. The treatment (independent variable) is active.  If we cannot randomly assign subjects to different groups, then it is a quasi-experiment. The independent variable is active.  If we cannot randomly assign subjects to groups because the groups contain the attribute of interest, and if we give all groups the same treatment, then it is an associational non-experiment. The independent variable is not active. 8

 Pre-test: allows us to check group equivalence before the intervention X is introduced.  Experimental manipulation: An independent variable (X) that the experimenter manipulates.  Post-test: allows us to check group equivalence after intervention X has been introduced. 9 Hypothesis  Random Assignment  Measure D.V.  Treatment  Measure D.V. (pre-test)(post-test)

10 Common types of true experiments R O O XO O R X O O R O O XO O (1) (2) XO(3) O(4) Pretest-Posttest Control Post-only Control Solomon 4-group

 Question: Do individuals in Japan and the US make differential choices about ‘unique’ versus ‘less unique’ items when given a choice? 11

 Independent Variable  Cultural difference: Japanese students compared to US students  Assignment  Subjects were not randomly assigned because they already fell into one of the two groups.  Dependent Variable:  Pen layout (3 of one type, 1 of another)  Would they choose the ‘common’ pen or the ‘unique’ one? 12

 Question: Do increased risk-taking behaviors over time increase interpersonal trust? 13

Independent Variable  Experiment Condition (2 conditions):  Fixed partner on every trial, cannot control amount to entrust to partner  Fixed partner on every trial, can control amount to entrust to partner Assignment  Random assignment of participants to one of the 2 conditions.  Same experiment conducted in Japan and US, and comparisons made between the two studies. Dependent Variable  Cooperation rate (i.e., whether they returned the coins to the partner or not) 14

 Measurement Concerns  Construct Validity  Design Concerns  Internal Validity  External Validity  Ecological Validity 15

 “Face” validity deals with subjective judgement of appropriate operationalization  “Content” validity is a more direct check against relevant content domain for the given construct. 16 How do we know that our independent variable is reflecting the intended causal construct and nothing else?

17 Internal Validity deals with questions about whether changes in the dependent variable were caused by the treatment.

18 Effect ? ? ??

 History ▪ additional I.V. that occurs between pre- test and post-test  Maturation ▪ Subjects simply get older and change during experiment  Testing ▪ Subjects “get used” to being tested  Regression to the Mean ▪ Issue with studies of extremes on some variable 19

 Demand Characteristics ▪ Anything in the experiment that could guide subjects to expected outcome  Experimenter Expectancy ▪ Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy) 20

 Evaluation Apprehension  Solutions  Double-blind experiments  Experiments in natural setting (i.e., subjects do not know they are in an experiment)  Cover stories  Hidden measurements 21

 Naïve experimenter  Those conducting study are not aware of theory or hypotheses in the experiment  Blind  Researcher is unaware of the experiment condition that he/she is administering  Standardization  Experimenter follows a script, and only the script  “Canned” Experimenter  Audio/Video/Print material gives instructions 22

 Selection Bias ▪ Issue with non-random selection of subjects  Mortality ▪ Departure of subjects in the experiment  Diffusion, Sharing of Treatments ▪ Control group unexpectedly obtains treatment  Other ‘social’ threats? ▪ Compensatory rivalry, resentful demoralization, etc. 23

 Setting  Population  History 24 External Validity– How far does the given experiment generalize to similar groups, individuals, etc?

25

26 Internal Validity External & Ecological Validity Balance is important between the types of validity, but internal validity is usually (if not always) the more important factor.

 Some experiments can be conducted in a real-world setting while maintaining random assignment and manipulation of treatments 27 Milliman (1986) Study of music tempo and restaurant customer behavior Cheshire and Antin (2008) Study of Incentives and Contributions of Information in an Online Setting

 1998 Total Solar Eclipse: testing temperature of sun’s corona 28

 Pro’s  Gives researcher tight control over independent factors  Allows researcher to test key relationships with as few confounding factors as possible  Allows for direct causal testing  Con’s  Often very small N; enough for statistical purposes but not ideal for generalizability  Sometimes give up large amounts of external validity in favor of construct validity and direct causal analysis  Require a large amount of planning, training, and time– sometimes to test relationship between only 2 factors! 29

Cost and Effort  Is the effort worth it to test the concepts you are interested in? Manipulation and Control  Will you actually be able to manipulate the key concept(s)? Importance of Generalizability  Are you testing theory, or trying to establish a real-world test? 30