1. Experimental Methodology
Seda Ertac
Fall 2009
Econ 430/530

2. Experimental Methodology
Disclaimer: These slides are meant to give you guidelines for going over your notes on what we discussed in class. They are in no way a substitute for taking notes during class, and anything I mention in class is fair game for the exams

3. Some Terminology
Treatment: a particular condition of the experiment. Often we have a (main) treatment and a control treatment (or more). The control is “untreated”. For example, if we want to look at the effects of a drug, the control treatment would be not taking the drug (or taking a placebo).
An experiment usually consists of several sessions. In a session a group of people takes part in the experiment at a particular date and place.
In a session, there are often a number of rounds, or periods.
Subjects = participants in the experiment

4. Treatment vs. control
In a lab we have the advantage of being able to control many variables. E.g. number of bidders in an auction, how much information people have about others etc. etc.
We can keep these variables constant or vary them.
The ones you vary will be your treatment variables.
The more treatment variables you have, the more you
can learn about the effects of these variables, but at the same time your experiment will be more expensive since you will need to collect more data.
For example, if I looked at bidding in two different auction formats, first-price and second-price, and held the number of bidders constant at 2 in each auction, I would be controlling the number of bidders. My treatment variable would be the auction format.
Note: If you don’t know what an auction is, see next slide to be able to understand this example…

5. “Aside” for people who have never heard of auctions:
An auction is a selling mechanism where different people bid for an object. The highest bidder wins. There are different auction formats. Two examples:
First Price Auction: The highest bidder wins, and pays her own bid.
Second Price Auction: The highest bidder wins, but pays the 2nd highest bid (not his own bid).

6. Avoiding “Confounds”
Suppose again that you are interested in bidding in two different auction formats, first-price sealed-bid and 2nd price auctions. But let’s say you are also interested in seeing how bidding changes with the number of bidders.
Suppose you did two treatments:
First-price sealed-bid auction with 2 bidders.
2nd price auction with 4 bidders.
The problem is that the effects will be CONFOUNDED. You only learn about the interaction; nothing about the two variables on their own.
You should never change two variables at the same time!
That is, you should vary all treatment variables independently.

7. Indirect control: Randomization
When we do experiments, there will be “nuisance variables” that may or may not be observable. These are things that we are not interested in, but may affect behavior.
The key is they should not be confounded with your focus variables (what you are interested in).
Example:
Suppose I always assign early comers to the lab to one treatment, and the late-comers to another treatment. One problem may be selection: maybe the people who come early are a different type of person than the late-comer, and this could, say, affect how they behave in my experiment.
This can be avoided by randomization.=>Assign people to treatments, roles, etc. RANDOMLY.

8. Controlling Preferences in the Lab: Induced Value Theory
In many experiments, the experimenter wants to control subjects’ preferences. How can this be achieved? Subjects’ “homegrown” preferences must be neutralized and new preferences that fit the design of the experiment must be “induced”. Subjects’ actions must be driven by these induced preferences.

9. For example, if we are having them play a game, we want subjects’ preferences to be represented by the payoffs that we want to implement in the game.
Cooperate
Defect
2,2
0, 3
3,0
1,1

10. Way to do it: Monetary Rewards
Consider the following model of how subjects behave: Subject’s unobservable preferences are given by: V(m, z) where m is money and z represents all other motives (e.g. boredom, jealousy about others’ payoffs, expectations etc.). We cannot observe z. m=(m0+∆m), where m0=subjects’ outside money, and is the money earnings during the experiment. Now, for us to be able to induce preferences, we need a few things.

11. Interpretation of z:
For example, if it is boredom, it could create game playing incentives or random decisions.
e.g. If you have pressed 30 times the x-button you may be bored and like to see what happens if you press the y-button…Especially relevant when you have long experiments.
Public information on everyone’s payoffs could make relative comparison motives important (envy, concerns about fairness)
In some cases, subjects may realize what the experiment is about and want to help or hinder the experimenter (“experimenter demand effects”)
e.g. There was an “experiment” being done at a factory to see if some new procedure would increase workers’ productivity. Workers realized that increased productivity was what the management expected from the experiment, and therefore worked harder, but not because of the effect of the new procedure. Because they wanted to look good to the management. This is the canonical example of an experimenter demand effect.

12. PRINCIPLES FOR INDUCED VALUES:
1. Monotonicity: Subjects must prefer more of the reward medium to less and not become satiated.
Formally: ∂V/ ∂m exists and is strictly positive for every feasible combination of (m,z).
2. Salience: The reward Δm should depend on a subject’s
actions (e.g. a fixed show-up fee is not salient).
Dominance: Changes in a subject’s utility from the experiment come predominantly from Δm and the influence of z is negligible (this assumption is the most critical).
*** If these conditions are satisfied, the experimenter has control about the subjects’ preferences, i.e., there is an incentive to perform actions that are paid.
Problem: V and z are not observable.

13. Potential solutions: Make Δm sufficiently large Avoid public information about payoffs Do not give hints about the purpose of the experiment Use a neutral language in the instructions 1. Do not use suggestive language in instructions (e.g. “if you do this, you can get a larger payoff…”) 2. Use strategy names such as Up/down, A, B, C (generic things) rather than things that may invoke personal values (e.g. don’t call strategies “cooperate” or “cheat”, call them A and B).

16. Economics vs. Psychology
Many common areas of interest (e.g. judgment and decision-making), different methodological practices. Economics: 1.Salient monetary rewards (vs. participation fee only). 2.Context-free to the extent possible.(*) 3.No deception!!!

17. Between- versus Within- Subject Design
Between-subject design: Each subject participates only in one treatment.
Within-subject design: Each subject participates in more than one treatment.
Let y be the outcome we are interested in and  the treatment effect. “i” is an index for subject i. t is for “treated” and u is for “untreated”. Then,
Within person design: i = yi1 – yi0
Between person design:  = yt* - yu*

18. Within-Subject-Design:
Allows individual comparison
Control for individual fixed effects (things that we do not observe, that are constant for an individual, e.g. how much she cares about others’ payoffs, how “rational” she is etc.)
More powerful statistical tests than possible with between-subjects design, especially when the sample size is small.
Cheaper to run, since you need to use fewer subjects.
But, there is an “order effect” problem.
– In the second treatment subjects have learned
something already—what happens in the first treatment can affect what happens in the 2nd
Possible Solution: reverse order to control for order effects—some subjects go through first treatment A then treatment B, some subjects go through first treatment B then treatment A.
Question for you: What if we have 5 treatments? 10 treatments?

19. In between-subjects design, we rely on randomization to be able to say something about treatment effects. How? Suppose that each individual has different characteristics that we cannot see. Since the same individual does not go through both treatments, we are comparing different people under the different treatments. But, if you assign people to treatments randomly, and if you have a large enough sample, the groups under different treatments will be similar. You can therefore say something about treatment effects.

20. How “valid” are laboratory data?
1. Internal validity: Do the data permit correct causal inferences? If you design your experiment well and control the things you need to control, internal validity can be achieved. 2. External validity: Is it possible to generalize from lab to field? Can what you observe in the lab say something about the “real world”? The experimentalist’s answer is yes. =>When possible, use experiments in conjunction with field experiments and/or naturally occurring data. Studies that show that behavior in “games” correlate with real choices are very popular.

21. Field vs. Lab Experiments:
Field experiments can address some of the criticisms directed at lab experiments. They usually are more realistic, but it is harder to achieve “control” outside the lab.
conventional lab experiment (Lab)
employs a standard subject pool of students, an abstract framing, and an imposed set of rules. (e.g. make them play a game with strategies up, down, left, right, and give them payoffs accordingly).
artefactual field experiment (AFE)
same as a conventional lab experiment but with a non-standard subject pool. If I went to a firm and ran the above experiment with actual workers, still using the same structure, if would be an AFE.
framed field experiment (FFE)
same as an artefactual field experiment but with field context in the commodity, task, information, stakes, time frame, etc. We saw an example of this in List (2003), endowment effect field experiment. He let people trade sports cards , and went to an actual market.
natural field experiment (NFE)
same as a framed field experiment but where the environment is the one that the subjects naturally undertake these tasks, such that the subjects do not know that they are in an experiment. Example: Suppose I am looking at the determinants of donating to charity, specifically, whether it matters if I include a gift or not. Suppose I send some people a donation letter with a small gift and to others a donation letter with no gift, and look at which group donates more, then this would be a natural field experiment, since subjects have no idea that they are in an experiment.