1. Experimental Methodology
Disclaimer: These slides are meant to give you guidelines for going over your notes on what we discussed in class. They are in no way a substitute for taking notes during class, and anything I mention in class is fair game for the exams

2. Some Terminology
Treatment: a particular condition of the experiment. Often we have a (main) treatment and a control treatment (or more). The control is “untreated”. For example, if we want to look at the effects of a drug, the control treatment would be not taking the drug (or taking a placebo).
An experiment usually consists of several sessions. In a session a group of people takes part in the experiment at a particular date and place.
In a session, there are often a number of rounds, or periods.
Subjects = participants in the experiment

3. Treatment vs. control
In a lab we have the advantage of being able to control many variables. E.g. number of bidders in an auction, how much information people have about others etc. etc.
We can keep these variables constant or vary them.
The ones you vary will be your treatment variables.
The more treatment variables you have, the more you
can learn about the effects of these variables, but at the same time your experiment will be more expensive since you will need to collect more data.
For example, if I looked at bidding in two different auction formats, first-price and second-price, and held the number of bidders constant at 2 in each auction, I would be controlling the number of bidders. My treatment variable would be the auction format.

4. Avoiding “Confounds”
Suppose again that you are interested in bidding in two different auction formats, first-price sealed-bid and 2nd price auctions. But let’s say you are also interested in seeing how bidding changes with the number of bidders.
Suppose you did two treatments:
First-price sealed-bid auction with 2 bidders.
2nd price auction with 4 bidders.
The problem is that the effects will be CONFOUNDED. You only learn about the interaction; nothing about the two variables on their own.
You should never change two variables at the same time!
That is, you should vary all treatment variables independently.

5. Indirect control: Randomization
When we do experiments, there will be “nuisance variables” that may or may not be observable. These are things that we are not interested in, but may affect behavior.
The key is they should not be confounded with your focus variables (what you are interested in).
Example:
Suppose I always assign early comers to the lab to one treatment, and the late-comers to another treatment. One problem may be selection: maybe the people who come early are a different type of person than the late-comer, and this could, say, affect how they behave in my experiment.
This can be avoided by randomization.=>Assign people to treatments, roles, etc. RANDOMLY.

6. Controlling Preferences in the Lab: Induced Value Theory
In many experiments, the experimenter wants to control subjects’ preferences. How can this be achieved? Subjects’ “homegrown” preferences must be neutralized and new preferences that fit the design of the experiment must be “induced”. Subjects’ actions must be driven by these induced preferences. For example, if we are having them play a game, we want subjects’ utility to correspond to the payoffs that we want to implement in the game.

7. One very important thing here is the rewards we give during the experiment. Consider the following model of how subjects behave: Subject’s unobservable preferences are given by: V(m, z) where m is money and z represents all other motives (e.g. boredom, jealousy about others’ payoffs, expectations etc.). We cannot observe z. m=(m0+∆m), where m0=subjects’ outside money, and is the money earnings during the experiment. Now, for us to be able to induce preferences, we need a few things.

8. Interpretation of z:
For example, if it is boredom, it could create game playing incentives or random decisions.
e.g. If you have pressed 30 times the x-button you may be bored and like to see what happens if you press the y-button…Especially relevant when you have long experiments.
Public information on everyone’s payoffs could make relative comparison motives important (envy, concerns about fairness)
In some cases, subjects may realize what the experiment is about and want to help or hinder the experimenter (“experimenter demand effects”)
e.g. There was an “experiment” being done at a factory to see if some new procedure would increase workers’ productivity. Workers realized that increased productivity was what the management expected from the experiment, and therefore worked harder, but not because of the effect of the new procedure. Because they wanted to look good to the management. This is the canonical example of an experimenter demand effect.

9. ASSUMPTIONS FOR INDUCED VALUES:
1. Monotonicity: Subjects must prefer more of the reward medium to less and not become satiated.
Formally: ∂V/ ∂m exists and is strictly positive for every feasible combination of (m,z).
2. Salience: The reward Δm should depend on a subject’s
actions (e.g. a fixed show-up fee is not salient).
Dominance: Changes in a subject’s utility from the experiment come predominantly from Δm and the influence of z is negligible (this assumption is the most critical).
*** If these conditions are satisfied, the experimenter has control about the subjects’ preferences, i.e., there is an incentive to perform actions that are paid.
Problem: V and z are not observable.

10. Potential solutions: Make Δm sufficiently large Avoid public information about payoffs Do not give hints about the purpose of the experiment Use a neutral language in the instructions 1. Do not use suggestive language in instructions (e.g. “if you do this, you can get a larger payoff…”) 2. Use strategy names such as Up/down, A, B, C (generic things) rather than things that may invoke personal values (e.g. don’t call strategies “cooperate” or “cheat”, call them A and B).

13. Between- versus Within- Subject Design
Between-subject design: Each subject participates only in one treatment.
Within-subject design: Each subject participates in more than one treatment.
Let y be the outcome we are interested in and  the treatment effect. “i” is an index for subject i. t is for “treated” and u is for “untreated”. Then,
Within person design: i = yi1 – yi0
Between person design:  = yt* - yu*

14. Within-Subject-Design:
Allows individual comparison
Control for individual fixed effects (things that we do not observe, that are constant for an individual, e.g. how much she cares about others’ payoffs, how “rational” she is etc.)
More powerful statistical tests than possible with between-subjects design, especially when the sample size is small.
Cheaper to run, since you need to use fewer subjects.
But, there is an “order effect” problem.
– In the second treatment subjects have learned
something already—what happens in the first treatment can affect what happens in the 2nd
Possible Solution: reverse order to control for order effects—some subjects go through first treatment A then treatment B, some subjects go through first treatment B then treatment A.
Question for you: What if we have 5 treatments? 10 treatments?

15. In between-subjects design, we rely on randomization to be able to say something about treatment effects. How? Suppose that each individual has different characteristics that we cannot see. Since the same individual does not go through both treatments, we are comparing different people under the different treatments. But, if you assign people to treatments randomly, and if you have a large enough sample, the groups under different treatments will be similar. You can therefore say something about treatment effects.

16. How “valid” are laboratory data?
1. Internal validity: Do the data permit correct causal inferences? If you design your experiment well and control the things you need to control, internal validity can be achieved. 2. External validity: Is it possible to generalize from lab to field? Can what you observe in the lab say something about the “real world”? The experimentalist’s answer is yes. =>When possible, use experiments in conjunction with field experiments and/or naturally occurring data. Studies that show that behavior in “games” correlate with real choices are very popular.

17. Field vs. Lab Experiments:
Field experiments can address some of the criticisms directed at lab experiments. They usually are more realistic, but it is harder to achieve “control” outside the lab.
conventional lab experiment (Lab)
employs a standard subject pool of students, an abstract framing, and an imposed set of rules. (e.g. make them play a game with strategies up, down, left, right, and give them payoffs accordingly).
artefactual field experiment (AFE)
same as a conventional lab experiment but with a non-standard subject pool. If I went to a firm and ran the above experiment with actual workers, still using the same structure, if would be an AFE.
framed field experiment (FFE)
same as an artefactual field experiment but with field context in the commodity, task, information, stakes, time frame, etc. We saw an example of this in List (2003), endowment effect field experiment. He let people trade sports cards , and went to an actual market.
natural field experiment (NFE)
same as a framed field experiment but where the environment is the one that the subjects naturally undertake these tasks, such that the subjects do not know that they are in an experiment. Example: Suppose I am looking at the determinants of donating to charity, specifically, whether it matters if I include a gift or not. Suppose I send some people a donation letter with a small gift and to others a donation letter with no gift, and look at which group donates more, then this would be a natural field experiment, since subjects have no idea that they are in an experiment.

18. Eliciting Preferences
Sometimes, rather than “inducing” preferences, we are interested in finding out about subjects’ preferences or valuations (this is called “eliciting” preferences). But, since we pay money, and that could create incentives to misrepresent valuations, we need to find what we call an “incentive compatible” way of eliciting preferences. For example: let’s say I want to know how much you value a certain good (e.g. a coffee mug). If you know that I will pay money, you may have an incentive to overstate your valuation (e.g. say you would sell it at $10 minimum, although you in fact would be willing to sell it at $8). An incentive compatible way of eliciting preferences: Becker, DeGroot, Marschak (BDM) Procedure

19. How BDM Works Use in eliciting WTP or WTA.
Example: Eliciting maximum WTP
Subject is asked to state her maximum WTP for an experimental object.
The computer draws a random price from an interval.
If subject’s stated max. WTP>=p, subject pays p and buys the good.
If subject’s stated max. WTP<p, she does not buy the good.
WHY IS THIS INCENTIVE COMPATIBLE (WHY DOESN’T THE SUBJECT HAVE ANY INCENTIVE TO UNDERSTATE WTP)?
EQUIVALENTLY, ONE COULD DO A 2ND PRICE AUCTION FOR THE OBJECT AND ASK PEOPLE TO SUBMIT BIDS. THE HIGHEST BIDDER WINS THE OBJECT BUT PAYS THE 2ND HIGHEST BID (NOT HER OWN BID). THIS MECHANISM WOULD ALSO ELICIT VALUES TRUTHFULLY. WHY?

20. Strategy Method
Used in game-theoretic experiments, mainly. Elicit a whole strategy rather than just one action. The idea is that, instead of just playing the game and responding to whatever the other person does, subjects are asked to indicate an action at each possible information set. e.g. Suppose we are interested in a sequential prisoner’s dilemma game. Player 1 moves first, then player 2 decides after observing player 1’s move.

21. Here, the strategy method would involve asking: -What will you do if player 1 plays Cooperate? -What will you do if player 1 plays Defect? So, we elicit responses in each contingency. Then, once player 1 makes a decision, player 2’s payoffs are automatically determined according to what her strategy said for that contingency. Advantage: Get more information about actions in parts of the game tree that are not likely to be reached. Disadvantage: Especially in games where some actions lead to anger or emotions, the strategy method may not be able to predict what will happen.

22. Inducing Risk-Neutrality
Controlling for risk preferences can be important for proper inference from behavior in experiments. We can either get info on risk preferences by giving subjects lottery choices (like we did in class) at the end of any experiment, or we can try to make subjects behave in a risk-neutral way (induce risk-neutrality). One method that has been used to induce risk-neutrality (not clear if it works or not in reality!) is denominating the payoffs in the experiment in terms of “lottery tickets”. Specifically, let n be the number of tickets earned, out of a theoretical maximum of N tickets. The second stage of the experiment is a lottery, where there is a high prize (wH) and a low prize (wL), and the probability of getting the high prize is n/N. The expected utility from this lottery is then given by: EU=(n/N) U(wH)+ (1-(n/N))U(wL) (Note: if U(.) is concave, the subject is risk-averse).

23. Without loss of generality, normalize U(wH)=1 and U(wL)=0
Without loss of generality, normalize U(wH)=1 and U(wL)=0. The above equation reduces to EU=n/N, which is linear in n. Which means, even a risk-averse subject will be risk-neutral in decisions that earn payoffs in terms of lottery tickets.
Example: Consider the choice between two decisions: Decision 1 gives 50 tickets (out of a possible 100) with certainty, and Decision 2 gives 25 tickets and 75 tickets with equal probability. Both decisions imply a 50% chance of winning the high prize. Therefore, any EU maximizer will be indifferent between the choices, i.e. a risk-averse subject will behave like a risk-neutral subject when the payoffs are in lottery tickets themselves.

24. Eliciting Beliefs
How do we find out what subjects believe about the realization of a random variable, or about what another subject is likely to play?
Having information about subjects’ beliefs is important both in individual decision problems and in games.
Example from individual decision-making: Suppose we want to find out how subjects’ beliefs about an uncertain event will change with new information. I can give people information, then ask what they think. E.g. 2 urns (urn A and urn B) with different numbers of red and black balls. Suppose each urn is equally likely. I draw one ball from the urn, it’s red. I want to know what is the subject’s perceived probability of the urn being A after this information.
Example from game theory: Prisoner’s dilemma
Before subjects make their decisions both players are asked what is the probability with which they think the other player will cooperate/defect?

25. THE GOOD THING ABOUT BELIEF ELICITATION:
Beliefs can be very informative to understand the motivations behind subjects’ behavior
Beliefs are of particular importance to check the rationality of decisions (e.g. guessing game)
GENERAL ISSUES/PROBLEMS WITH BELIEF ELICITATION
Experimenter-Demand–Effect (you might make people think about stuff they would not have thought about otherwise)
Directs focus on particular problems, e.g., guessing game!
Desire to be consistent: people state beliefs to “match” their actions
Desire to justify actions: Someone defects or does something “selfish”, and states that she thought the other person would defect also.
Truthful elicitation mechanisms can be complicated to explain (e.g., payment dependent on distance measure between true outcome and expected outcome)
Procedures’ incentive compatibility can depend on risk-neutrality.
Can pollute incentives in the experiment if people “hedge” decisions and beliefs, to guarantee a sure payoff.

26. In general, suppose that the state can be either A or B
In general, suppose that the state can be either A or B. I want to find out the probability with which the subject thinks that the state is A. I want to elicit this in some way that will lead people to be truthful about what they think.
A commonly-used method for belief elicititation is the “quadratic scoring rule” procedure. We ask the subject to report her subjective probability for state A happening, and then pay her according to the actual realization of the state.
Let r be the reported probability for state A. Let I be an indicator variable: I=1 if state A realizes. I=0 if state A does not realize. Then,
Payoff=1-(r-I)2
Payoff=1-r2 if I=0, 2r-r2 if I=1
Note that the worst payoff (=0) happens when you assign probability 1 to A and the actual state is not A, and the best payoff (=1) happens when you assign probability 1 to A and the actual state is A.

27. This mechanism is “incentive-compatible” (induces truthful revelation) if the subject is a risk-neutral expected utility maximizer. If her subjective probability is p, and her reported probability is r, the subject’s expected payoff is: EU=p(2r-r2)+(1-p)(1-r2) It is possible to show that the EU will be maximized when r=p (when the subject tells the truth). Verify this! Potential problem: If you are not risk-neutral, the mechanism is no longer incentive compatible.

28. What is a good experiment
What is a good experiment? (No need to memorize these but good to know for your project. You should address these in your project paper).
Seven questions by Shyam Sunder (1994):
1. What is the question that you would like to have answered after the experiment? (Your answer should be a single sentence with a question mark at the end.)
2. What do you know already about the possible answers to the question you have stated above? (e.g. what does economic theory say? What do other studies say?)
3. What are the various possible ways of finding an answer to the question you have stated above? Include both experimental as well as any other methods you know.

29. 4. What are the advantages and disadvantages of using an experiment to find an answer? (e.g. disadvantage: may be hard to do in the lab, unrealistic etc.) 5. What are the chances that the answer you get from the experiment will surprise you or others? What are the chances that it will change someone's mind? 6. How would you conduct the experiment? (Write down a design, full set of procedures and instructions.) 7. Is your experimental design the simplest possible design to help answer the question you have stated? (usually, the simpler, the better; otherwise it could be hard to interpret the data).