1. Experiments
Liang Dai

2. Self introduction Liang Dai 戴亮 Assistant professor of finance, 2015.8-
Ph.D. in economics, Princeton University, 2015
Bachelor in economics and finance, The University of Hong Kong, 2009
First time teaching this course
All material based on my own knowledge
So Feedback more than welcomed!

3. Research methods Theory :
1) understand underlying mechanisms behind observed facts; e.g. wage premium of large cities
2) thought experiments; e.g. MM Theory
3) provide tractable analytical framework. E.g. rational inattention
Empirics: 1) summarize previously unknown facts; 2) test existing theories; 3) counterfactual analysis for policy recommendations
Reduced form approach (focus of my lectures)
Structural approach

4. Structural v.s. Reduced form
Key difference: whether to start with a model
E.g. Data: a time series of price and quantity of a good, plus people’s average income, wage, capital rent.
Challenge: supply or demand shock?
Structural:
start with a partial equilibrium model with consumers and producers
Assume players in reality act as in the model
Derive structural equations (consumers’ FOC, producers’ FOC, market clearing conditions)
Estimate parameters of utility functions and production functions by fitting the data
Pro: low data requirements, easy to do counterfactual analysis
Con: strong assumptions due to the model
Reduced form: find special data episodes and/or use the methods to be introduced
E.g. weather in Brazil on coffee market in Shanghai

5. Reduced form approach Endogeneity problem
Experiments/quasi-experiments
Natural experiments
Instrumental variables (IV)
Differences in Differences (DD)
Regression Discontinuity (RD)
Successful empirical work using Chinese data (time permitted)
Only intuitively: NOT an econometrics class

6. Examples Top 5 economics journal publications
American Economic Review (AER)
Econometrica (ECMA)
Quarterly Journal of Economics (QJE)
Journal of Political Economy (JPE)
Review of Economic Studies (RES)
Illustration of methodology
Showcase classic and successful economic research papers from different fields
Guidance for your own topic selection

7. Selection Bias Yi=Y0i+(Y1i-Y0i)*Di
Alice
Bob
Potential outcome (wake-up time) without alarm clock: Y0i 10 7
Potential outcome with alarm clock: Y1i 8
Treatment (alarm clock used): Di 1
Actual outcome: Yi
Treatment effect: Y1i-Y0i -2
Yi=Y0i+(Y1i-Y0i)*Di
Causal effect of alarm clock on Alice’s wake up time is:
Y1,Alice – Y0, Alice = 8-10=-2
Analogy:
Average treatment effect on the treated
=E[Y1i-Y0i|Di=1]=E[Y1i|Di=1]-E[Y0i|Di=1]

8. Selection Bias (cont.) Data actually observable:
Alice
Bob
Treatment (alarm clock used): Di 1
Actual outcome: Yi 8 7
Observed difference in group= E[Y1i|Di=1]-E[Y0i|Di=0]
Here, YAlice-YBob = 8-7=1
Alarm clock delays wake-up?!
Note that here, we don’t observe “types” of subjects (i.e., unobservable factors, as residual in a regression)

9. Selection Bias (cont.)
1=YAlice-YBob = (Y1,Alice –Y0,Alice)+(Y0,Alice-Y0,Bob)
=(8-10)+(10-7)
Y1,Alice –Y0,Alice : causal effect of alarm clock on Alice, what we want
Y0,Alice-Y0,Bob : selection bias

10. Selection Bias (cont.) Observed difference in group means
=E[Y1i|Di=1]-E[Y0i|Di=0]
= {E[Y1i|Di=1]-E[Y0i|Di=1]}- {E[Y0i|Di=1]-E[Y0i|Di=0]}
=Average causal effect + Selection bias
Average causal effect here=Treatment effect of the treated (ToT)
We want to remove the latter to obtain the former
How?

11. Endogeneity problem Selection bias is also called Endogeneity problem.
E[Y0i|Di=1]-E[Y0i|Di=0], or equivalently, corr(u,D)!=0.
i.e. Treatment is endogenous to unobservable factors
Forms: omitting variables, reverse causality, measurement error.
Elimination of selection bias makes corr(u,D)=0, eliminating other forms of endogeneity problem

12. Omitting variables
Relevant variables omitted, either due to negligence or lack of data, and
Variables omitted are correlated with treatment variables
E.g. Alice’s type that makes her use an alarm clock is not observed.
E.g. Celebrating birthdays makes you live longer?

13. Reverse causality
Economists are interested in underlying mechanisms that drive the economic phenomena we see.
Models and theories are developed to understand causal relations between economic variables.
Thus, empirical work should help identify such causal relations, in addition to correlations between variables.
But in data, we do not observe the direction of causality directly.

14. Reverse causality
If Y and D are positively correlated, it could be higher D causes higher Y as anticipated, but also higher Y caused higher D.
The latter is Reverse Causality
Y-> Unobservables->D, if U and D correlated
Alice: “Because I know I can wake up earlier with an alarm clock, I use it.”
Celebrating birthday is conducive to health?

15. Measurement error
In data we see, variables are measured with error, as part of unobservables U
Error could be correlated with treatment D
“Alice is more eager to see the results than Bob.”
U correlated with D.
It might be the case that data show Alice wake up earlier but Bob doesn’t, because only Alice carefully observes and reports it while Bob doesn’t.
Accounting data of big firms v.s. small firms

16. Ideal Experiment
Ideally, the most intuitive and straightforward way to get rid of selection bias is to randomly select students, and compel those selected to use alarm clocks.
Those who use alarm clocks are a priori identical to those who don’t, since they have equal chance to be selected.

17. Method 1: Experiments or Quasi Experiments
Randomly Assign Di
This makes E[Y0i|Di=1]=E[Y0i|Di=0],
and E[Y1i|Di=1]=E[Y1i|Di=0]
Thus E[Y1i|Di=1]-E[Y0i|Di=0]
= E[Y1i-Y0i|Di=1]=E[Y1i-Y0i|Di=0]=E[Y1i-Y0i]
I’ll only cover field experiments. Jiang Ming’ll cover lab experiments later.

18. Example 1: MTO on neighborhood effect (Quasi-experiment)
罗老师： 下面是我们系秘书王梦发来的机票代理的联系方式。请您联系代理，告诉她你要的航班。 谢谢！ 戴亮
Example 1: MTO on neighborhood effect (Quasi-experiment)
Neighborhood environment can have substantial impact on individuals’ economic, health, and educational background （孟母三迁）
Yet hard to find clean evidence. Why?

19. Example 1: MTO on neighborhood effect (Quasi-experiment)
罗老师： 下面是我们系秘书王梦发来的机票代理的联系方式。请您联系代理，告诉她你要的航班。 谢谢！ 戴亮
Example 1: MTO on neighborhood effect (Quasi-experiment)
Neighborhood environment can have substantial impact on individuals’ economic, health, and educational background （孟母三迁）
Yet hard to find clean evidence due to self-selection problem (e.g. family background)
Moving To Opportunity (MTO) experiment randomly assigns families to different neighborhoods

20. MTO: details
1994~1998, in 5 large U.S. cities, by U.S. Dept. of Housing and Urban Development
Eligible if have children and reside in public housing or Section 8 assisted housing in census tracts with 1990 poverty rate >40%
4604 families randomly assigned to 3 groups:

21. MTO: details
1) experimental group: received a voucher to lease a unit only in census tracts with 1990 poverty rate <10%
2) Section 8 comparison group: received the same voucher but without geographical restriction
3) Control group: did not receive voucher
Collected data on health, economic sufficiency, behavior problems

22. A series of papers over time…
Katz, Jeffrey & Kling (QJE 01): based on Boston subsample, on all 3 outcome categories;
Kling, Ludwig & Katz (QJE 05): on crime;
Kling, Liebman & Katz (ECMA 06): on all 3 categories;
Chetty, Hendren & Katz (AER 16): on long-term outcomes ……

23. Example 2: Cai, Chen & Fang, AER 09
Two channels of social learning: direct communication, observational learning
Difference: whether individuals have to be close in time, space and social distance
Purpose of this research: to provide convincing empirical evidence for the latter
Why worthwhile?
different policy implications: whether info campaign is effective

24. Challenge Two confounding mechanisms:
Saliency effect v.s. conformity effect
Different information content
Need to distinguish, but hard
This paper: randomly “assign” subjects into 3 groups: control; saliency treatment; ranking treatment

25. Experiment design
Restaurant “Meizhou Dongpo” (眉州东坡), w/ a thick menu of 60 hot dishes and ~50 tables each location
Control table: no additional info other than the menu
Ranking treatment table: display of the names of “top 5” dishes sorted by actual number of plates sold in the previous week
Saliency treatment table: plaque listing 5 “sample dishes”

26. Advantages AA is uncommon in China => Table as unit of analysis
Direct communication ruled out
No way to communicate with past customers
Typically don’t talk to other tables
Commonality of decision problems
Easy to implement randomization and observe diners’ choice, can survey on spot their subjective dining experience

27. Results
Demand for top 5 dishes by 13%~20% depending on specification on ranking treatment, but not significantly on saliency treatment
Modest evidence based on survey that:
Observational learning effect stronger among infrequent customers;
Subjective experience improved on ranking treatment, but not on saliency treatment

28. Summary of Example 2
Empirical work done by Chinese researchers in China, but universal and important research question=>top journal publication
Special features in Chinese specific context help resolve key identification issues

29. Summary of Experiment
Resolve selection bias by randomization on subjects’ reception of treatment
Most straightforward
Although hard and costly to implement on a large scale, feasible for smaller scale on less ambitious research questions

30. Natural Experiments
“Experiments” that happened “naturally”, i.e. not designed or controlled by researchers
Much Less expensive than field experiments with similar sample size
Thus more widely used in empirical work

31. Example 1 Ashenfelter & Krueger, AER 94 Economic return to education
Treatment: years of education
Outcome: wage
Challenge: ?
Ideal experiment: ?

32. Example 1 Ashenfelter & Krueger, AER 94 Economic return to education
Treatment: years of education
Outcome: wage
Challenge: selection bias due to endogeneity of schooling
Solution: compare twins with different years of education

33. Data collection
Interview twins at 16th Annual Twins Days Festival in Twinsburg, Ohio, in Aug 1991
The largest gathering in the world that year, attracting >3000 sets of twins, triplets and quadruplets.
495 separate individuals over 18 y.o. interviewed in 3 days
Each twin interviewed separately for cross-checking
Analysis focused on subsample of identical twins (同卵双胞胎), instead of maternal twins (异卵双胞胎)

34. Main result

35. Example 2 Jones & Olken, QJE 05
Research question: Do (national) leaders matter for economic growth?
Challenge: ?
Ideal experiment: ?

36. Example 2 Jones & Olken, QJE 05
Research question: Do (national) leaders matter for economic growth?
Challenge: Endogeneity of leader turnover
In particular, could be driven by economic conditions, i.e. reverse causality

37. Solution
focus on cases where leaders’ rule ended at death for natural causes or accidents
i.e., treatment group: the country with the new leader
Control group: the same country with the original leader
Selection bias killed by randomization by “nature”
Sample: 57 such leader transitions with growth data available from Penn World Tables
Field experiments infeasible in this context

38. Main Results
Growth pattern changes significantly in a sustained fashion across these leadership transitions
Death of leaders in autocratic regimes leads to changes in growth, while death of leaders in democratic regimes does not.
Significant effect on monetary policy, but mixed evidence on fiscal and trade policies, and insignificant effect on external conflict or civil wars