Experiments Liang Dai

Self introduction Liang Dai 戴亮 Assistant professor of finance, 2015.8- Ph.D. in economics, Princeton University, 2015 Bachelor in economics and finance, The University of Hong Kong, 2009 First time teaching this course All material based on my own knowledge So Feedback more than welcomed!

Research methods Theory : 1) understand underlying mechanisms behind observed facts; e.g. wage premium of large cities 2) thought experiments; e.g. MM Theory 3) provide tractable analytical framework. E.g. rational inattention Empirics: 1) summarize previously unknown facts; 2) test existing theories; 3) counterfactual analysis for policy recommendations Reduced form approach (focus of my lectures) Structural approach

Structural v.s. Reduced form Key difference: whether to start with a model E.g. Data: a time series of price and quantity of a good, plus people’s average income, wage, capital rent. Challenge: supply or demand shock? Structural: start with a partial equilibrium model with consumers and producers Assume players in reality act as in the model Derive structural equations (consumers’ FOC, producers’ FOC, market clearing conditions) Estimate parameters of utility functions and production functions by fitting the data Pro: low data requirements, easy to do counterfactual analysis Con: strong assumptions due to the model Reduced form: find special data episodes and/or use the methods to be introduced E.g. weather in Brazil on coffee market in Shanghai

Reduced form approach Endogeneity problem Experiments/quasi-experiments Natural experiments Instrumental variables (IV) Differences in Differences (DD) Regression Discontinuity (RD) Successful empirical work using Chinese data (time permitted) Only intuitively: NOT an econometrics class

Examples Top 5 economics journal publications American Economic Review (AER) Econometrica (ECMA) Quarterly Journal of Economics (QJE) Journal of Political Economy (JPE) Review of Economic Studies (RES) Illustration of methodology Showcase classic and successful economic research papers from different fields Guidance for your own topic selection

Selection Bias Yi=Y0i+(Y1i-Y0i)*Di Alice Bob Potential outcome (wake-up time) without alarm clock: Y0i 10 7 Potential outcome with alarm clock: Y1i 8 Treatment (alarm clock used): Di 1 Actual outcome: Yi Treatment effect: Y1i-Y0i -2 Yi=Y0i+(Y1i-Y0i)*Di Causal effect of alarm clock on Alice’s wake up time is: Y1,Alice – Y0, Alice = 8-10=-2 Analogy: Average treatment effect on the treated =E[Y1i-Y0i|Di=1]=E[Y1i|Di=1]-E[Y0i|Di=1]

Selection Bias (cont.) Data actually observable: Alice Bob Treatment (alarm clock used): Di 1 Actual outcome: Yi 8 7 Observed difference in group= E[Y1i|Di=1]-E[Y0i|Di=0] Here, YAlice-YBob = 8-7=1 Alarm clock delays wake-up?! Note that here, we don’t observe “types” of subjects (i.e., unobservable factors, as residual in a regression)

Selection Bias (cont.) 1=YAlice-YBob = (Y1,Alice –Y0,Alice)+(Y0,Alice-Y0,Bob) =(8-10)+(10-7) Y1,Alice –Y0,Alice : causal effect of alarm clock on Alice, what we want Y0,Alice-Y0,Bob : selection bias

Selection Bias (cont.) Observed difference in group means =E[Y1i|Di=1]-E[Y0i|Di=0] = {E[Y1i|Di=1]-E[Y0i|Di=1]}- {E[Y0i|Di=1]-E[Y0i|Di=0]} =Average causal effect + Selection bias Average causal effect here=Treatment effect of the treated (ToT) We want to remove the latter to obtain the former How?

Endogeneity problem Selection bias is also called Endogeneity problem. E[Y0i|Di=1]-E[Y0i|Di=0], or equivalently, corr(u,D)!=0. i.e. Treatment is endogenous to unobservable factors Forms: omitting variables, reverse causality, measurement error. Elimination of selection bias makes corr(u,D)=0, eliminating other forms of endogeneity problem

Omitting variables Relevant variables omitted, either due to negligence or lack of data, and Variables omitted are correlated with treatment variables E.g. Alice’s type that makes her use an alarm clock is not observed. E.g. Celebrating birthdays makes you live longer?

Reverse causality Economists are interested in underlying mechanisms that drive the economic phenomena we see. Models and theories are developed to understand causal relations between economic variables. Thus, empirical work should help identify such causal relations, in addition to correlations between variables. But in data, we do not observe the direction of causality directly.

Reverse causality If Y and D are positively correlated, it could be higher D causes higher Y as anticipated, but also higher Y caused higher D. The latter is Reverse Causality Y-> Unobservables->D, if U and D correlated Alice: “Because I know I can wake up earlier with an alarm clock, I use it.” Celebrating birthday is conducive to health?

Measurement error In data we see, variables are measured with error, as part of unobservables U Error could be correlated with treatment D “Alice is more eager to see the results than Bob.” U correlated with D. It might be the case that data show Alice wake up earlier but Bob doesn’t, because only Alice carefully observes and reports it while Bob doesn’t. Accounting data of big firms v.s. small firms

Ideal Experiment Ideally, the most intuitive and straightforward way to get rid of selection bias is to randomly select students, and compel those selected to use alarm clocks. Those who use alarm clocks are a priori identical to those who don’t, since they have equal chance to be selected.

Method 1: Experiments or Quasi Experiments Randomly Assign Di This makes E[Y0i|Di=1]=E[Y0i|Di=0], and E[Y1i|Di=1]=E[Y1i|Di=0] Thus E[Y1i|Di=1]-E[Y0i|Di=0] = E[Y1i-Y0i|Di=1]=E[Y1i-Y0i|Di=0]=E[Y1i-Y0i] I’ll only cover field experiments. Jiang Ming’ll cover lab experiments later.

Example 1: MTO on neighborhood effect (Quasi-experiment) 罗老师： 下面是我们系秘书王梦发来的机票代理的联系方式。请您联系代理，告诉她你要的航班。 谢谢！ 戴亮 Example 1: MTO on neighborhood effect (Quasi-experiment) Neighborhood environment can have substantial impact on individuals’ economic, health, and educational background （孟母三迁） Yet hard to find clean evidence. Why?

Example 1: MTO on neighborhood effect (Quasi-experiment) 罗老师： 下面是我们系秘书王梦发来的机票代理的联系方式。请您联系代理，告诉她你要的航班。 谢谢！ 戴亮 Example 1: MTO on neighborhood effect (Quasi-experiment) Neighborhood environment can have substantial impact on individuals’ economic, health, and educational background （孟母三迁） Yet hard to find clean evidence due to self-selection problem (e.g. family background) Moving To Opportunity (MTO) experiment randomly assigns families to different neighborhoods

MTO: details 1994~1998, in 5 large U.S. cities, by U.S. Dept. of Housing and Urban Development Eligible if have children and reside in public housing or Section 8 assisted housing in census tracts with 1990 poverty rate >40% 4604 families randomly assigned to 3 groups:

MTO: details 1) experimental group: received a voucher to lease a unit only in census tracts with 1990 poverty rate <10% 2) Section 8 comparison group: received the same voucher but without geographical restriction 3) Control group: did not receive voucher Collected data on health, economic sufficiency, behavior problems

A series of papers over time… Katz, Jeffrey & Kling (QJE 01): based on Boston subsample, on all 3 outcome categories; Kling, Ludwig & Katz (QJE 05): on crime; Kling, Liebman & Katz (ECMA 06): on all 3 categories; Chetty, Hendren & Katz (AER 16): on long-term outcomes ……

Example 2: Cai, Chen & Fang, AER 09 Two channels of social learning: direct communication, observational learning Difference: whether individuals have to be close in time, space and social distance Purpose of this research: to provide convincing empirical evidence for the latter Why worthwhile? different policy implications: whether info campaign is effective

Challenge Two confounding mechanisms: Saliency effect v.s. conformity effect Different information content Need to distinguish, but hard This paper: randomly “assign” subjects into 3 groups: control; saliency treatment; ranking treatment

Experiment design Restaurant “Meizhou Dongpo” (眉州东坡), w/ a thick menu of 60 hot dishes and ~50 tables each location Control table: no additional info other than the menu Ranking treatment table: display of the names of “top 5” dishes sorted by actual number of plates sold in the previous week Saliency treatment table: plaque listing 5 “sample dishes”

Advantages AA is uncommon in China => Table as unit of analysis Direct communication ruled out No way to communicate with past customers Typically don’t talk to other tables Commonality of decision problems Easy to implement randomization and observe diners’ choice, can survey on spot their subjective dining experience

Results Demand for top 5 dishes by 13%~20% depending on specification on ranking treatment, but not significantly on saliency treatment Modest evidence based on survey that: Observational learning effect stronger among infrequent customers; Subjective experience improved on ranking treatment, but not on saliency treatment

Summary of Example 2 Empirical work done by Chinese researchers in China, but universal and important research question=>top journal publication Special features in Chinese specific context help resolve key identification issues

Summary of Experiment Resolve selection bias by randomization on subjects’ reception of treatment Most straightforward Although hard and costly to implement on a large scale, feasible for smaller scale on less ambitious research questions

Natural Experiments “Experiments” that happened “naturally”, i.e. not designed or controlled by researchers Much Less expensive than field experiments with similar sample size Thus more widely used in empirical work

Example 1 Ashenfelter & Krueger, AER 94 Economic return to education Treatment: years of education Outcome: wage Challenge: ? Ideal experiment: ?

Example 1 Ashenfelter & Krueger, AER 94 Economic return to education Treatment: years of education Outcome: wage Challenge: selection bias due to endogeneity of schooling Solution: compare twins with different years of education

Data collection Interview twins at 16th Annual Twins Days Festival in Twinsburg, Ohio, in Aug 1991 The largest gathering in the world that year, attracting >3000 sets of twins, triplets and quadruplets. 495 separate individuals over 18 y.o. interviewed in 3 days Each twin interviewed separately for cross-checking Analysis focused on subsample of identical twins (同卵双胞胎), instead of maternal twins (异卵双胞胎)

Main result

Example 2 Jones & Olken, QJE 05 Research question: Do (national) leaders matter for economic growth? Challenge: ? Ideal experiment: ?

Example 2 Jones & Olken, QJE 05 Research question: Do (national) leaders matter for economic growth? Challenge: Endogeneity of leader turnover In particular, could be driven by economic conditions, i.e. reverse causality

Solution focus on cases where leaders’ rule ended at death for natural causes or accidents i.e., treatment group: the country with the new leader Control group: the same country with the original leader Selection bias killed by randomization by “nature” Sample: 57 such leader transitions with growth data available from Penn World Tables Field experiments infeasible in this context

Main Results Growth pattern changes significantly in a sustained fashion across these leadership transitions Death of leaders in autocratic regimes leads to changes in growth, while death of leaders in democratic regimes does not. Significant effect on monetary policy, but mixed evidence on fiscal and trade policies, and insignificant effect on external conflict or civil wars