Global Workshop on Development Impact Evaluation in Finance and Private Sector Rio de Janeiro, June 6-10, 2011 Making the Most out of Discontinuities Florence Kondylis

Introduction (1)  Context  we want to measure the causal impact of an intervention  the assignment to this intervention cannot be randomized  selection into program participation cannot be exploited to establish an adequate comparison group  In general:  Individuals, households, villages, or other entities, are either exposed or not exposed to a treatment / policy regime  selection into the program makes it impossible to compare treated / non- treated to establish the impact of the program  Example: Individuals who wish to take part in a micro-finance program and those who don’t – participation is likely driven by key characteristics 2

Introduction (2)  When randomization is not feasible, how to exploit the roll-out of an intervention to measure its causal impact?  Proposal: we can use quasi/non-experimental methods  Difference-In-Differences and Matching  Regression Discontinuity Design (RDD) 3

Regression Discontinuity Designs  RDD is closer cousin of randomized experiments than other competitors  RDD is based on the selection process  When in presence of an official/bureaucratic, clear and reasonably enforced eligibility rule  A simple, quantifiable score  Assignment to treatment is based on this score  A threshold is established ▪ Ex: target firms with sales above a certain amount ▪ Those above receive, those below do not ▪ compare firms just above the threshold to firms just below the threshold 4

RDD in Practice  Policy: US drinking age, minimum legal age is 21  under 21, alcohol consumption is illegal  Outcomes: alcohol consumption and mortality rate  Observation: The policy implies that  individuals aged 20 years, 11 months and 29 days cannot drink  individuals ages 21 years, 0 month and 1 day can drink  however, do we think that these individuals are inherently different?  wisdom, preferences for alcohol and driving, party-going behavior, etc  People born “few days apart” are treated differently, because of the arbitrary age cut off established by the law  a few days or a month age difference could is unlikely to yield variations in behavior and attitude towards alcohol  The legal status is the only difference between the treatment group (just above 21) and comparison group (just below 21) 5

RDD in Practice  In practice, making alcohol consumption illegal lowers consumption and, therefore, the incidence of drunk-driving  Idea: use the following groups to measure the impact of a minimum drinking age on mortality rate of young adults  Treatment group: individuals 20 years and 11 months to 21 years old  Comparison group: individuals 21 years to 21 years and a month old  Around the threshold, we can safely assume that individuals are randomly assigned to the treatment  We can then measure the causal impact of the policy on mortality rates around the threshold 6

RDD Example 7 MLDA (Treatment) reduces alcohol consumption

RDD Example 8 Total number of Deaths Higher alcohol consumption increases death rate around age 21 Total number of accidental deaths related to alcohol and drug consumption Total number of other deaths

RDD Logic  Assignment to the treatment depends, either completely or partly, on a continuous “score”, ranking (age in the previous case):  potential beneficiaries are ordered by looking at the score  there is a cut-off point for “eligibility” – clearly defined criterion determined ex-ante  cut-off determines the assignment to the treatment or no-treatment groups  These de facto assignments often result from administrative decisions  resource constraints limit coverage  very targeted intervention with expected heterogeneous impact  transparent rules rather than discretion used 9

Example: matching grants (fuzzy design)  Government gives matching grants to firms  Eligibility rule based on annual sales: If annual sales < $5,000 then firm receives grants If annual sales >= $5,000 then no matching grants  A firm with sales=$5,001 would not be treated (be eligible) but would be very similar to a firm with sales=$5,000  Need to measure annual sales before the scheme is announced to prevent manipulation of the figure  RDD would compare firms just above and just below the $5,000 threshold 10

Subtle point …  Question: How to address incomplete compliance to the treatment  Ex: Low take-up of a matching grant scheme  There are two types of discontinuity  Sharp (near full compliance, e.g. a law)  Fuzzy (incomplete compliance, e.g. a subsidy)  Going back to our example … 11

Example: matching grant (fuzzy design)  Now suppose that not all the eligible firms receive the grants. Why?  limited knowledge of the program  voluntary participation  these reasons signal a selection bias into the program: decision to enter the program is correlated with other firm characteristics  Yet, the percentage of participants still changes discontinuously at cut-off  from zero to less than 100%  this is called a fuzzy discontinuity (vs. sharp) 12

Probability of Participation under Alternative Designs 100% 0% 75% 0% Sharp Design for Grant receiptFuzzy Design for Grant receipt 13 Variations above the threshold

Sharp and Fuzzy Discontinuities (1)  Ideal setting: Sharp discontinuity  the discontinuity precisely determines treatment status ▪ e.g. ONLY people 21 and older drink alcohol, and ALL drink it! ▪ Only small firms receive grants ▪ Progressive taxation rate 14

Sharp and Fuzzy Discontinuities (2)  Fuzzy discontinuity the percentage of participants changes discontinuously at cut-off, but not from zero to 100% ▪ e.g. rules determine eligibility but amongst the small firms there is only partial compliance / take-up ▪ Some people younger than 21 end up consuming alcohol and some older than 21 don’t consume at all 15

Internal Validity  General idea  the arbitrary cut off implies that individuals to the immediate left and right of the cut-off are similar  therefore, differences in outcomes can be directly attributed to the policy.  Assumption  Nothing else is happening: in the absence of the policy, we would not observe a discontinuity in the outcomes around the cut off. ▪ there is nothing else going on around the same cut off that impacts our outcome of interest  would not hold if, for instance: ▪ 21 year olds can start drinking however the moment they turn 21 they have to enroll in a “drinking responsibly” type seminar ▪ Grants: there is another policy that gives grants to firms with sales bigger than $5,

Outcome Profile Before and After the Intervention 17 Different shape

External Validity  How general are the results?  Counterfactual: individuals “marginally excluded from benefits”  just under 21  sales just under $5,000  get results for these neighborhoods only  Causal conclusions are limited to individuals, households, villages and firms at the cut-off The effect estimated is for individuals “marginally eligible for benefits” extrapolation beyond this point needs additional, often unwarranted, assumptions (or multiple cut-offs)  [Fuzzy designs exacerbate the problem] 18

Graphical Analysis 19

The “nuts and bolts” of implementing RDDs  Major advantages of RDD  transparency  graphical, intuitive presentation  Major shortcomings  requires many observations around cut-off ▪ (down-weight observations away from the cut-off)  Why?  only near the cut-off can we assume that people find themselves by chance to the left and to the right of the cut-off  think about firms with $1M sales vs. firms with $1,000  or compare a 16 vs a 25 year-old 20

Wrap Up  Can be used to design a prospective evaluation when randomization is not feasible  The design applies to all means tested programs  Multiple cut-offs to enhance external validity ▪ Menu of subsidies targeting various types of firms  Can be used to evaluate ex-post interventions using discontinuities as “natural experiments”. 21

Thank you Financial support from: Bank Netherlands Partnership Program (BNPP), Bovespa, CVM, Gender Action Plan (GAP), Belgium & Luxemburg Poverty Reduction Partnerships (BPRP/LPRP), Knowledge for Change Program (KCP), Russia Financial Literacy and Education Trust Fund (RTF), and the Trust Fund for Environmentally & Socially Sustainable Development (TFESSD), is gratefully acknowledged.