Measuring Impact 1 Non-experimental methods 2 Experiments Vincenzo Di Maro Development Impact Evaluation Some parts of this presentation build on Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material, The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarily those of the World Bank.

Impact evaluation for causal impacts

How we can evaluate this? Regression Discontinuity Design Case: pay reward offered on the basis of exam score new scheme under which all tax inspectors will have to take a compulsory written exam Grades for this exam range from 0 to 100 where 0 means worst outcome and 100 is the best outcome. At the end of the exam all tax inspectors that achieve a minimum score (of 50) will be offered to enter the pay reward scheme. Idea: compare tax inspectors with score a bit below 50 (and unable to choose the reward scheme)…. ….with inspectors with score a bit above 50 (and so eligible for the scheme).

Regression Discontinuity At Baseline Revenues Average tax revenue 50 100 Score

Regression Discontinuity After treatment offered Revenues Discontinuity 50 100 Score

Regression Discontinuity Close to 50 very similar characteristics Revenues Discontinuity 40 50 60 100 Score

Regression discontinuity Powerful method if you have: Continuous eligibility index Clearly defined eligibility cut-off. It gives a causal impact but with a local interpretation Method Treated Control Difference in % Regression Discontinuity $80,215 $69,753 $10,463 15% Problem: Impact is valid only for those subjects that are close to the cut-off point, that is only for tax inspectors that have an exam score close around 50. Is this the group you want to know about?

Summary of impacts so far Method Treated Control/Comparison Difference in % Participants - Non-participants $93,827 $70,800 $23,027 33% Before - After $72,175 $21,653 30% Difference-in-differences 1 (P1-NP1) (P0-NP0) $3,347 $19,590 29% Difference-in-differences 2 (P1-P0) $21,652 (NP1-NP0) $2,062 Regression Discontinuity (RD) $80,215 $69,753 $10,463 15% Weak methods can lead to very misleading results RD (causal impact) is only around half of the impact estimated with the other weaker methods. Valid results from IE only if you use rigorous methods.

Experiments Other names: Randomized Control Trials (RCTs) or Randomization Assignment to Treatment and Control is based on chance, it is random (like flipping a coin) Treatment and Control groups will have exactly the same characteristics (balanced) at baseline. Only difference is that treatment receives intervention, control does not

Experiments: plan Design of experiments How to implement RCTs One treatment and many treatments Encouragement design

Random assignment 3. Randomize treatment 1. Population 2. Evaluation sample Comparison For presentation (with animation effects) Treatment External Validity Internal Validity

Unit of Randomization Choose according to type of program Keep in mind Individual/Household School/Health Clinic/catchment area/Government agency Block/Village/Community Ward/District/Region As a rule of thumb, randomize at the smallest viable unit of implementation. Keep in mind Need “sufficiently large” number of units to detect minimum desired impact: Power. Spillovers/contamination Operational and survey costs

Implementation (pure) Randomization might be not feasible because some eligibles would be excluded from benefits. Usually there are constraints within project implementation that allow randomization: Budget constraints  Lottery Not enough treatments for all eligible subjects Lottery is fair, transparent and ethical way to assign benefits Limited capacity  Randomized phase-in Not possible to treat all the units in the first phase Randomize which group of units is control (they will be treated at later stage, say after 1 year)

Multiple Treatments Different level of benefits Randomly assign people to different intensity of the treatment (20% vs. 30% reward) No evidence on which alternative is best: test variations in treatment: Randomly assign subjects to different interventions Compare one to another Assess complementarities

X Multiple Treatments: 2X2 design Assess complementarities Intervention 2 Control Treatment Intervention 1 X Social recognition reward Monetary reward Both rewards Assess complementarities Overall reward effect

Encouragement design Not always possible to randomly assign to control group: Political and ethical reasons Participation is voluntary and all eligible Randomized promotion/encouragement program available to everyone But provide additional promotion, encouragement or incentives to a random sub-sample: Additional Information. Incentives (small gift or prize). Transport (bus fare).

Encouragement design Randomize Incentive to participate. Ex. small gifts High participation (ex. 80%) Encouraged For presentation (with animation effects) Low participation (ex. 10%) Not encouraged

How we can evaluate this? Randomized Control Trials Case: pay scheme offered to a subset of inspectors selected randomly. Out of the 482 inspectors, 218 randomly assigned to the treatment group, the rest (264) to control group No pre-treatment difference between control and treatment as only reason that explains assignment to one of the groups is chance Comparison of treatment and control group gives a causal impact only difference is one group receives treatment, the other does not

Treatment and control group balance All key variables are balanced at baseline That is: the difference between control and treatment is zero before the intervention starts This happens because of randomization

RCT causal impact Impact can be attributed to the intervention Benchmark to assess other methods Method Treated Control Difference in % RCT $75,611 $68,738 $6,874 10% Problem: Implementation of experiments External validity

Summary of impacts so far Method Treated Control/Comparison Difference in % Participants - Non-participants $93,827 $70,800 $23,027 33% Before - After $72,175 $21,653 30% Difference-in-differences 1 (P1-NP1) (P0-NP0) $3,347 $19,590 29% Difference-in-differences 2 (P1-P0) $21,652 (NP1-NP0) $2,062 Regression Discontinuity (RD) $80,215 $69,753 $10,463 15% RCT $75,611 $68,738 $6,874 10% Different methods: quite different results RCT is the benchmark Other methods can be vastly wrong RD close to RCT

Testing other schemes 3 versions of performance pay incentive were tested: “Revenue” scheme provided incentives solely on revenue collected above a benchmark predicted from historical data. “Revenue Plus” under which adjustments for whether teams ranked in the top, middle, or bottom third (based on an independent survey of taxpayers) were applied “Flexible Bonus” under which rewards both on the basis of pre-specified criteria set by the tax department and on subjective adjustments based on period-end overall performance (based on subjective assessment by managers of the tax units). Treatment Control Difference % RCT "Revenue Incentive" $75,611 $68,738 6,874 10% RCT "Revenue Plus" $72,174 3,437 5% RCT "Flexible Bonus" $69,425 687 1%

Experiments If experiments are not possible choose methods that are still valid Before-After Participants-Non-participants RD Diff-in-Diff Multiple treatments X X

Thank You! WEB http://dime.worldbank.org facebook.com/ieKnow #impacteval blogs.worldbank.org/impactevaluations microdata.worldbank.org/index.php/catalog/impact_evaluation