1. AADAPT Workshop Latin America Brasilia, November 16-20, 2009 Non-Experimental Methods Florence Kondylis

2. Aim: We want to isolate the causal effect of our interventions on our outcomes of interest  Use rigorous evaluation methods to answer our operational questions  Randomizing the assignment to treatment is the “gold standard” methodology (simple, precise, cheap)  What if we really, really (really??) cannot use it?! >> Where it makes sense, resort to non-experimental methods

3.  Find a plausible counterfactual  Every non-experimental method is associated with a set of assumptions  The stronger the assumption, the more doubtful our measure of the causal effect ▪ Question our assumptions ▪ Reality check, resort to common sense! 3

4.  Principal Objective ▪ Increase maize production  Intervention ▪ Fertilizer Vouchers distribution ▪ Non-random assignment  Target group ▪ Maize producers, land over 1 Ha & under 5 Ha  Main result indicator ▪ Maize yield 4

5. 5 (+) Impact of the program (+) Impact of external factors

6. 6 (+) BIASED Measure of the program impact “Before-After” doesn’t deliver results we can believe in!

7. 7 « After » difference btw participants and non-participants « Before» difference btw participants and nonparticipants >> What’s the impact of our intervention?

8. Counterfactual: 2 Options 1. Non-participant maize yield after the intervention, accounting for the “before” difference between participants/nonparticipants (the initial gap between groups) 2. Participant maize yield before the intervention, accounting for the “before/after” difference for nonparticipants (the influence of external factors)  1 and 2 are equivalent 8

9. Underlying assumption: Without the intervention, maize yield for participants and non participants’ would have followed the same trend >> Graphic intuition coming…

10. 10

11. 11

12. 12 NP 2008 -NP 2007 =0.8 Impact = (P 2008 -P 2007 ) -(NP 2008 -NP 2007 ) = 0.6 – 0.8 = -0.2 Impact = (P 2008 -P 2007 ) -(NP 2008 -NP 2007 ) = 0.6 – 0.8 = -0.2 P 2008 -P 2007 =0.6

13. 13 P-NP 2008 =0.5 Impact = (P-NP) 2008 -(P-NP) 2007 = 0.5 - 0.7 = -0.2 Impact = (P-NP) 2008 -(P-NP) 2007 = 0.5 - 0.7 = -0.2 P-NP 2007 =0.7

14. Impact=-0.2

15.  Negative Impact:  Very counter-intuitive: Increased input use should increase yield once external factors are accounted for!  Assumption of same trend very strong  2 groups were, in 2007, producing at very different levels ➤ Question the underlying assumption of same trend! ➤ When possible, test assumption of same trend with data from previous years

16. >> Reject counterfactual assumption of same trends !

17. 17

18. 18 NP 08 -NP 07 =0.2 Impact = (P 2008 -P 2007 ) -(NP 2008 -NP 2007 ) = 0.6 – 0.2 = + 0.4 Impact = (P 2008 -P 2007 ) -(NP 2008 -NP 2007 ) = 0.6 – 0.2 = + 0.4

19. Impact = +0.4

20.  Positive Impact:  More intuitive  Is the assumption of same trend reasonable? ➤ Still need to question the counterfactual assumption of same trends ! ➤ Use data from previous years

21. >>Seems reasonable to accept counterfactual assumption of same trend ?!

22.  Assuming same trend is often problematic  No data to test the assumption  Even if trends are similar in the past… ▪ Where they always similar (or are we lucky)? ▪ More importantly, will they always be similar? ▪ Example: Other project intervenes in our nonparticipant villages…

23.  What to do? >> Be descriptive!  Check similarity in observable characteristics ▪ If not similar along observables, chances are trends will differ in unpredictable ways >> Still, we cannot check what we cannot see… And unobservable characteristics might matter more than observable (ability, motivation, etc)

24. Match participants with non-participants on the basis of observable characteristics Counterfactual:  Matched comparison group  Each program participant is paired with one or more similar non-participant(s) based on observable characteristics >> On average, participants and nonparticipants share the same observable characteristics (by construction)  Estimate the effect of our intervention by using difference-in-differences 24

25. Underlying counterfactual assumptions  After matching, there are no differences between participants and nonparticipants in terms of unobservable characteristics AND/OR  Unobservable characteristics do not affect the assignment to the treatment, nor the outcomes of interest

26.  Design a control group by establishing close matches in terms of observable characteristics  Carefully select variables along which to match participants to their control group  So that we only retain ▪ Treatment Group: Participants that could find a match ▪ Control Group: Non-participants similar enough to the participants >> We trim out a portion of our treatment group!

27.  In most cases, we cannot match everyone  Need to understand who is left out  Example Score Nonparticipants Participants Matched Individuals Wealth Portion of treatment group trimmed out

28.  Advantage of the matching method  Does not require randomization 28

29.  Disadvantages:  Underlying counterfactual assumption is not plausible in all contexts, hard to test ▪ Use common sense, be descriptive  Requires very high quality data: ▪ Need to control for all factors that influence program placement/outcome of choice  Requires significantly large sample size to generate comparison group  Cannot always match everyone… 29

30.  Randomized-Controlled-Trials require minimal assumptions and procure intuitive estimates (sample means!)  Non-experimental methods require assumptions that must be carefully tested  More data-intensive  Not always testable  Get creative: Mix-and-match types of methods! 30

31. 31