1. Quasi-Experimental Methods
Florence Kondylis (World Bank)
This presentation draws on previous presentations by Markus Goldstein, Leandre Bassole, and Alberto Martini

2. Objective Find a plausible counterfactual
Reality check
Every method is associated with an assumption
The stronger the assumption the more we need to worry about the causal effect
Question your assumptions

3. Program to evaluate Fertilizer vouchers Program (2007-08)
Main Objective
Increase maize production
Intervention: vouchers distribution
Target group:
Maize producers
Farmers owning >1 Ha, <3 Ha land
Indicator: Yield (Maize)

4. I. Before-after identification strategy
Counterfactual:
Yield before program started
EFFECT = After minus Before
Counterfactual assumption:
There is no other factor than the vouchers affecting yield from 2007 to 2008
years

5. Year
Number of farmers
Maize Production
(T per Ha)
2007
5000
1.3
2008
2.1
Difference
+0.8

6. Questioning the counterfactual assumption
Question: what else might have happened in to affect maize yield ?

7. Examine assumption with prior data
Year
Number of farmers
Maize Production
(T per Ha)
2006
5000
1.5
2007
1.3
2008
2.1
Assumption of no change over time not so great ! >> There are external factors (rainfall, pests…)

8. II. Non-participant identification strategy
Counterfactual:
Rate of pregnancy among non-participants
Counterfactual assumption:
Without vouchers, participants would as
productive as non-participants in a given year

9. Group
Number of farmers
Maize Production
in 2008 (T per Ha)
Participants
5000
2.1
Non-participants
1.5
Difference
+0.6

10. Questioning the counterfactual assumption
Question: how might participants differ from non-participants?

11. Test assumption with pre-program data
REJECT counterfactual hypothesis of same productivity

12. III. Difference-in-Difference identification strategy
Counterfactual:
Non-participant maize yield, purging pre-program differences between participants/nonparticipants
“Before vouchers” maize yield, purging before-after change for nonparticipants (external factors)
1 and 2 are equivalent

13. Average maize yield
(T / Ha)
2006
2008
Difference ( )
Participants (P)
1.5
2.1
-0.6
Non-participants (NP)
0.5
1.3
-0.8
Difference (P-NP)
1.0
0.8
0.2

14. Effect = 3.47 – =
Participants
66.37 – = 3.47
= 11.13
Non-participants

15. Effect = 8.87 – 16.53 = - 7.66 Before After 66.37 – 57.50 = 8.87
62.90 – = 16.53
After

16. Counterfactual assumption:
Without intervention participants and nonparticipants’ pregnancy rates follow same trends

17. 74.0
16.5

18. 74.0
-7.6

19. Questioning the assumption
Why might participants’ trends differ from that of nonparticipants?

20. Examine assumption with pre-program data
Average rate of teen pregnancy in
2004
2008
Difference ( )
Participants (P)
54.96
62.90
7.94
Non-participants (NP)
39.96
46.37
6.41
Difference (P=NP)
15.00
16.53
+1.53 ?
Or with other outcomes not affected by the intervention: household consumption
counterfactual hypothesis of same trends doesn’t look so believable

21. IV. Matching with Difference-in-Difference identification strategy
Counterfactual:
Comparison group is constructed by pairing each program participant with a “similar” nonparticipant using larger dataset – creating a control group from similar (in observable ways) non-participants

22. Counterfactual assumption:
Unobserved characteristics do not affect outcomes of interest
Unobserved = things we cannot measure (e.g. ability) or things we left out of the dataset
Question: how might participants differ from matched nonparticipants?

23. Matched nonparticipant
73.36
Effect =
66.37
Matched nonparticipant
Participant

24. Can only test assumption with experimental data
Studies that compare both methods (because they have experimental data) find that:
unobservables often matter!
direction of bias is unpredictable!
Apply with care – think very hard about unobservables

25. Summary
Randomization requires minimal assumptions needed and procures intuitive estimates (sample means !)
Non-experimental requires assumptions that must be carefully assessed
More data-intensive

26. Example: Irrigation for rice producers + Enhanced Market Access
Impact of interest measured by:
Input use & repayment of irrigation fee
Rice yield
(Cash) income from rice
Non-rice cash income (spillovers to other value chains)
Data: 500 farmers in project area / 500 random sample farmers
Before & after treatment
Can’t randomize irrigation so what is the counterfactual?

27. Plausible counterfactuals
Random sample difference in difference
Are farmers outside the scheme on the same trajectory ?
Farmers in the vicinity of the scheme but not included in scheme
Selection of project area needs to be carefully documented (elevation…)
Proximity implies “just-outside farmers” might also benefit from enhanced market linkages
What do we want to measure?
Propensity score matching
Unobservables determining on-farm productivity ?

28. Thank You