1. Impact Evaluation Methods
July 9, 2011
Dhaka
Sharon Barnhardt, Assistant Professor
Institute for Financial Management and Research (IFMR)

2. Impact evaluation methods
Non- or Quasi-Experimental Methods
a. Pre-Post
Simple Difference
Differences-in-Differences
Multivariate Regression
Statistical Matching
Interrupted Time Series
Instrumental Variables
Regression Discontinuity

3. Methods to estimate impacts
Let’s look at different ways of estimating the impacts using the data from the schools that got a balsakhi
Pre – Post (Before vs. After)
Simple difference
Difference-in-difference
Simple or Multivariate regression
Randomized Experiment
Program participants before program
Individuals who did not participate (data collected after program)
Same as above, plus: (data collected before and after)
Same as above plus: Also have additional “explanatory” variables

4. 1 - Pre-post (Before vs. After)
Average change in the outcome of interest
before and after the programme
Example:
when establishing a causal link is not feasible
measuring the impact of a govt. run literacy campaign state-wide; where it is difficult to construct a comparison group since everyone in the state is mandated to receive the “treatment” at the same time (Sakshar Bharat Campaign in India)
Issues:
does not take into account time-trend
“response-shift bias” – change in the participant’s metric for answering questions from the pre to the post test
Response shift bias is described as a “change in the participant’s metric for
answering questions from the pre test to the post test due to a new understanding of a concept
being taught (Klatt and Taylor-Powell, 2005).”
For example, if a program is designed to teach parents effective discipline for children
and some participants don’t know that spanking a child is considered inappropriate, they may
report at the beginning of the program (pre test) that they are engaging in age appropriate
discipline practices at home. However, if over the course of the program they begin to
understand the relationship between age of a child and the “appropriate” discipline techniques,
they may also report that they understand age appropriate discipline on the post testt – and thus,
their evaluation data would reflect no program impact at all.

5. 1 - Pre-post (Before vs. After)
QUESTION: Under what conditions can this difference (26.42) be interpreted as the impact of the balsakhi program?
Average post-test score for children with a balsakhi
51.22
Average pretest score for children with a balsakhi
24.80
Difference
26.42

6. Pre-post Method 1: Before vs. After Impact = 26.42 points? 75 50 25
26.42 points?

7. 2 - Simple difference Example: Issues:
A post- programme comparison of outcomes between
the group that received the programme and
a “comparison” group that did not
Example:
programme is rolled out in phases leaving a cohort for comparison, even though assignment of treatment is not random
if Sakshar Bharat was rolled out in a few districts only at a time
Issues:
does not take into account differences that exist before the treatment (selection bias)

8. 2 - Simple difference Compare test scores of… With test scores of…
Children who got balsakhi
Children who did not get balsakhi

9. 2 - Simple difference
QUESTION: Under what conditions can this difference (-5.05) be interpreted as the impact of the balsakhi program?
Average score for children with a balsakhi
51.22
Average score for children without a balsakhi
56.27
Difference
-5.05

10. What would have happened without balsakhi?
Method 2: Simple Comparison Impact = points? 75 50 25
-5.05 points?

11. 3 – Difference-in-Differences or Double Difference
Comparison of outcome between
a treatment and comparison group (1st difference) and
before and after the programme (2nd difference)
Suitability:
programme is rolled out in phases leaving a cohort for comparison, even though assignment of treatment is not random
If Sakshar Bharat was rolled out in a few districts only at a time
Issues:
failure of “parallel trend assumption”, i.e. impact of time on both groups is not similar

12. 3 – Difference-in-Differences
Compare gains in test scores of…
With gains in test scores of…
Children who got balsakhi
Children who did not get balsakhi

13. 3 - Difference-in-differences
Pretest
Post-test
Difference
Average score for children with a balsakhi
24.80
51.22
26.42

14. What would have happened without balsakhi?
Method 3: Difference-in-differences 75 50 25
26.42

15. 3 - Difference-in-differences
Pretest
Post-test
Difference
Average score for children with a balsakhi
24.80
51.22
26.42
Average score for children without a balsakhi
36.67
56.27
19.60

16. What would have happened without balsakhi?
Method 3: Difference-in-differences 75 50 25
19.60
6.82 points?
26.42

17. 3 - Difference-in-differences
QUESTION: Under what conditions can 6.82 be interpreted as the impact of the balsakhi program?
Pretest
Post-test
Difference
Average score for children with a balsakhi
24.80
51.22
26.42
Average score for children without a balsakhi
36.67
56.27
19.60
6.82

18. 4- Simple/Multivariate Regression
Change in outcome in the treatment group controlling for observable characteristics
requires theorizing on what observable characteristics may impact the outcome of interest besides the programme
Issues:
how many characteristics can be accounted for? (omission variable bias)
requires a large sample if many factors are to be controlled for

19. 4 - Regression: controlling for pretest

20. Impact of Balsakhi - Summary
Method
Impact Estimate
(1) Pre-post
26.42*
(2) Simple Difference
-5.05*
(3) Difference-in-Difference
6.82*
(4) Regression
1.92
* Statistically significant at the 5% level

21. Impact of Balsakhi - Summary
Method
Impact Estimate
(1) Pre-post
26.42*
(2) Simple Difference
-5.05*
(3) Difference-in-Difference
6.82*
(4) Regression
1.92
(5) Randomized Experiment
5.87*
* Statistically significant at the 5% level

22. Impact of Balsakhi - Summary
Method
Impact Estimate
(1) Pre-post
26.42*
(2) Simple Difference
-5.05*
(3) Difference-in-Difference
6.82*
(4) Regression
1.92
(5) Randomized Experiment
5.87*
*Statistically significant at the 5% level
Bottom Line: Which method we use matters!

23. Another Example: Jaunpur Study
280 villages in Jaunpur, UP
Intervention:
Provided information to communities on education status and responsibilities of VECs
Encouraged to test children and create community report cards to assess status of education
Trained volunteers in villages to conduct after-school reading classes

24. Jaunpur Study Different impact estimates on reading level scores
Method
Impact Estimate
(1) Pre-post
0.6*
(2) Simple Difference
-0.68*
(3) Difference-in-Difference
0.31*
(4) IV Regression
0.88*
(5) Propensity Score Matching
-0.03

25. Jaunpur Study Different impact estimates on reading level scores
Method
Impact Estimate
(1) Pre-post
0.6*
(2) Simple Difference
-0.68*
(3) Difference-in-Difference
0.31*
(4) IV Regression
0.88*
(5) Propensity Score Matching
-0.03
(6) Randomized Experiment
0.06*