1. Research Department Inter-American Development Bank
Beyond Beneficiaries: The Use of Information Systems for Cost-Effective Evaluation
Suzanne Duryea
Research Department Inter-American Development Bank
December 10, 2003

2. Information Systems
Information systems such as SISBEN, CAS, SIPO were developed to better target beneficiaries
(predominantly the poor and at-risk populations).
A quantitative score is used as a proxy for poverty and/or the lack of income generating resources within the family. Families scoring below a threshold are eligible to participate in certain programs.

3. Information Systems for Social Programs
Can facilitate certain types of evaluations
With some complementary additions: the databases can form a stronger foundation for evaluation

4. What do we mean by “impact evaluation?”
“What would have happened to the beneficiaries in the absence of the program?”
“What was the effect of the program?”

5. Some options 1) Randomized Designs Examples:
a) Geographic regions are randomly selected for later phase-in of a program (Oportunidades, PRAF)
b) Individuals are randomly selected to form a control group (such as school vouchers Colombia)
Randomized evaluations are viewed by the experts as the best option because they solve many of the problems encountered in evaluation. (Duflo and Kremer, 2003)
However, randomization can be politically difficult and is not always possible or appropriate.
Randomized evaluations are not the focus of today’s talk.

6. Some options 2) Ex-Post Evaluation
Aim is to compare outcomes of beneficiaries and non-beneficiaries who were similar BEFORE program participation
Only observe individuals at one point in time -- after enough time has passed for program to have an effect.
Critical assumption: After controlling for observed characteristics, beneficiaries and non-beneficiaries do not differ in unobserved characteristics
Information Systems can help in this case.

7. Some options 3) Before and After Comparisons -- “Double Difference”
Monitor the beneficiaries and non-beneficiaries over the length of time necessary for the program to have effects
I.e., Take baselines for both groups then compare the change in the beneficiaries’ behavior to the change in the non-beneficiaries’ behavior. All time invariant unobservable differences are removed.
Critical assumption: No time varying unobserved differences between beneficiaries and non-beneficiaries.
Information Systems can help in this case.

8. Creation of Counterfactual Group
The most critical part of evaluation design: who can be used as a natural comparison
Information Systems Can Plan an Important Role
There are families who will not quality for programs because they are slightly above the eligible score (threshold). These families look very similar to families with similar scores who have qualified. Discontinuity Design: (Campbell, 1969, Buddlemeyer and Skoufias 2003)
Assign program and then compare outcomes across the families who are close to the threshold.
Information Systems can help in this case.

9. Important Caveats
This is a local average treatment effect: not necessarily comparing the effect on the poorest families
If there are heterogeneous effects at different levels of scores this is only measuring the effect at the cutoff score
Threshold must exist: enforcement of rules necessary.
Information Systems can help in this case.

10. Example of an ex-post evaluation: Superémonos
S. Duryea and A. Morrison (2003)
Relied heavily on information systems from
Instituto Mixto de Ayuda Social (IMAS), Costa Rica:
Sistema de Información sobre la Población Objetivo (SIPO)
Sistema de Atención a Beneficiarios (SAB)

11. Superémonos Food coupon (U.S. $30 per month during school)
Targets poor households with school age kids (ages 6 to 18) at risk for poor school attainment using SIPO (proxy means test)
Conditional transfer: families agree that all children will regularly attend school

12. Sistema de Información sobre la Población Objetivo (SIPO)
Targeting mechanism
SIPO score depends on
Occupation of household head
Material used in house construction
Household income
Education of household head
Net household wealth
Over 250,000 households
Sistema de Atención a Beneficiarios (SAB)* for beneficiaries of IMAS

13. SIPO and SAB Very efficient
We provided IMAS with a list of characteristics (requested 3 regions, ages 10-16, started program in 2001) and they provided a cross-referenced list of beneficiaries within minutes
Were in the field with our survey within two months
Very fast in contrast to adding questions to a national household survey

14. First however: we were missing a critical component for evaluation
No information was available in SAB regarding potential counterfactual group (those just above the threshhold)
We formed a counterfactual group by conducting surveys in the same neighborhoods and getting families with similar probabilities of participating in Superemonos (propensity matching approach)
Cost of data collection for evaluation of
Superémonos households under $30,000
Mexico Progresa 24,407 households $450,000
Argentina Trabajar 2,800 households $350,000
Source: Blomquist 2003

15. Summary Information on Survey Samples
Superemonos-
IMAS list
Not from list
number of total observations
746
1032
average age of child
12.90
12.95
percentage female
50.27
48.55
percentage in San Jose
60.19
56.88
percentage in Alajuela
6.03
4.94
percentage in Cartago
33.78
38.18
percentage of mothers with
incomplete primary education
36.46
35.76
percentage of households
lacking working electricity
4.29
3.88
If the two groups were drawn from identical distributions of observables and unobservables we could compare means across the groups.
They are similar but not identical. So we use two methodologies:
1) standard parametric regressions: advantages – using more observations (more likely to have significant results), very robust to choice of explanatory variables (in this case) disadvantages - methodology using disparate observations (not in dense parts of the sample)
2) propensity score matching methods: advantage: non-parametric, and matching based on a priori probability of program participation
Disadvantage: complex, throws out poor matches so reduces sample size, loss of significance, tends to be highly sensative to variables used in participation equations

16. Survey design and implementation
Tailor-made survey
Pilot tested
Power tested
Sample size:
746 beneficiaries of Superémonos
1,042 non-beneficiaries
Data collected in 3 urban centers: San Jose, Alajuela and Cartago
Collected information on labor force participation, school attendance, school performance, and a host of household characteristics
Approx. cost: $15 per survey (no addresses)
Cost of $15 per survey

17. Duryea Morrison Strategy: Ex-Post Evaluation Information System Used to Identify Beneficiaries Only
We found that beneficiaries ages were 5 percentage points more likely to attend school than non-beneficiaries.
In this program we have no reason to think there are serious selection problems. Academic record not considered for eligibility. Ex-post methodology may be appropriate.

18. Alternative Strategy: What we might have done Double Difference: Information Systems Used to Identify Treatment and Control Groups
Create comparison group from those who fall just above the threshold on SIPO for Superemonos participation
How: Match 3 non-beneficiaries to each beneficiary (based on SIPO score within same geographic unit)
Follow beneficiaries and non-beneficiaries over time (double difference estimation)
Suppose something unobserved is correlated with outcome. Suppose in another program the children in the program spend less time on school activities because they are more likely to be working in the entrepreneurial sectors with their parents. Suppose these hard-working parents are also more likely to be signed up with the program – because of knowledge or influence. Then with an ex-post evaluation design you might see the following.

19. A B C Hypothetical Example:
Effect of Conditional Transfer Program on Hours of School Activities
Double Difference Measures Positive Effect of Program
Ex-Post Measure Misses Positive Effect of Program 40 A
Double Difference = C - B = = 9 35
amount of change by beneficiaries is larger B 30
A = Ex-
Post
Non-
Difference 25
Beneficiaries C
Hours Spent on School Activities 20 15 10
Beneficiaries 5
Before
After 1 2 3 4 5 6 7 8 9
Months of Execution of Project

20. Policy Conclusions and Recommendations
Information systems are critical, both for targeting as well as for evaluation.
Information systems can facilitate cost-effective ex-post evaluations.
Possible to modify information systems to provide more rigorous evaluations at reasonably low cost. Monitoring non-beneficiaries in addition to beneficiaries is the most important modification. Part of evaluation strategy for Chile Solidario.
Encourage stricter enforcement of the implementation of scoring thresholds.