1. The PNPM-Generasi Project One-Year Impact Evaluation Preliminary Findings Presented by: Susan Wong, EASER, The World Bank Ben Olken, M.I.T. Department of Economics
November 5, 2009

2. Structure of Today’s Presentation
Indonesian context
Description of PNPM-Generasi project design
PNPM-Generasi implementation update
Impact evaluation design and analysis
Preliminary findings of one-year PNPM-Generasi implementation
Impact on health indicators
Impact on education indicators
Effect of community incentives
Heterogeneity
Conclusion, Further Analysis, and Next Steps

3. Indonesian Context
Remarkable progress in poverty reduction and key human development indicators over past few decades.
Economic growth, increased access to education and health services, expanded anti-poverty programs, and improvements in infrastructure have all helped to reduce poverty.
Poverty headcount is 14.1% in 2009.
However, 32.5 m Indonesians still live below poverty line & one-half of all HHs remain clustered around national poverty line. High vulnerability.
Regional disparities with Eastern Indonesia lagging behind other parts of country, esp. Java.
Human development areas require more attention: child malnutrition, infant and maternal mortality, primary to secondary school transition, access to safe water and sanitation. Quality of services also a major concern.

4. Two Pilot Projects In 2007, GoI started two pilot projects:
Household CCT – the traditional model
Quarterly tranches of cash transfers
Statistically identified 633,000 poor households with children
Currently in 13 provinces, 70 districts, 629 municipalities
Annual budget of IDR 1.2 trillion 120 m)
PNPM-Generasi Community Block Grants
Addresses the same health and education indicators, but at the community level
5 provinces, 21 districts, 178 subdistricts
Covering approx. 3.1 million beneficiaries
Total budget from m

5. WB Support Role
Collaboration between PREM, HD, and Social Development Units in Indonesia
Provide TA for design, implementation and evaluations of two pilots.
Portion of KDP/PNPM WB loan funds support the PNPM Generasi pilot in 5 provinces.

6. The PNPM-Generasi Project
Objectives: Accelerate the achievement of MDGs
Reduce child mortality
Reduce maternal mortality, and
Ensure universal coverage of basic education
Conditionalities: Places incentives on communities to identify problems and seek solutions to improving 12 health and education indicators

7. The PNPM-Generasi Project
Community incentives:
Version A: 20% of year 2 allocation depends on previous year’s village performance
Version B: Village performance not linked to fund allocation. Otherwise identical to Version A.
Implemented through KDP/PNPM-Rural with:
Same management structures at the national, provincial, and district levels as PNPM-Rural
Facilitated by 2 subdistrict facilitators 7

8. PNPM-Generasi Project Design
12 indicators: communities are required to work on the same indicators as HH-CCT (Program Keluarga Harapan)
Health:
Four prenatal care visits during pregnancy
Taking iron tablets during pregnancy
Delivery assisted by trained professional
Two postnatal care visits
Complete childhood immunization
Ensuring monthly weight increases for infants
Regular weighing for under-fives
Taking Vitamin A twice a year for under-fives
Education:
Primary school enrolment (7-12 year olds)
Regular primary school attendance >85%
Junior secondary school enrolment (13-15 year olds)
Regular secondary school attendance >85%

9. PNPM-Generasi Design

10. PNPM-Generasi Project Implementation
Geographical coverage:
178 subdistricts in 21 districts, five provinces
Approx 3.1 million beneficiaries
Block grant amounts:
2007 average per village amount USD 8,400
2008 average per village amount USD 11,600
2009 average expected per village amount USD 14,400
Timeframe:
First block grant disbursed to villages in Oct-Dec 2007
Second year disbursement to villages in Oct-Dec 2008
Third year disbursement to villages in Oct-Dec 2009

11. Village Fund Allocation in 2007
56% of block grants on education:
School materials, equipment and uniforms (59%)
Financial assistance and school fees (31%)
Infrastructure (satellite classrooms and access roads) (5%)
Financial incentives for honorarium teachers (4%)
Training and behavior change communication (1%)
44% of block grants on health activities:
Supplementary feeding activities (40%)
Financial assistance for pregnant mothers to use services (30%)
Infrastructure (13%)
Facilities and equipment (11%)
Training and behavior change communication (3%)
Incentives for health workers (3%)

12. Impact Evaluation Design
Uses a randomized evaluation
Subdistricts allocated by lottery into three groups:
with performance incentives,
without performance incentives, and
controls
Subdistrict level randomization addresses spillovers and crowding out

13. Impact Evaluation Design
Three rounds of surveys:
- Baseline/Wave I (2007): PNPM-Generasi & PKH (HH CCT)
Wave II (2008): PNPM-Generasi only
Wave III (scheduled to begin Nov 2009): PNPM-Generasi & PKH
Survey design:
12,000 households per wave spread over 300 subdistricts including
Anthropometric measurements of children <3
Math and Indonesian tests administered to school-aged children (Baseline and Wave III)
School and health provider interviews to track supply-side effects
Qualitative studies to understand bottlenecks in use and provision of services (Baseline and Wave III)

14. Impact Evaluation Design
Sample size per survey for PNPM-Generasi:
33,000 total respondents
- 12,000 households
10,800 married women in reproductive age
4,850 pregnancies (2 years prior to the survey)
9,500 school-aged children (7-15 years old)
4,750 children under-three
2,313 villages
300 subdistrict health centers (puskesmas)
1,157 midwives
2,391 village health posts (posyandu) (Waves II & III only)
847 junior secondary schools
1,065 primary schools (Waves II & III only)

15. Mid-Term Impact Analysis
Regressions run for:
PNPM-Generasi vs No PNPM-Generasi
Incentivized (version A) vs Non-Incentivized (version B)
Regression specifications:
Uses baseline data as control variables
Subdistrict average
Individual baseline values for panel respondents (0 for non-panel)
District fixed effects
Province * previous KDP experience fixed effects
HH sampling category dummies

16. Overview of Preliminary Findings
Substantial improvements in health indicators
No impact on education indicators
Performance-based incentives lead to consistently better outcomes
Substantial regional heterogeneity with strongest improvements in Sulawesi
Provider effort, especially for midwives in incentivized locations, increased substantially.
Greater community engagement particularly through service provision at the village health posts. 16

17. Preliminary Findings: Health
Strongest improvements on health services coverage:
Participation in growth monitoring
Deliveries assisted by doctors or midwives, particularly in Java and Sulawesi
Large increase in village health post participation
Long-term health outcomes:
Large reductions in neonatal and infant mortality (although some small differences noted at baseline)
Some reductions in malnutrition (<2SD weight-for-age) among children under-three in NTT and Sulawesi

18. Notes on the figures Each bar represents the percentage
change in the indicator in Generasi
treatment areas compared to control
areas
Key messages of the figure
Bars in
patterns
are not
statistically
significant
Control avg.
7.52 visits
1.975 sachets
75%
2.91 visits
70%
2.18 times
1.62 Vit A
21%
Control group
averages
Bars in solid
colors are
statistically
significant
Zero percent is the control group average. Bars to the
right indicate increase while bars to the left indicate reduction. 18

19. Control avg.
7.52 visits
1.975 sachets
75%
2.91 visits
70%
2.18 times
1.62 Vit A
21%

20. Large increase in Village Health Post participation :
Numbers of children weighed; receiving supplementary feeding; immunized; receiving Vit A
Numbers of pregnant mothers receiving antenatal care; iron pills
No increase in non-targeted village health post services
Control avg.
41.21 children
34.99 children
11.85 children
4.54 mothers
4.79 mothers
44.61 children
2.83 mothers 20

21. Reduction in neonatal mortality and infant mortality (although some small differences noted at baseline)
Similarly large infant mortality reductions found in other randomized community health programs in Uganda and Bangladesh
Control avg.
5.9%
20.8%
34.3%
18 deaths per 1000 live births
8 deaths per 1000 live births

22. Preliminary Findings: Education
No overall project impact on education
Negative impact on enrollment and attendance of year olds, primarily among those year olds who would have been in primary
No impact on primary or jr. secondary net enrollment
Overall jr. secondary school enrollments increased in both treatment and control over this period

23. Control avg.
93.7%
89.1%
97.9%
90.7%
66.0%
24.8%
97.8% 23

24. Some Hypotheses on Education Findings
Primary school enrollment already high at 95%.
Jr. secondary gross enrollment increased overall in treatment and control areas.
Great deal of fluctuation over last couple of years.
Generasi targets only years old for jr. secondary, so communities may have interpreted this age conditionality strictly.
Program missed the school enrollment period of June-July.
Communities seemed to be favoring more assistance towards children already in school vs. focusing on out-of-school children. “Help the greatest number vs. the few” mentality.

25. Preliminary Findings: Community Incentives
Version A, incentivized version outperformed non-incentivized version in improving health service coverage
Prenatal care
Postnatal care
Growth monitoring
Version A had larger impact on long-term health outcomes:
Acute morbidity (ARI or diarrhea)
Malnutrition
Version A was more effective in increasing midwives’ work efforts in:
Outreach services
Time spent seeing patients in their public capacity (reduced time for private practice)

26. Incentivized version outperformed non-incentivized in improving health service coverage
Control avg.
7.52 visits
1.975 sachets
74.6%
2.91times
2.18 times
1.63 times
69.5%
20.8% 26

27. Incentivized version outperformed non-incentivized in improving health outcomes
Control avg.
5.9%
20.8%
34.3%
18 deaths per 1000 live births
8 deaths per 1000 live births 27

28. Preliminary Findings: Heterogeneity
Regional heterogeneity
Sulawesi - strongest and consistent effects
Java - some positive impacts
NTT – smallest effects

29. Sulawesi - strongest and consistent effects
Java - some positive impacts
NTT – smallest effects
All
NTT
Sulawesi
Java

30. Incentivized version had larger impact on average health outcomes overall and in NTT30

31. Negative impact on education largely seen in Sulawesi
All
NTT
Sulawesi
Java

32. Conclusions PNPM-Generasi has:
Improved health service coverage mainly through increased village health post (posyandu) participation
Reduced infant mortality, acute morbidity and malnutrition
Increased number of hours midwives spend on outreach and services in their public capacity
PNPM-Generasi has not yet improved formal education indicators
Community incentives ensures better outcomes with the same project funds and design
Certain heterogeneity in outcomes were observed:
Regional: Sulawesi strongest, positive in Java, small in NTT
Increased community participation and engagement, especially through service provision at village health posts. 32

33. Future Analysis Cost-benefit analysis
Where/for whom PNPM-Generasi works best
Spillovers
Details of community incentives
Changes in prices
Targeting
What communities spent their funds on 33

34. Next Steps
PNPM-Generasi will likely expand in 2010 to an additional 1-2 provinces.
Adjustment of some of the education indicators
Conduct wave III survey in 660 kecamatan jointly with HH CCT (Nov 2009– Jan 2010)
Finalize operations paper on lessons learned thus far 34