Evaluation Team Andrew Beath (Harvard University) Fotini Christia (M.I.T.) Ruben Enikolopov (New Economic School, Moscow) Shahim Kabuli (World Bank) Sakhi Frozish (World Bank)Partners Vulnerability Analysis Unit (MRRD) AfghanAid, CHA, IC, IRC, NPO/RRAA, Oxfam, PiN National Solidarity Program (NSP)Funding National Solidarity Program Food and Agriculture Organisation(FAO) World Bank TFs

National Solidarity Programme Executed by Ministry of Rural Rehabilitation and Development (MRRD) Implemented by 28 Facilitating Partners (local and int’l NGOs) Funded by Multilateral, Bilateral Donors 2003 – 2010: Covered 22,500 communities at a cost of $929 million Phase-I: ; Phase-II: ; Phase-III: 2010+

Create Gender-Balanced Community Development Councils (CDCs) through Secret Ballot, Universal Suffrage Election Fund Projects Selected by CDCs and Villagers and Managed by CDCs (Average Grant: $33,000; Max.: $60,000) Community Development Council (CDC) Projects National Solidarity Programme Two Principal Village-Level Interventions: 6 months18-24 months Water Supply24% Roads & Bridges25% Irrigation18% Electricity13%

What areas does NSP potentially impact? Community Development Council (CDC) Projects Local Governance Access to Services Economic Activity NSP Social Cohesion & ConflictPolitical Attitudes Gender

Local Governance Access to Services Economic Activity The evaluation measures impact of NSP-II on...The evaluation estimates these impacts by... Social Cohesion & Conflict Gender Political Attitudes

Similarity of Treatment and Control Villages Create CDCs Treatment Villages (NSP) Select Sub-Projects Implement Sub-Projects Sub- Projects Finished Control Villages (Non-NSP) May – Oct. 2009Spring 2011 Interim Estimates (18% of Projects Complete at Survey) Final Estimates Baseline Survey Aug. – Sep st Follow- Up Survey 2 nd Follow- Up Survey The evaluation estimates these impacts by... collecting data over 3½ years in 500 villages: 250 NSP (treatment) & 250 non-NSP (control) Structure of Evaluation and Data Collection The evaluation uses this data to...

Baseline Survey 1 st Follow- Up Survey Change in Treatment Villages Change in Control Villages Baseline Survey Treatment Villages (NSP) Control Villages (Non-NSP) 1 st Follow- Up Survey The evaluation uses this data to... compare changes in treatment villages (NSP) with changes in control villages (non-NSP) - - = = - = Impact of NSP = - Estimation of Impacts of NSP Baseline Survey Differences estimation is accurate because... Difference-in-Difference EstimatesDifference Estimates If treatment and control villages are identical at baseline...

Control Group (Non-NSP) Treatment Group (NSP) Differences estimation is accurate because treatment villages selected randomly from 500 surveyed villages - other villages to control group study is a randomized control trial Village AVillage B Partitioned Randomization: FPs denoted 15 villages to be excluded from randomization and evaluation Partitioned Randomization & Contractual Embedding Improved Chance of Successful Randomization Due to funding constraints and lack of village-level data, randomization was fairest way to decide which villages received NSP Randomization embedded in FP contracts

+ 4% Perceptions of Gov’t, Civil Society, and Military + 6% + 7% + 5% + 4% + 3% Male Villager Believes Official Works for the Benefit of All Villagers Political Attitudes Gender Social Cohesion Economic Activity Access to Services Local Governance

Community Development Council (CDC) Election Type Method of Sub-Project Selection Sub-Treatment Interventions (STIs) STIs test different implementation strategies or changes in program design STIs provide real-time evidence-based feedback on how to improve program effectiveness NSP-II impact evaluation incorporates two STIs which test changes in two program components: Test of Two Different Types of CDC Elections

Neighborhood Election Village Election Treatment Village ATreatment Village B 125 Villages 250 Treatment Villages Neighborhood Election: One male & one female from each neighborhood elected to CDC Village Election: Highest male & female vote-getters in entire village elected to CDC → Guarantees representation → Preferred candidates are elected Test of Two Different Types of CDC Elections 250 treatment villages randomly assigned to elect CDC either by neighborhood or village election Effect of Different Types of CDC Elections

60% 85% Neighborhood Election Village Election Village Elections (compared to Neighborhood Elections) increase electoral competitiveness Effect of Different Types of CDC ElectionsTest of Two Different Types of Project Selection... on Electoral Competitiveness

Consultation Meeting Secret Ballot Referendum 125 Villages 250 Treatment Villages Consultation Meeting: Villagers hold meeting and decide collectively which project is best Secret Ballot Referendum: Villagers select projects through selecting preferred project → consensus-based → directly democratic Test of Two Different Types of Project Selection 250 treatment villages randomly assigned to select project either by consultation meeting or secret-ballot referendum Impact of Different Types of Project Selection

Demographic Sub-Project Selection Method Stage of Sub-Project Selection ProposalSelectionPrioritization Male Villagers Meeting ~++ Referendum ~++ Village Leaders Meeting +++ Referendum ~~~ Female Villages Meeting ~~~ Referendum ~~~ Preferences of male villagers influence selection and prioritization in both meetings and referenda Preferences of village leaders influence proposals, selection, and prioritization in meetings, but not referenda Preferences of female villages do not influence proposals, selection, or prioritization Preferences of village leaders more able to influence selection in consultation meetings (but influence of villagers in affecting selection is not affected) Baseline Survey asked male villagers, female villagers, and village leaders which village projects they preferred Obtained information from FPs on which projects were proposed, selected, and prioritized for implementation Compared projects preferred by different groups with projects proposed, selected, and prioritized to find out who is influential in selection process Compare influence of different groups under two selection processes to find out how selection type affects elite capture Impact of Different Types of Project SelectionTest of Election / Project Selection Interactions... on Elite Capture of Project Selection

250 Treatment Villages Consultation Meeting Secret Ballot Referendum 125 Villages 250 Treatment Villages Neighborhood Election Village Election 125 Villages 250 Treatment Villages 62 Villages 63 Villages 62 Villages Neighborhood Election & Referendum Village Election & Consultation Meeting Village Election & Referendum Randomization of Election Type Randomization of Project Selection Type Randomization of Election Type and Project Selection Type Done Separately Four Randomly-Assigned Combinations of Election and Selection Type Test of Election / Project Selection Interactions Use to Test Interactions between Election and Selection Type Neighborhood Election & Consultation Meeting

InstrumentCombinationProposalSelectionPrioritization Male Villager Neighborhood Election / Meeting ~++ Neighborhood Election / Referendum ~+~ Village Election / Meeting ~+~ Village Election / Referendum ~~+ Difference between Types ~~~ Village Leaders Neighborhood Election / Meeting ~~~ Neighborhood Election / Referendum ~~~ Village Election / Meeting +++ Village Election / Referendum ~~~ Difference between Types +++ Male villagers influence selection and prioritization Combinations of election and referendum type do not affect influence of male villagers Impact of Combinations on Selection Outcomes Villager leaders influence proposals, selection, and prioritization when village elections are combined with consultation meetings Probability of Elite Capture Maximized by Combination of Village Elections with Consultation Meetings... on Elite Capture of Project Selection

Problems with Solutions CDD is interesting b/c it can change behaviors, attitudes, institutions, social cohesion etc. Very difficult to measure well ‘Parrot Bias’ in Surveys: Does program really change attitudes and behaviors or does it just responses to survey questions? → measure actual behaviors rather than simply asking questions Randomization: theoretically easy, practically difficult → tweak procedure to be resistant to pressures & communicate → successful quasi-experimental eval. better than failed RCT Prospective Evaluations take ages... → STIs can provide (quicker) real-time feedback to program → invest in extensive pilot-testing and consult widely → Manage expectations and communicate...

Issues That Remain Independence of Researchers vs. Accountability of Evaluations to Programs, Implementers, and Donors Importance of Non-Results vs. Lack of Interest and Lack of Incentives for Researchers or Program to Disseminate Small proportion of programs are subjected to prospective IEs. How do we ensure the evaluated ones aren’t punished for non- or bad results? Who will synthesize IE results for policy-makers and make sure the results are used in policy decisions and program design? Who will coordinate researchers to make sure evaluations address questions useful for policy decisions and program design?