2. 3ie  Context: the results agenda  The call for evidence  Rigorous impact evaluation  The international response  Reactions

3. 3ie  Methodological developments  Results agenda › US: Government Results and Performance Act, GRPA -1993 – USAID adopted six ‘strategic development goals’ e.g. broad-based economic growth, defining outcome indicators for each e.g. per capita growth › UK: Public Service Agreements and Service Delivery Agreements (very similar to Report Cards and Development Indicators in South Africa)  MDGs › Focus on outcomes › Raises questions about attribution

4. 3ie Goals were ‘so broad, and progress affected by many factors other than USAID programmes, that the indicators cannot realistically serve as measures of the agency’s specific efforts ’ In 2000 USAID abandoned the goal outcome measures as a basis for monitoring its performance

5. 3ie  Timescale, exacerbated by data lags  Decentralized programs not aligned with higher-level objectives  Data quality  Impact? Development agencies may not know exactly what impact their efforts are having given the wide range of other agencies and external political, economic and social factors involved.

6. 3ie  Use existing data systems, but with a view to data quality and timeliness  Invest in improving existing data systems rather than making new (often parallel) ones  Know the advantages and limitations of different sorts of data (administrative versus survey data)  Avoid indicator proliferation  Don’t be all M and no E

7. 3ie Outcome monitoring is not a valid basis for rigorous performance measurement … hence the need for impact evaluation

8. 3ie  International agency response › World Bank: DIME and IEG › Inter-American Development Bank*  At country level response strongest in Latin America › Progressa › In Mexico and Colombia social programs legally require impact evaluations to secure funding Note: * See IDS Bulletin March 2008 for discussion

9. 3ie After decades in which development agencies have disbursed billions of dollars for social programs, and developing country governments and nongovernmental organizations (NGOs) have spent hundreds of billions more, it is deeply disappointing to recognize that we know relatively little about the net impact of most of these social programs.

10. 3ie  UNICEF review (1995) found 15% had ‘impact assessments’ but ‘many evaluations were unable to properly assess impact because of methodological shortcomings’  ILO review of 127 studies of 258 community health programs found only 2 which could give robust conclusion on impact on access  Recent (2008) NORAD review found many reports draw impact conclusions on basis of weak evidence (or none at all)

11. 3ie Billions of dollars are being spent with no idea if they are the best intervention to achieve the desired outcome – or even if the help achieve it at all. Rigorous impact evaluation – and only rigorous impact evaluation – will provide the evidence needed for optimal resource allocation

12. 3ie  When will we ever learn? › An approach which can attribute change in outcomes to the intervention › Requires a well-defined control group › Strongest design is developed at project design stage › Where feasible, a randomized approach should be considered  Poverty action lab › Active promotion of randomized control trials (RCTs) › Evaluations in conjunction with NGOs especially in India and Kenya

13. 3ie Joint AfrEA, NONIE, 3ie impact evaluation conference Cairo May 2009 NONIE3ie HistoryOrigins in DAC meeting November 2006 Evaluation gap working group PurposeDemand generation Guidance Coordination Promote quality impact studies, including financing Advocacy MembershipEvaluation networksGovernments and NGOs IE practitioners are associates EngagementOPSC representedSign up, membership and studies

14. 3ie  Websites › World Bank (PREM, IEG and NONIE) › Poverty Action Lab › 3ie – www.3ieimpact.org  Training courses › PAL › IPDET › Africa Impact Evaluation Initiative  Opportunities to conduct IEs › DIME › 3ie › Own resources

15. 3ie  Each year two themes and open window  Themes being determined – see 3ie website for details and to submit proposed questions  Southern-driven, issues-based with southern-led evaluation teams  Under each of these three windows › 10-12 ‘quick wins’ › 6-8 large studies › 6 baseline studies › 6 synthetic reviews

16. 3ie  Only promoting RCTs – not true  RCTs seen as gold standard – not exactly true, issues-led not methods-led, but where feasible an RCT should certainly be considered (best available method)  A positivist approach – true, but evidence-based policy making is an inherently positivist approach  Another northern initiative – partly true, but 3ie offers great potential for (1) Southern ownership of evaluation agenda, (2) harmonization around quality standards

17. 3ie The current policy framework in South Africa implicitly demands quality impact evaluations, and resources are available to help meet the supply

18. 3ie  The move to performance-based management leads to questions being asked about attribution  Impact evaluation is the approach to be adopted to address attribution  Impact evaluation has to adopt certain technical standards if it is to provide proper evidence  The new international initiatives exist to support efforts to expand programs of impact evaluation

19. 3ie VISIT www.3ieimpact.org