Presented by: Shubha Chakravarty (Economist, AFTPM) Impact Evaluation team: Mattias Lundberg (Sr. Economist, HDNCY) Markus Goldstein (Sr. Economist, AFTPM and DEC) AGI Technical Meetings July 22-23, 2010, Washington, DC Impact Evaluation of the Adolescent Girls Initiative

Objective: To measure the impact of the program on the well- being of participants and their families Method: Comparison of treatment and control groups – Why do we need a control group?  Allows us to attribute a causal effect to the program Design Options: Randomized Pipeline (Liberia), Lotteries among eligible girls (Jordan), Random selection of villages (S Sudan), Discontinuity with quantitative scoring (Nepal), Matching Timeline: – Baseline surveys 2010 (2011 in some cases) – Follow up surveys 2012 – Final reports by early 2013 Impact Evaluations of AGI 2

What is the impact of the program on: – Economic outcomes for participants and their households : employment, earnings, investment, savings, borrowing, and lending. – Socioeconomic behaviors and outcomes: marriage, fertility, time use, experience of gender-based violence, and attitudes toward risk – Empowerment of participants: physical mobility, aspirations for the future, control over household resources, self-confidence – Knowledge: Financial literacy, HIV/AIDS How do the program impacts vary according to the demographic and personal characteristics of the participants? – Variation by age, level of family support, level of stress Does job training work in post-conflict settings? How does childhood experience of trauma, displacement, schooling interruption, and/or loss of relatives affect the success of program participants?  Why don’t these indicators correspond to the results framework? What We Will Learn 3

Cross-country learning – Results shared across countries – International workshops at different stages of project to share lessons learned – Core evaluation team works across countries to ensure consistency Expansion: Potential to add AGI programs in new countries Sustainability: By comparing outcomes, we can learn what methods work best.  These lessons can inform design of future programs and the scale-up of these pilot projects to ensure their success in the future Why Impact Evaluation? 4

“M” vs. “E” – Monitoring: ongoing process of data collection on the outputs of a program (Are we doing the program right?) – Evaluation: assessment of the consequences of a program (Are we doing the right program?)  Evaluation can’t work unless we know what we are evaluating  What services were delivered?  Which participants received which services? The Role of Monitoring 5

Essential data from monitoring for IE – Who participated? How often? – How many hours of training were delivered? – What was the content of the training (curriculum, OJT, etc.) – Who completed the training? Who dropped out and why? – Which participants were with which trainers? What was the composition of each classroom? – How satisfied were the trainees with the program? The Role of Monitoring 6

Project M&E officer: Primary liaison between implementing agency and IE team; responsible for project monitoring system IE Field Coordinator: Primary liaison between Bank IE team, implementing agency, and local survey firm during surveys Washington IE Team: Responsible for design of IE; assist with development of monitoring system; coordinate with Bank project team; data analysis, writing, and dissemination Survey Firm: An independent agency (preferably local) to conduct data collection at baseline and endline  Huge potential for learning, sharing, capacity building Many Actors, One Goal 7

Improve program design – How did participants benefit? What were the unintended consequences? Improve targeting – Which girls benefit most from your program? How can you better reach the others? Improve program implementation – Which trainers/ courses were most effective? Hard evidence of program’s impact – Provide justification for expansion from an external source Guidance for replication – Which program components are the most essential? How does all this help you? 8

Thank you 9