Impact evaluations of the UNICEF-IKEA Foundation programme on Improving Adolescents Lives in Afghanistan, India and Pakistan: Integrating an equity and gender equality focus ex- ante in programme and evaluation design UNEG Geneva, 25 April 2016

Key messages Methodological requirements of impact evaluations provide opportunity to integrate equity and gender equality concerns in programme and evaluation designs from the beginning Clearly understanding the theory of change and key evaluation questions ex-ante, eliminates the need to scramble for appropriate approaches later

Impact evaluations in UNICEF Perspective: Most UNICEF evaluation are retrospective, ex-post Increasingly: Evaluations embedded in programme design ex-ante Especially critical in “impact evaluations”: attribution of results to programme, not pitched at OECD-DAC “impacts” Setting up programme design with a view to demonstrating attribution

Programme background Experience of designing impact evaluations for a multi-country programme on “improving adolescents’ lives” funded by the IKEA Foundation, worth some US$16 m Focus on: – Reducing child marriage – Reducing adolescent pregnancies – Increasing secondary school enrolment

Programme objectives Programme, by definition – like most UNICEF programmes – has an explicit focus on equity and gender equality: – Child marriage affects predominantly girls – Is especially prevalent amongst the most economically and socially disadvantaged groups – Secondary school enrolment much lower amongst girls – Idea that increasing secondary school enrolment will reduce child marriage and teenage pregnancies

Impact evaluation rationale i/ii Little evidence, internationally, on what works and what doesn’t, in reducing child marriage Complex issues, social norms Programme conceived as a pilot or demonstration project – aims to yield lessons that can be used to scale up interventions Knowledge to be used for policy advocacy

Impact evaluation rationale ii/ii Decision to conduct impact evaluations aimed at demonstrating change that can be attributed to the programmes Decision to apply experimental and quasi- experimental designs: – India and Pakistan: RCTs – Afghanistan: Propensity score matching + DD

Impact evaluation overview Contracting of evaluation ex-ante, for duration of programme Primary partner: 3ie – provides oversight, management of institutions Evaluation institutions: – Afghanistan: Johns Hopkins University – India, Pakistan: University of Mannheim Average budget US$ 600,000

Impact evaluation design i/ii Bringing in evaluators during the programme planning phase – makes for integrated, iterative programme and evaluation design Intention to identify clearly: – Theory of change – What are programme results – How will they be achieved – What indicators will be used – What approaches will allow us to demonstrate attribution

Impact evaluation design ii/ii Evaluation designs aim to demonstrate: – Overall programme vis-à-vis non-programme areas – Effectiveness of “basic package” vis-à-vis different treatment arms (factorial design) – Impact on different population groups, disaggregated by sex, ethnicity, religion, income, language, etc. – “Why” approaches worked the way they did (supplementary qualitative analysis)

Challenges Programme staff and partners not used to thinking programmes through in such detail Critique of cost Delays in launch (implementation focus/dynamic) Need to bring on board all stakeholders up-front: especially to explain approach (randomization) Critique that evaluation tail wags programme dog

Opportunities Integrated focus on equity and gender equality in programme and evaluation design Promises the generation of highly rigorous evidence that will demonstrate effectiveness of approaches – + impact on different stakeholder groups

Some implications Change in programming and evaluation culture: embedding evaluation in programme design Need to enter into longer-term commitments (4-5 years) with research company Risks of evaluators developing a stake in the programme

Conclusions Impact evaluation approach focuses attention ex-ante on equity and gender equality – Need to clearly establish related indicators, targets and means of verification (data collection methods, evaluation approaches) Eliminates challenge of needing to chase the evidence tail – evidence generation on key equity and gender equality issues in-built

Final thoughts In UN context where centralized ‘independent’ evaluation is often ex-post – How to encourage the conduct of rigorous impact evaluations at the decentralized level? When rigorous evaluative building blocks are in place, easier to conduct meta assessments and systematic reviews at the macro level – For broader thematic / programmatic evaluations – For evaluations tracking progress towards SDGs