Systematic analysis and synthesis in qualitative evaluation Case study evaluation of the Oxfam Novib programme in Burundi ( ) Ferko Bodnar CDI Conference on Impact Evaluation Wageningen, March 2013 ‹#›

Presentation outline 1.Purpose of the evaluation 2.Oxfam Novib programme under evaluation 3.Design (1): evaluation method 4.Analysis and synthesis of different opinions 5.Expectations underpinning design 6.Expectations underpinning communication 7.Factors affecting use of Evaluation results 2

Purpose of the evaluation Accountability: Dutch Ministry, other donors, Dutch public Learning: – input for strategic decisions in current and upcoming Oxfam-Novib (ON) programme in Burundi; – input for ON thematic policies and ON strategic programme , – input for Oxfam International; – support organisational learning by local partner organisations and other stakeholders. ‹3›

Oxfam Novib programme under evaluation Oxfam Novib (ON) worked on 5 themes: 1)Sustainable livelihood: food security, income, employment, markets, 2)Social services: health, education, 3)Security: emergency aid, conflict prevention, 4)Social and political participation, 5)Gender and diversity : 31 projects, 12 partner NGOs, 5.7m Euros. Post-conflict context. Weak social cohesion. Programme outcomes and impact, not outputs individual projects 4

5

6

7

8

9

Design (1): evaluation method overview Reconstruct intervention logic with ON staff Netherlands Identify judgement criteria with partner organisations Burundi Interviews with resource persons : partner organisations, other organisations and government Focus group discussions with beneficiaries and non-targeted households Survey in targeted and non-targeted communities (300 households) Restitution / synthesis meeting Analysis and synthesis table for different opinions 10

Design (2): Reconstruction intervention logic per theme Impact beneficiaries Outcome beneficiaries (behaviour) Outcome other organisations Outcome partner organisations (PO) (Outputs PO) (Activities PO) ‹11›

Analysis and synthesis of different opinions (1) Sources: – S: Survey – B: Discussion beneficiaries – N: Discussion non-beneficiaries, as reference – P: partner organisations, implementing ON programme – O: Other organisations (NGO) – G: Government Interpretation opinion: + Confirms expected effect of ON programme □ Does not confirm expected effect, or change not attributed to ON - Confirms a change contrary to the expected effect 12

Analysis and synthesis of different opinions (2) 13 Evaluation question 1. Judgement criterion 1.1SBNPOG Opinion Opinion 2- Judgement criterion 1.2SBNPOG Opinion 3- Opinion 4+ - Draft conclusion, answering evaluation question (indicating the source : S, B, N, P, O or G)

14 1. Impact on food security? % HH eating 2 meals / daySBNPOG Number of meals has not changed □□ (□)(□) Better-off, active HH increased, but poor HH reduced number of meals per day - Careful, HH tend to underestimate, hoping for WFP assistance □ HH eat better now than 5 years ago, due to:++(+)  Irrigation swamp rice fields+  Peace and collaboration Number of months per year with sufficient food increased SBNPOG Number of months with sufficient food decreased, especially among vulnerable groups -(-) Number of months with sufficient food increased, due to disease resistant cassava cuttings + Irrigation in swamp rice has increased production of actively involved participants (B), but this has not improved nutrition for the majority of surveyed households in targeted areas (S). The positive effect of irrigation is annulled by the general trend of declining food production (S, B). [incomplete] Example:

15

Expectations underpinning design 1.Individual project objectives fit in programme objectives – Outcome and impact: + – Project outputs: - 2.Inclusion independent sources strengthens contribution to ON – Valuable info, context, unintended effects, contribution: + – Relatively few targeted beneficiaries: - 3.Systematic presentation different opinions reduces evaluator bias – Balancing findings, reduce bias: + – (Time consuming: -) 16

Expectations underpinning communication 1.Transparent presentation and synthesis makes conclusions more acceptable – PO who felt their results were underappreciated could add clarifications and opinions without dominating conclusion. – PO appreciated the overview and focus on higher level outcome and impact. Recommend this for planning 17

Factors affecting use of Evaluation results Design ++: Concern attribution / contribution: including independent sources  Stronger recommendations. Concern acceptable conclusions: transparent analysis  More consensus ON concern IOB evaluation assessment: rigorous evaluation questions, criteria, triangulation  More reliable conclusions  ON adaptation (see ON mgt response) Communication +/-: Restitution workshop: consensus  acceptable conclusions ON management response adopted recommendations Follow up in planning ON programme : limited? 18