1. Reflections from a realist evaluation in progress: Scaling ladders and stitching theory Melanie Punton, Isabel Vogel and Rob Lloyd 21 April 2016

2. ‘The effective use of research and evidence can play a crucial role in making policy more successful.’ 2

3. Session overview What is BCURE? What is realist evaluation? The stages of our realist journey, and challenges we faced along the way – Developing theory – Testing theory – Refining theory What can realist evaluation offer international development? 3

4. Building Capacity to Use Research Evidence (BCURE) 4 Improved capacity to use evidence in policy making (O) Evidence is used more (and better) in policy making (O) Better quality policy (O) Training Mentoring Networking Events Workshops New tools, systems and guidelines DFID funded: £13 million, 11 countries, 6 projects, 2013-2017 Working with champions

5. The six BCURE projects 5 PartnerProject nameCountries (bold = case study country) Adam Smith International (ASI) Africa Cabinet Decision-Making Programme South Sudan, Liberia and Sierra Leone ECORYSBuilding Capacity for the Use of Research Evidence Bangladesh HarvardData and Evidence for Smart Policy Design Pakistan, India African Institute for Development Policy (AFIDEP) SECURE-HealthKenya, Malawi University of Johannesburg (UJ) UJ-BCURESouth Africa VakaYiko ConsortiumVakaYikoZimbabwe, South Africa, Ghana

6. What did the commissioners want from the evaluation? 1.To strengthen the global evidence base on the effectiveness of capacity building approaches to support evidence-informed policy 2.To evaluate the effectiveness and value for money of the six BCURE programmes Three year evaluation, accompanying programme, 2013-2017 6

7. Why realist evaluation? BCURE is a pilot. DFID wanted to understand how and why capacity building can contribute to increased use of evidence in policy making… …To inform decisions within and beyond DFID about whether to fund and how to design this type of programme in future. 7

8. Theory based approach, developed by Pawson & Tilley (1997) Not ‘what works’ but ‘what works, for whom, in what circumstances, and why?’ Answers this through opening up the black box: theories about how the resources introduced by programmes in particular contexts ‘spark’ mechanisms which generate outcomes. What is realist evaluation? THEORY Mechanisms operating in particular contexts to generate specific outcomes (CMOs)

9. … the ‘magic spark’ that leads to change Mechanism Self-efficacy ‘Aha moment’ Sufficient pre-existing knowledge Participants are working on (and struggling with?) ‘live’ policy processes Increased use of evidence in day job Learning put straight into use to shape a policy BCURE training course Peer support and encouragement Positive group dynamics; participants on same ‘level’ Attendance linked to promotion ‘Crystalliser’ Awareness of ‘this thing called EIPM’ Context Mechanism Outcome +=

10. Underlying theoretical assumptions 10 Type of assumptionAssumption The nature of reality (ontology) The material and social worlds are real How we can we learn about reality (epistemology) We can acquire true knowledge about the world, but knowledge will always be partial and incomplete (as opposed to positivism: 'what you see is what you get‘, or constructivism, ‘all knowledge is subjective’) How causality worksGenerative logic of causality This is important because it has implications for the approach, tools, way of thinking and what the results look like

11. Logics for establishing causality 11 Correlational / counterfactual logic Multiple-conjectural logic Generative logic Statistical approaches, RCTs Qualitative Comparative Analysis Realist evaluation, contribution analysis, process tracing To what extent did the intervention make a difference? Did the intervention make a difference and through which patterns? How and why did the intervention make a difference Schatz, F., & Katharina, W. (2016). Qualitative Comparative Analysis: A Valuable Approach to Add to the Evaluator’s Toolbox? Lessons from Recent Applications. Centre for Development Impact, (13).

12. How do you generalise from realist evaluation findings? 12 Causal mechanisms are real forces that exist in the (social and physical) world and cause things to happen The same mechanisms are present in very different situations Realist findings are therefore portable.

13. How do you do a realist evaluation? Overall aim: refined and evidenced set of theories about what works to build capacity for EIPM, for whom, in what circumstances and why Three broad iterative stages

14. 14 Developing theory

15. Initial Common Theory of Change 15 Within this overarching CTOC, explanations of particular causal links and processes are hypothesised in the form of intervention-context-mechanism-outcome configurations (ICMOs)

16. 16

17. Initial CMOs ‘What works, for who, in what circumstances and why?’ Building on existing theories, not starting from scratch ‘Capacity building for EIPM’ chock full of theories and assumptions ‘Good quality’ ‘evidence’ ‘Good quality’ ‘policy’ ‘Capacity’ Rich literature from diverse fields Psychology Adult learning Political science Health Role of evidence in policy International development

18. The literature review shaped how we think about ‘building capacity for evidence informed policy making’… ‘Research evidence’ just one form of evidence required to make policy Appropriateness of evidence as important as ‘quality’ Evidence is never neutral Need to look beyond rational and linear models of policy processes, to encompass the role of power and politics, networks and interactions, cognitive limits of rationality Capacity is multi- dimensional

19. Challenge 1: Developing, unravelling and re-stitching CMOs “Is this really a mechanism?” What helped: Recognising that CMOs are heuristics, not reality Introducing an ‘I’ into the ‘CMO’ Sentences, metaphors, catchy names 19

20. Challenge 2: How many ICMOs? What helped: Pragmatism! Focussing on operational relevance Constrained by resources and access Guided by literature: where could we add most value? 20

21. 21 Testing theory Testing theory

22. Testing theory Three rounds of data collection and analysis, with each round encompassing: – 6 country visits – Around 30 qualitative interviews per country: project staff, intervention participants, high level government informants etc. – Review of monitoring data collected by programmes, policy documentation that provides evidence of change in processes, etc 22

23. Challenge 3: Mastering the art of the realist interview “I’ll show you my theory if you show me yours” What about confirmation bias? What (we hope will) help: More training Asking for examples Adjudicate between rival theories 23

24. Challenge 4: Analysing our data in a realist way 24

25. 25 Refining theory Refining theory

26. Refining theory “How do we get from the granular findings to revised ICMOs?” Ladder of abstraction Elements of meta-ethnography 26

27. How and why does capacity building lead to change? ICMO 2: the ‘eye opener’ Based on 14 interviews from two countries: Zimbabwe and Kenya; plus ACD regional conference

28. Challenge 5: encompassing complexity CMOs looked like this Reality more like this: 28 What helped Thinking in terms of ‘levels of a system’ Layering ICMOs: outcome on one level may become a context on another level; feedback loops

30. What can realist evaluation offer international development? Systematic way of exploring context and complexity Operationally relevant findings (although be careful about your messaging!) But: 30 Tensions between RE and structures and incentives of aid industry?