Using realist evaluations to design and implement complex interventions Case study from a market-systems development programme in Ethiopia Matthew McConnachie and Clarissa Samson

Outline Background to the realist approach Case study: The Market-Approaches to Climate Resilience project Key lessons from the case study Example of linking realist approach with planning frameworks

The realist evaluation approach Evaluates what works, for whom and in what contexts Focused on understanding the mechanisms which underpin interventions and how they interact with the context

The realist approach is well-suited for extrapolating findings to new contexts Able to provide findings that are linked to specific contexts and target groups, and explain the generative mechanisms (why change happened) But limited work done on how to use realist evaluations to inform the planning and design of interventions

Project Background: Market Approaches to Climate Resilience in Ethiopia Part of the DFID Building Resilience and Adaptation to Climate Extremes and Disasters programme (2015 – 2019) Implemented by two NGOs: Farm Africa and Mercy Corps Aimed to strengthen the private-sector and community- management systems for lowland Ethiopians (financial services, natural resource management, urban business opportunities) Complex programme, worked across multiple levels Systems-based approach, transformational change

Evaluation Background: Market Approaches to Climate Resilience in Ethiopia All projects conducted realist evaluations, a realist synthesis was used at the programme level (see parallel session 5.1 for details) LTS did the evaluation work on behalf of the project implementers Evaluation deliverables included periodic reports (M&E plan, baseline, mid-term, endline and extension evaluation reports)

Project planning stage: could have integrated the realist approach in a more structured way Proposal and initial-design stages included the evaluation team We identified evidence gaps but not entirely through a realist lens Theory of change development was participatory but not done explicitly in a realist way

Using the realist approach during project implementation: what went well Learning and adaptation between the main evaluation deliverables: mid-term to endline and endline into the extension phase We used data collection templates that were context and target group specific Project implementers found it useful to have the evaluation findings structured around specific target groups and contexts Generated some useful findings for the BRACED programme-level evaluation

Using the realist approach during project implementation: challenges More ongoing real-time assessments, tightly linked with the monitoring work done by the project staff Sequencing of quantitative and qualitative data collection work Finding the balance between project-wide generalisable findings and realist case study deep-dives Project staff having sufficient time and capacity to support the realist evaluation work Relatively short-time frame of the project

Key lessons Situational knowledge will always be important Integrate the realist approach with planning focussed frameworks (e.g. co-design, adaptive management, developmental evaluation) Develop the realist program theory concurrently with the design of the program, iterating back and forwards between them Identify the learning-gaps before designing the project Establish adequate processes and systems (workplans, adequate resources) Be participatory (tailored communication, workshops)

Example of linking the realist approach with other approaches Westhorp, Stevens and Kaye (2016) used realist action research in ‘The Bridgewater Project’, to evaluate different social security payment options Action research seeks to solve real world problems, trialing solutions until a ‘best fit’ solution is reached. Also integrated this with a co-design approach to develop service, a collaborative approach to service design. Benefits: helping project staff make sense of findings and identifying the needs of sub-groups