1. The Rapid Evaluation, Action, and Learning (REAL) Approach
A toolkit to measure and refine changes and interventions in health campaigns

2. The Rapid Evaluation, Action, and Learning (REAL) Approach
Agenda
Principles of the Rapid Evaluation, Action, and Learning (REAL) approach
Five Steps of REAL
THE RAPID EVALUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

3. Primary users & target audience
This toolkit is geared towards readers familiar with operations research and monitoring and evaluation tools and approaches, including:
Staff of the Bill & Melinda Gates Foundation
Campaign implementers and key stakeholders
Campaign effectiveness grantees
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

4. Learning objectives
By the end of this module, users should be able to:
Identify what information is needed to make a decision to scale up, scale down, or revise a change to a campaign.
Describe the trade-offs and criteria for selecting the most feasible and appropriate (“fit-for- purpose”) approach to monitoring, evaluation, or operations research of campaign effectiveness changes.
Describe the steps and components of the REAL approach to measure changes in the processes, outputs, outcomes, and impacts of interventions and strategies to improve campaign effectiveness.
Articulate how to iteratively and continuously adapt interventions to optimize outputs and outcomes.
Identify what type of intervention and the most important characteristics of the intervention to test.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

5. Introduction to the REAL Approach
THE RAPID EVALUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

6. Principles of the REAL approach
Right-sized: Aims to balance rigor with efficiency.
Strategic: Prioritizes the production of information that is strategic for decision-making.
Context matters: Aims to identify how a change is working, not just whether it is working.
Participatory: Draws on multiple stakeholder views to strengthen the validity of logic models.
Timely: Emphasizes rapid and right-sized assessment to better align with policy windows.
Adaptable: Can be adapted to various types of change (simple, complicated, complex) or to changes in complex systems.
Ongoing: Integrates design, measurement, and adaptive management to rapidly test interventions and enable continuous learning and improvement.
THE RAPID EVALUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

7. Five Steps of REAL Identify a change to address challenges
Build a logic model for the change
Prioritize what to measure
Collect data
Use data for adaptive management
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

8. STEP 01 | Identify a change to address challenges
In the root cause analysis (RCA) toolkit, you identified a challenge or bottleneck that constrained the effectiveness of a campaign. Based on the actionable causes you identified, you now need to identify a change to make and test.
Write the question where everyone can see and make sure the team understands the question.
Brainstorm as many ideas as possible. We recommend using post-it notes. Encourage the team to build on others’ ideas and to not criticize any idea.
Bundle the post-its into groups of similar ideas. If you have a diagram of the RCA, you can map the ideas to the root causes and causal factors.
Choose a bundle of ideas to implement and test. Continue to consider feasibility, acceptability, how the solution will address identified challenges, and what evidence there might be elsewhere on the solution’s effectiveness.*
Other resources for designing a change:
The Field Guide to Human-Centered Design
Design for Health website
Six Steps in Quality Intervention Development (6SQuID)
*Adapted from The Field Guide for Human-Centered Design (IDEO.org, 2015)
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

9. STEP 02 | Define the key problem or challenge (or success)
A logic model is a graphic depiction (road map) that present the shared relationships among the resources, activities, outputs, and outcomes/impacts for your program (CDC, ).
This step is critically important because:
It establishes the underlying theory of how a change will contribute to the intended outcomes through one or more causal pathways.
An accurate logic model will help you to identify what to measure in Step 3.
The more complicated a change, the more important it is to explicitly articulate how the change(s) will produce the intended outcomes.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

10. STEP 2.2 Describe the intended outputs and outcomes of the change
List all the intended results of the change, then order them sequentially.
List the campaign change on the left with lines left-to-right showing each result that is produced.
List as many results as you can. You won’t have to measure them all, but it’s important to capture the full results chain.
Example logic model describing a campaign change and its results
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

11. STEP 2.1 | Describe the change
Describing the change will build from the participatory approach taken in Step 1 to identify or design the change.
Your description will be based on the perspectives and feedback from diverse and knowledgeable stakeholders.
Describe the campaign ‘change’ as an input or activity. For example:
Mobile phone-based micro-planning
Payment of campaign staff using mobile money
Transportation vouchers for campaign staff
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

12. STEP 2.3 Unpack assumptions in the causal links
This step unpacks the assumptions we make between the change and the results we think it will produce.
Causal leaps become visible.
Missed or forgotten steps are identified.
Uncover optimal sequencing (for complicated changes).
Measurement priorities become clearer (Step 3).
Logic model with added assumptions
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

13. STEP 2.4 Seek other perspectives and contextual factors
Vetting the logic model with diverse stakeholders using participatory approaches ensures its validity and improves the likelihood that findings will be used. You can do this through:
Regularly planned meetings
In-person interviews
Telephone interviews
Logic model with refined assumptions and contextual factors that may influence the results chain
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

14. Incorporating context in REAL
To answer questions about why and how the change works, and about potential for scale and transfer, a rich understanding of context is needed.
Contextual factors from the “Implementation Research Toolkit” (WHO, 2014).
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

15. STEP 2.5 Iterative updating of the logic model
Vetting the logic model with diverse stakeholders may identify the need to revise or adjust the change activity itself.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

16. STEP 03 | Prioritize what to measure
The choice of relationships to monitor should be prioritized based on which are most strategic for success. Focus on causal links where uncertainty exists or where there is disagreement between decision-makers. You can also prioritize what to measure base don the costs or risks of making the wrong decision.
This step is critically important because:
Not all causal links are necessary to measure
Choosing which to measure should be based on considerations of strategic importance:
What information do decision-makers need to scale up or down?
How much, and what type of information will they need?
Where is there the greatest uncertainty?
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

17. STEP 3.1 | Agree on what information is needed for ‘success’
Common questions include:
Does the change produce its intended effect?
Why or how does the change produce its intended effect?
How can we further improve the process?
What does the change cost?
How could the change be scaled up?
How could the change be transferred to another setting?
Questions such as ‘how and why did the intervention produce its result here, and how could it produce similar results elsewhere?’ are as important as ‘was the intervention effective?”
Identify the key decision-makers of the REAL findings. Discuss which questions are the greatest priority for each of them.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

18. STEP 3.2 | Identify causal links with the greatest uncertainty
A central feature of rapid cycle testing is that it targets the testing and measurement to a few priority areas instead of having to measure everything.
Consider which existing data are available for any of the assumptions.
During the stakeholder engagement for the logic model, ask the question, “How certain are you that this step will occur?”
If multiple assumptions/results are equally uncertain, focus on measuring earlier links in the process which are necessary for subsequent results.
Refined logic model assessed for uncertainty
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

19. STEP 3.3 | Combining the two criteria to clarify data to collect
Combine the priority strategic questions from Step 3.1 with the logic model’s causal links with the most uncertainty to develop more operational, actionable testing questions and indicators. The final articulation of questions should be informed by available data sources.
Example of adapting questions based on available data sources
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

20. STEP 04 | Collect data
In this step, you will develop a data collection and analysis plan that is fit-for-purpose (e.g. timely, feasible, affordable) and produced information required to make strategic decisions.
This step should include staff responsible for designing or implementing monitoring or data collection systems, or those involved in the analysis and interpretation of data.
This step is critically important because:
Strategic decisions should be based on data or evidence; this steps helps identify the most strategic data to collect/measure.
This step reminds us to think somewhat differently about which data are needed and how to collect them.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

21. STEP 4.1 | Develop a data collection plan with indicators and data sources
In this step, we will convert the actionable testing questions into measurable indicators. Indicators should be feasibly measured, whether through existing systems or simple primary data collection. The timing of measurement should align when the result is expected to occur. Remember to not lose track of contextual factors.
Example data collection plan
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

22. STEP 4. 2 | Work with systems owners to integrate
STEP 4.2 | Work with systems owners to integrate indicators/fields into data collection tools and processes
Early in the process, it’s important for the team to note all potential data collection systems and tools in use, and whether there are opportunities to alter or add data collection fields. Its important to be able to record the data collected, and to also be able to view or export the data for analysis.
Think carefully about how to look at the data, and which analyses may be performed. Explore the ease of running reports or exporting data from routine systems, including what support is needed and how frequently this could occur.
Be aware of and align with local Institutional Review Board (IRB) policies. In most cases, this is considered routine program monitoring (non-human subjects research). However, it’s important to consider whether any of the data could put campaign staff at personal or professional risk, or whether even causal conversations with campaign staff could introduce power imbalances and risks.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

23. STEP 4.3 | Implement data collection and look for trends
Data collection should begin as soon as is feasible and should be efficient (not all indicators may need to be collected immediately). Data collectors should be trained on how to enter new data and why they are being asked to do so. The team should look at the data collected to ensure there are no quality issues related to data collection or entry.
Common data quality checks include:
Completeness: proportion of stored data against the potential of 100% complete.
Timeliness: proportion of stored data submitted on time.
Validity: data are valid if it conforms to the definition for that data.
Accuracy: degree to which data correctly describes the ‘real world’ event.
Usefulness: for qualitative data, is the data collection useful in answering the testing questions and strategic questions? If not, data collectors may need additional coaching on how to frame questions, how to follow up with probes, etc.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

24. STEP 4.4 | Adapt and iterate data collection
Ongoing adaptation of questions, indicators, and data collection tools is encouraged.
Adaptation should be based on:
Experiences implementing the data collection plan:
If data collection is not producing the information needed, tweak it. Because this is not a research study, there is no harm changing an actionable testing questions mid-course to better respond to strategic questions
The need to respond to new information:
Because the logic model is dynamic, as the change is implemented, the team may identify new assumptions and contextual factors or observe other causal links or unexpected consequences.
Continue to iterate on the logic model and cycle through the entire rapid cycle testing process to update what needs to be measured and how.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

25. STEP 05 | Use data for adaptive management
In this step, you will use the data to answer the initial strategic questions posed in Step 2 so that you (or other decision-makers) can make the necessary decisions to scale up, scale down, or continue to refine the change.
This step is critically important because:
It helps identify which approach is appropriate to support strategic decisions related to your change.
It helps turn evidence into action – not only in terms of refinements to the campaign change, but also regarding broader evidence-informed decision-making for improving campaigns.
Ongoing analysis and interpretation of data will support the ethos of adaptive management and continuous quality improvement.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

26. STEP 5.1 Identify appropriate analytic methods for each strategic question
Examples of analytic approaches for various strategic questions
When deciding which analytic approaches to use, keep in mind the strategic question you are trying to answer, and the decision- makers evidence needs. Leverage existing meetings with key decision-makers to vet your approach. Perform this step together with other question development and data collection steps as it may identify new data that are needed.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

27. STEP 5.2 & 5.3 Carry out the analysis & take action based on information
Plan for who will be responsible for the analysis and whether the team includes the right skills. Depending on the analytical approach, this might happen continuously.
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

28. STEP 5.4 Summarize findings for strategic questions to key decision-makers
You involved key decision-makers in the choice and articulation of strategic questions, and now it is time to provide the answers to those questions. For each question or sub-question, you will want to provide the follow information:
RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT

29. STEP 5.5 |Transition to a change management mentality
THE RAPID EVAUATION, ACTION, AND LEARNING (REAL) APPROACH TOOLKIT