1. Evaluation of Nutrition-Sensitive Programs*
Presented by: Deanna Olney, PhD
October 19, 2016
* Olney, Leroy and Ruel. Evaluation of Nutrition Sensitive Interventions. Chapter 44. In: De Pee, Taren and Bloem (eds.) Nutrition and Health in a Developing World. 3rd Edition (forthcoming).
* Leroy, Olney and Ruel. Evaluation of Nutrition Sensitive Interventions. AOTR

2. Background
The Lancet Series on Maternal and Child Nutrition calls for greater investments in large-scale nutrition- sensitive programs.
Rigorous evidence of their effectiveness, how they work and cost-effectiveness is scant attributed to:
Weaknesses in program design and implementation
Poor evaluation designs.
There is a need for rigorous, theory based-evaluations that can provide evidence of:
What works to improve nutrition
How programs work to achieve nutrition impacts
What other impacts programs have
Cost and cost-effectiveness of achieving impacts

3. Key challenges: Program complexity
Complex in design and implementation:
Multiple goals, inputs, pathways of impacts
Span across different sectors (e.g. health, agriculture, education)
Within each program intervention potential variability in:
Delivery (quantity and quality)
Utilization
Adherence to the program protocol

4. Key challenges: Long time frames and impact pathways
Long time frames for implementation and evaluation:
Long pathways from program inputs to impact
Implement program at the level of quality desired
Achieve meaningful impacts on some biological outcomes such as child growth
Design and carry out a rigorous and comprehensive evaluation
… vs often short time frames imposed by donors

5. Priorities of implementers: Priorities of evaluators:
Key challenges: Differing priorities, expectations, incentives, and perceptions
Priorities of implementers:
Deliver a high quality program
Meet targets within the specified budget and time frame
Priorities of evaluators:
Rigorously evaluate the program
Answer key questions related to what impact was achieved why or why not it was achieved and at what cost
Misperceptions that can undermine trust:
Process evaluation viewed as evaluating the performance of the program implementers vs generating evidence to improve programming and understand impacts
Cost studies can be perceived as auditing the program’s finances

6. Key challenges: Implementation constraints vs. evaluation rigor
TNFP
Control
Regular program

7. Key challenges: Assessing benefits beyond targeted beneficiaries
Potential “spillover” of benefits beyond the targeted beneficiaries:
E.g. siblings, neighbors, etc.
Adds complexity:
Capturing all impacts of nutrition-sensitive programs
Measuring the cost-effectiveness of programs

8. Two essential ways to address these challenges
Build a strong partnership and collaboration between implementers and evaluators
Design and use a rigorous and comprehensive evaluation framework:
In-depth understanding of the program, program theory, program impact pathways, rigorous comprehensive evaluation design

9. Comprehensive evaluation: Embrace the complexity of the program
HKI partners with local NGOS
Village Model Farms (VMF) established and VMF owners trained
Small animal production established
Increased production of micronutrient-rich fruits and vegetables
Women’s groups established
Agriculture inputs (e.g. seeds, saplings and poultry distributed)
Provision of agriculture training
Provision of nutrition, health and hygiene education
Improved and developed gardens established
Increased poultry production
Increased household consumption
Increased Income
Improved maternal and child health and nutritional status
Health, hygiene and nutrition education received and understood by beneficiaries
Improved child care and feeding practices
Outcomes
Agriculture training received and understood by beneficiaries
Work with local authorities (e.g. village chief)
Village Health Support Group identified and trained
Improved women’s empowerment
Beneficiaries’ agriculture knowledge improved
Outputs
Inputs
Women’s increased control over resources resource
Impacts
Processes
Beneficiaries’ health, hygiene and nutrition knowledge improved
Cost
study
Impact
evaluation
Process + impact evaluation
What is the impact of the program?
How and why does the program (not) have an impact?
Cost of the program?
Complex in design and implementation:
Multiple goals, multiple inputs, multiple pathways of impacts, multiple outcomes and impacts.
Address both the underlying and direct causes of undernutrition
Span across different sectors (e.g. health, agriculture, education), requiring coordination and integration.
Within each program intervention potential variability in:
Delivery (quantity and quality)
Utilization
Adherence to the program protocol

10. Impact evaluation: Experimental designs
Designed to assess program impacts at different levels
Uses quantitative methods
Ideally conducted before the program starts and at the end of the program but can include intermediary time points
Requires a valid counterfactual and appropriate target groups, duration and indicators (with adequate sample sizes for each)
Experimental (or randomized) designs considered gold standard for impact evaluations.
Randomization: individual or group (cluster) level.
*If* done well, one can assume:
that both groups are comparable;
that the only difference between the groups is the program;
that the control group provides a valid counterfactual for the intervention group exposed to the program.
differences found in the outcomes of interest attributable to the program.
Consider time and timing in the evaluation design.
Essential to establish a valid counterfactual.

11. Impact evaluation: Invalid counterfactuals
Stunting (%)
Impact (pp)
0.0
-5.0
Randomized design
-5.5 pp
Before-and-after
+ 0.2 pp
With-and-without pp

12. How and why does the program (not) have an impact?
Two ways to assess how impact is achieved :
Measurement of intermediary measures (outcomes) in impact study
Process evaluation
Can also be used to improve on-going or future programs
Solid understanding of program theory and program impact pathways is key

13. Process evaluation
Examines the primary inputs, processes, outputs and outcomes along the primary program impact pathways
Conducted about one year after program implementation begins and can be repeated
Use a mix of quantitative and qualitative methods to examine the different steps along the impact pathways
Include different key stakeholders (e.g. program implementers and beneficiaries) and different program delivery points
Share the results:
Workshop with implementers
Identify what is feasible to improve and how improvements can be made

14. Cost study
Designed to estimate the overall cost of the program, the cost of the main program components and its cost-effectiveness
Can also be used to estimate savings or costs associated with adding, changing or dropping program activities or beneficiaries
One method that can be used is the Activity Based Costing Ingredients (ABC-I) approach.
Using the program impact pathways: provide detailed description of all program activities.
Identify the program’s main activities and “ingredients” needed for each.
Define the unit cost algorithms, i.e. the different types, quantities and costs of the “ingredients” necessary for each activity.
Calculate cost of each program activity and of full program.

15. In summary
Evidence of what works in nutrition-sensitive programs, how and at what cost is extremely limited.
Evaluation of nutrition-sensitive programs is complex but necessary.
Strong partnerships between program evaluators and implementers are needed to carry out rigorous program evaluations and to ensure that results are used to improve programming.
Comprehensive evaluations should include rigorous impact and process evaluations and when possible cost studies.
Guidance for future investments requires strong evidence from rigorous, theory-based comprehensive evaluations of different nutrition-sensitive program models that bring together interventions from a variety of sectors.