1. Module 10 Rapid, non-technical review of EFGR-responsive evaluation methodologies
Technical Assistance on Evaluating SDGs:
Leave No One Behind
EvalGender+ Network together with UN Women, UNEG and EvalPartners

2. Presentation developed by Michael Bamberger and Asela Kalugampitiya based on Chapter 4 of ”Evaluating the Sustainable Development Goals within a “No-one left behind” lens through equity-focused and gender-responsive evaluations”

3. Outline
Why it is important for non-evaluation specialists to understand the basic principles of evaluation design
Evaluation levels and types
The main evaluation methodologies
The integration of EFGR evaluations into the SDGs

4. 1. Why it is important for non-evaluation specialists to understand the basics of evaluation methodology
There is no “best” evaluation design
Different designs are required to answer different evaluation questions
Many evaluators have a preferred evaluation design that they will often recommend even when it is not the most appropriate
All evaluation designs have strengths and weaknesses
Consequently evaluation managers and other stakeholders have an important role in working with evaluators to select the best design for a particular evaluation

5. 2. Evaluation: Levels and types
Evaluations can be conducted at 3 levels
Policy level
To assess the effectiveness with which policies are implemented and how well they achieve their objectives
Program level
To assess the effectiveness with which programs are implemented and coordinated
How well each component achieves its objectives
The effectiveness with the overall program achieves its broad goals and objectives
Project level
To assess the effectiveness with which projects are implemented and how well they achieve their objectives

6. There are 4 main types of evaluation
Purpose
Policy evaluations
How well do the design, implementation and outcomes of national and sector policies contribute to EFGR goals?
Formative evaluations
Providing regular feedback to management and stakeholders on how the design and implementation of SDG projects contribute to EFGR goals
Proposing corrective measures
Learning lessons
Developmental evaluations
Similar to formative evaluation but focusing on innovative and emergent projects
Assessing how complexity dimensions affect implementation and EFGR goals
Summative (impact) evaluations
Estimating quantitative (and sometimes qualitative) changes in EFGR indicators
Assessing the extent to which these changes can be attributed to the SDG program

7. 3. The main evaluation methodologies
Experimental and quasi-experimental designs
Statistical designs
Theory-based evaluations (including theory of change)
Case study methods
Qualitative and participatory methods
Review and synthesis
Complexity responsive evaluations

8. A. Experimental and quasi-experimental designs
Experimental designs [Randomized control trial = RCT]
RCTs provide the strongest methodology for assessing project impacts
Subjects are randomly assigned to receive the project treatment [water, school meals, small business training etc] or to be in a control group that does not receive the project treatment
RCTs require large samples and are quite expensive
Works better for simple project designs with few components and few outcomes than for large, complex programs.
Ethical and political concerns using RCTs

9. Simple experimental design [RCT]
Before project treatment
Intervention
After project implemented
Project group P1 X P2
Control group C1 C2
Random assignment

10. Quasi-experimental design [QED]
Similar design to RCT but random assignment is not possible and groups are matched using statistics or judgment
Less precise than RCTs but can be used in a wider range of situations

11. Comparing RCTs and QEDs
RCTs must be planned and administered before the project begins
QEDs can be administered after the project has began
RCT are methodologically more precise but QEDs are more flexible
Both designs are quite expensive as they require large samples and a high level of research experience

12. B. Statistical designs Statistical modeling and econometric analysis.
Mainly used at the national level to assess the impacts of policies or country-wide programs by comparing experiences with other countries
Countries are matched on macro-indicators such as GDP growth, per capita income, investments in different sectors, rate of infrastructure construction.
Useful for national programs where it is not possible to identify a comparison group (counter-factual) within the country

13. C. Theory-based evaluation [theory of change]
Approaches such as theory of change are used to explain the steps through which an intervention is intended to achieve its objectives.
Identifies the key assumptions on design, implementation strategies and potential constraints which must be tested
A good theory of change should also focus on broader social, political, economic, legal and other factors affecting project implementation and outcomes
Program impact and effectiveness can be assessed by comparing actual implementation experience with the TOC

14. Theory-based evaluation [continued]
Theories of change (TOC) can be used to provide a framework for every kind of evaluation design
TOCs help provide a framework for learning lessons from the experience of project design, implementation and outcomes.

15. D. Case-based approaches
The case is taken as the unit of analysis
A case may be an individual, household, community, organization, state or even a country
A sample of cases are selected to:
be representative of a total population
To illustrate particular sub-groups (the most or least successful, outliers etc)
The analysis can be qualitative and descriptive or quantitative
Case-based methods can be used as stand-alone evaluations or to illustrate and explore groups identified in surveys.

16. Qualitative comparative analysis (QCA)
QCA is a form of case study that has become popular in recent years
It requires a higher level of technical expertise but can provide more rigorous estimates of project effectiveness and outcomes
QCA identifies:
the combination of factors that must be present for a project to achieve its intended outcomes
The combination of factors that prevents a project from achieving its intended outcomes
Usually requires a sample of at least 15 cases

17. E. Participatory and qualitative analysis
Involves a wide range of stakeholders in design, implementation and interpretation of the evaluation
Can be used for methodological reasons or to support a rights-based/ empowerment approach
Provides a deeper understanding of the lived-experience of different groups
Gives voice to poor and vulnerable groups
Analysis of processes
Helps understand the influence of contextual factors and processes of social control

18. Participatory and qualitative analysis [continued]
Uses a mixed methods approach combining a wide range of tools and techniques for data collection and analysis
Participatory approaches are in-line with the human rights and gender equality evaluations proposed by UNEG

19. Examples of participatory and qualitative methods
Participatory consultative methods such as PRA
Outcome mapping and outcome harvesting
Most significant change
Participant observation
Key informant interviews
Community consultations and some kinds of focus groups
Longitudinal case studies

20. F. Review and synthesis studies
Identification of all evaluations conducted on a particular topic (e.g. the effects of micro-credit on women’s empowerment)
Selection of studies that satisfy standards of methodological rigor
Often only randomized control trials are acepted
May incorporate a theory of change to structure findings
Summary of findings and lessons that are statistically sound
Provides a useful starting point for designing programs and indicates what kinds of outcomes can be expected – and what is not likely to be achieved

21. G. Complexity-responsive evaluations
Not widely-used but important for EFGR as programs with a gender or equity focus frequently involve dimensions of complexity
Complexity-responsive evaluations can largely use familiar evaluation tools described in this session but it is necessary to
begin with a complexity diagnostic to understand why and how the program is complex
Conclude by reassembling all of the individual evaluations to understand the big picture
Complexity is important because often each program component has a positive rating but the overall program fails to achieve its broader objectives

22. 4. Principles for integrating EFGR evaluations into the SDGs
Begin with a review of lessons learned from past approaches and evaluations
build gender and equality into the theory of change and results framework
Develop a checklist of areas where EFGR principles and indicators can be integrated into the evaluation process
Begin with rapid diagnostic studies to help understand the EFGR issues which should be addressed
Integrate gender and equality into ongoing or planned evaluations