1. Gender and Impact Evaluation
Urmy Shukla
Capacity Building Manager
CLEAR South Asia / J-PAL South Asia at IFMR

2. Overview What is impact evaluation? What is a randomized evaluation?
How can we use impact evaluation as a tool for gender policy and advocacy?
How can gender be successfully integrated into impact evaluation?
Examples from J-PAL

3. J-PAL’s Approach

4. J-PAL is a Centre at MIT’s Department of Economics, with Regional Offices Worldwide
J-PAL is headquartered at MIT’s Department of Economics with independent regional offices in Africa, Europe, Latin America, South Asia, and Southeast Asia that are hosted by local universities. 

5. How to Measure Impact? Impact is defined as a comparison between:
The outcome some time after the program has been introduced
The outcome at that same point in time had the program not been introduced
This is know as the “Counterfactual”
The critical objective of impact evaluation is to establish a credible comparison group – a group of individuals who in the absence of the program would have had outcomes similar to those who were exposed to the program.

6. Impact: What is it? Primary Outcome Time Intervention
[Repeat from Lectures 1 AND 2]
Time

7. Impact: What is it? Primary Outcome Time Counterfactual Impact
Intervention
Primary Outcome
[Repeat from Lectures 1 AND 2]
Time

8. Impact: What is it? Intervention Primary Outcome Impact Counterfactual
Measuring impact isn’t trivial – not just about change that occurs during the program, but being able to isolate the changes that occurred due to your product or service. figuring out what the change is in comparison to a parallel universe without the product and service.
Impact
Counterfactual
Time

9. Randomised Evaluations
Individuals, villages, or districts are randomly selected to receive the treatment, while other villages serve as a comparison
Groups are Statistically Identical before the Program =
Treatment Group
Village 2
Village 1
Comparison Group
Presentation Tip: (Animation tip: this slide has been animated to show the process of randomisation which is why it looks as it does i.e. with the arrows blocking the picture. When viewed in full-screen mode that does not happen. For the animation to work, please press the “downward” arrow once and the rest of the animation will automatically follow. DO NOT keep pressing the downward arrow, else the animation will not work!)
This slide is optional. This can be used if we want to explain our research methodology
Any Difference at the Endline can be Attributed to the Program
Two groups continue to be identical, except for treatment. Later, compare outcomes (health, test scores) between the two groups. Any differences between the groups can be attributed to the program.

10. How can we use impact evaluation as a tool for gender policy and advocacy?
Surprisingly little hard evidence on what works, coupled with popular notions on how to combat gender issues:
Example: Toilets and gender-based violence
The growing role of impact evaluation in government and international development
Positive evidence encourages funding, advocacy, and programme expansion

11. How can gender be successfully integrated into impact evaluation?
Interventions targeting women and girls with easily quantifiable outcomes (education levels, employment, fertility rates, etc.)
Gender-based or gender-defined outcomes
More difficult, but possible

12. Examples of J-PAL Research: Targeted Interventions
Effect of a merit-based girls' scholarship program on attendance and academic achievement in Kenya
Objective: Increase girls’ enrolment and reduce drop-out in middle school
Program: Girls’ Scholarship Program carried out in randomly selected schools. High ranking girls receive scholarship for school fees, grant to the family for school supplies, public recognition
Findings: Improvements in performance, attendance and parental involvement. Positive externalities also for girls with low chance of winning the award. High ranking girls won the scholarship, but effects were also seen in non-winners.
Program by International Child Support Africa.

13. Examples of J-PAL Research: Targeted Interventions
Menstruation and Education in Nepal
Objective: Increase girls’ enrolment and reduce drop-out in 7th and 8th standard.
Program: Girls’ randomly selected to receive menstrual products and instructions.
Findings: No impact on school attendance, no impact on test scores. Positive impact on time-saved (20 minutes/day).
Project Partner: Institute for Social and Environmental Research (ISER-Nepal)

14. Example of J-PAL Research: Gender-based outcomes
Empowerment of Adolescent Girls in Rural Bangladesh
Objective: Empowerment and holistic development of adolescent girls, increased age of marriage, changes in attitudes and aspirations
Program:
Literacy and numeracy training
Financial skills: entrepreneurship, budgeting, etc.
Food incentive to delay marriage
Outcomes of interest: education, health, nutrition, income generation, savings/investment, decision-making, attitudes, awareness, marital and reproductive outcomes.

15. Empowerment of Adolescent Girls in Rural Bangladesh
Evaluation Approach – How to measure empowerment?
1) In-depth situation analysis by Save the Children
A review of government policies
Qualitative assessment of adolescent girls: What is required for ‘empowerment’
Participation in community & household decision-making
Mobility
Education
Skills for earning an income

16. Empowerment of Adolescent Girls in Rural Bangladesh
Evaluation Approach – How to measure empowerment?
2) Development of survey instruments by J-PAL affiliate and researchers
Comprehensive questions on decision making:
Ability to travel to local event
Medical decisions
Mental health indices
Aspirations
Etc.

17. Gender and Impact Evaluation: Conclusions
Impact Evaluation can be a powerful tool for planning and advocacy
Impact Evaluation does not negate the need for more in-depth gender analyses
It is important to foster cross-disciplinary and interdisciplinary collaborations