1. Module 9 Designing and using EFGR-responsive evaluation indicators
Technical Assistance on Evaluating SDGs: Leave No One Behind
EvalGender+ Network together with UN Women, UNEG and EvalPartners

2. Outline The characteristics of indicators
Developing indicators for evaluation
Gender-responsive indicators
Indicators for assessing reduction of inequalities

3. The characteristics of M&E indicators
An indicator is information that can be collected from existing data sets or from data collected in surveys or using other research methods to provide information on a topic of interest to a monitoring or evaluation or related research activity.

4. Indicators can provide information on:
Social, economic, anthropometric, demographic and other characteristics of a population of interest.
Contextual factors that can affect project implementation or outcomes
Project/program inputs
Project/program implementation
Project/program outputs
Project/program outcomes and impacts

5. The quality and adequacy of indicators can be assessed in terms of:
Relevance – to the topic being measured
Comprehensive – covers all important aspects of the topic
Clear and unambiguous – easy to interpret and use without confusions
Economical – the information is cheap to collect
Accessible – the information is easy to access
Monitorable – the data can be easily collected over time
User friendly – uses widely accepted and easy to use concepts
Stable over time – definitions and collection methods do not change
Comparable across agencies and locations
Ethical – there are no ethical concerns affecting collection or use

6. Construct validity
Constructs are the combination of indicators used to measure the key components or outcomes of a program
e.g. women’s empowerment, reduction in inequality, poverty
It is essential to ensure that the set of indicators adequately measure the construct
Often indicators are drawn from available secondary data and may not cover all important aspects of the construct

7. Examples of weak construct validity
Inequality may only be measured on the basis of income and may not address unequal access to health, education and other services
Women’s empowerment may only be measured in terms of participation in community or political activities and may not address women’s decision-making and control of resources within the household

8. Additional considerations for EFGR indicators
The information is objective and not based on value judgments
When using disaggregated data the information is accurate and reliable
The information is collected from the right person [e.g. not based on the opinion of limited knowledge of a spouse or other household member]
Sensitive information can be collected in a way that does not cause problems for the respondent or interviewer
There should be agreement among stakeholders on the appropriateness of the indicators

9. 2. Developing indicators for evaluation
Step 1: Defining program outcomes
The first step is to produce clear definitions of intended program outcomes
These must be based on the theory of change
Outcomes must adequately cover all of the program goals
Step 2: Identifying indicators that adequately measure all of the outcomes
Assess construct validity
Ensure agreement from key stakeholders on the choice of indicators

10. 2. Developing indicators for evaluation (cont)
Step 3: Assess the adequacy of the indicators (see slide 5) Step 4: Ensure that data can be collected.

11. 3. Gender responsive indicators
Sex disaggregation of standard indicators is useful, but this is only the first stage of defining adequate gender-responsive indicators
Ideally the indicators should be based on a gender-responsive theory of change
Assess the adequacy of the indicators in terms of construct validity
Indicator reliability is a challenge as much of the information covers sensitive topics that respondents may be unwilling to discuss
There is always a risk of under-reporting sensitive topics such as domestic violence, or women’s increasing economic empowerment

12. Going beyond sex-disaggregation of standard survey data
Ideally indicators should also measure
Women’s multiple roles and responsibilities and the resulting time constraints
Women’s access to and control over productive resources
Domestic violence and sexual harassment
Constraints on women’s mobility
Mechanisms of social control and the enforcement of socially defined gender roles [through legal, economic, political and social pressures]

13. 4. Indicators for measuring reduction of inequalities
3 dimensions of inequality must be studied:
The services and activities to which there may be unequal access
The potentially vulnerable groups that may suffer from unequal access
The mechanisms creating or perpetuating inequality

14. The services and activities to which there may be unequal access
These can include
Labor markets and income generating opportunities
Education
Health
Other public services
Communication technology
Freedom of expression
Mobility
Sexual orientation

15. Groups that may suffer inequalities
Men or women
Youth or the elderly
People suffering physical and mental handicap
Particular geographical groups
Undocumented and illegal populations
Ethnic minorities
Groups with particular sexual orientation
Drug users or people with socially disapproved health conditions such as HIV/AIDS

16. Mechanisms for creating or perpetuating inequalities
Social
Legal
Political
Economic
War and violence
Mass media

17. Considerations in defining and applying indicators for assessing reduction of inequality
The are many dimensions of inequality so it is important to identify which dimensions are, and are not available in current the proposed SDG indicators.
The selection of inequality indicators should ideally be based on a theory of change explaining how the SDG will contribute to the reduction of inequalities
It is important to assess the adequacy of the proposed indicators