Module 9 Designing and using EFGR-responsive evaluation indicators Technical Assistance on Evaluating SDGs: Leave No One Behind EvalGender+ Network together with UN Women, UNEG and EvalPartners

Outline The characteristics of indicators Developing indicators for evaluation Gender-responsive indicators Indicators for assessing reduction of inequalities

The characteristics of M&E indicators An indicator is information that can be collected from existing data sets or from data collected in surveys or using other research methods to provide information on a topic of interest to a monitoring or evaluation or related research activity.

Indicators can provide information on: Social, economic, anthropometric, demographic and other characteristics of a population of interest. Contextual factors that can affect project implementation or outcomes Project/program inputs Project/program implementation Project/program outputs Project/program outcomes and impacts

The quality and adequacy of indicators can be assessed in terms of: Relevance – to the topic being measured Comprehensive – covers all important aspects of the topic Clear and unambiguous – easy to interpret and use without confusions Economical – the information is cheap to collect Accessible – the information is easy to access Monitorable – the data can be easily collected over time User friendly – uses widely accepted and easy to use concepts Stable over time – definitions and collection methods do not change Comparable across agencies and locations Ethical – there are no ethical concerns affecting collection or use

Construct validity Constructs are the combination of indicators used to measure the key components or outcomes of a program e.g. women’s empowerment, reduction in inequality, poverty It is essential to ensure that the set of indicators adequately measure the construct Often indicators are drawn from available secondary data and may not cover all important aspects of the construct

Examples of weak construct validity Inequality may only be measured on the basis of income and may not address unequal access to health, education and other services Women’s empowerment may only be measured in terms of participation in community or political activities and may not address women’s decision-making and control of resources within the household

Additional considerations for EFGR indicators The information is objective and not based on value judgments When using disaggregated data the information is accurate and reliable The information is collected from the right person [e.g. not based on the opinion of limited knowledge of a spouse or other household member] Sensitive information can be collected in a way that does not cause problems for the respondent or interviewer There should be agreement among stakeholders on the appropriateness of the indicators

2. Developing indicators for evaluation Step 1: Defining program outcomes The first step is to produce clear definitions of intended program outcomes These must be based on the theory of change Outcomes must adequately cover all of the program goals Step 2: Identifying indicators that adequately measure all of the outcomes Assess construct validity Ensure agreement from key stakeholders on the choice of indicators

2. Developing indicators for evaluation (cont) Step 3: Assess the adequacy of the indicators (see slide 5) Step 4: Ensure that data can be collected.

3. Gender responsive indicators Sex disaggregation of standard indicators is useful, but this is only the first stage of defining adequate gender-responsive indicators Ideally the indicators should be based on a gender-responsive theory of change Assess the adequacy of the indicators in terms of construct validity Indicator reliability is a challenge as much of the information covers sensitive topics that respondents may be unwilling to discuss There is always a risk of under-reporting sensitive topics such as domestic violence, or women’s increasing economic empowerment

Going beyond sex-disaggregation of standard survey data Ideally indicators should also measure Women’s multiple roles and responsibilities and the resulting time constraints Women’s access to and control over productive resources Domestic violence and sexual harassment Constraints on women’s mobility Mechanisms of social control and the enforcement of socially defined gender roles [through legal, economic, political and social pressures]

4. Indicators for measuring reduction of inequalities 3 dimensions of inequality must be studied: The services and activities to which there may be unequal access The potentially vulnerable groups that may suffer from unequal access The mechanisms creating or perpetuating inequality

The services and activities to which there may be unequal access These can include Labor markets and income generating opportunities Education Health Other public services Communication technology Freedom of expression Mobility Sexual orientation

Groups that may suffer inequalities Men or women Youth or the elderly People suffering physical and mental handicap Particular geographical groups Undocumented and illegal populations Ethnic minorities Groups with particular sexual orientation Drug users or people with socially disapproved health conditions such as HIV/AIDS

Mechanisms for creating or perpetuating inequalities Social Legal Political Economic War and violence Mass media

Considerations in defining and applying indicators for assessing reduction of inequality The are many dimensions of inequality so it is important to identify which dimensions are, and are not available in current the proposed SDG indicators. The selection of inequality indicators should ideally be based on a theory of change explaining how the SDG will contribute to the reduction of inequalities It is important to assess the adequacy of the proposed indicators