Monitoring and Evaluation Systems for NARS Organisations in Papua New Guinea Day 2. Session 6. Developing indicators

Learning objectives By the end of this session, participants will be able to: Explain what indicators are and how they are used in M&E Describe how quality indicators are defined and formulated. Use criteria to assess the suitability of indicators to measure the success of an intervention. Present with appreciation the logical framework as a tool linking planning and M&E.

Developing indicators Activity that overlaps the planning process and the design of the M&E arrangements for an intervention. Indicators are usually first developed during the planning process for an intervention. Designers of the M&E system must ensure that the planning process has delivered high quality indicators. Designing the M&E system requires an assessment of the quality of the indicators, sometimes even developing the indicators.

What are indicators? data or signs that allow the objective verification of the achievement of objectives signal progress towards the objectives measures of achievement, change or performance set the reference point for monitoring record how the achievement of each objective is measured (adapted from Bojanic et al. 1995).

Developing indicators Task 1: Clarify the objectives or results and identify what needs to be measured. What does success look like for the intervention that is being measured? Task 2: Brainstorm ideas and prepare a list of possible indicators for the objectives. Task 3: Assess each potential indicator from a technical perspective using indicator selection criteria. Task 4: Select the most relevant indicators based on the assessment (Task 3) Task 5: Define indicators and finalize the indicator selection. (adapted from Görgens and Kusek 2009) 2.6.5 5

Criteria to assess indicators Measurable: There must be some practical way to quantify or measure the indicator, either in a quantitative (numerical) way or in a qualitative (descriptive) way.describe in detail the intervention’s strategy NO! People’s feelings about the elections YES! Percentage of the population who voted Note: There must be some practical way to quantify or measure the indicator, either in a quantitative (numerical) way or in a qualitative (descriptive) way.

Criteria to assess indicators Practical: It must be possible to collect, process and analyse data in time and at reasonable cost NO! Number of targeted population who understand their voting rights (census) YES! Percentage of targeted population who understand their voting rights (representative sample, through a poll)

Criteria to assess indicators Reliable: The indicators should be open to independent validation. The values of the indicators should be reliable and comparable over time when collected using the same methods. NO! Number of people receiving quality HIV/AIDS care and support services through workplace programs YES! Number of people who were tested for HIV at work in the last 12 months

Criteria to assess indicators Relevant: Indicators must be appropriate for the objective to be measured. They must be caused by at least in part to the activities of the intervention to be monitored. NO! Agriculture production yield in the country YES! Agriculture production yield in the district where the project is being implemented

Criteria to assess indicators Objective: Indicators must be unambiguous about 1) what is being measured and 2) what data are being collected. Ensure everyone agrees on what they mean. For example: what is meant by ‘improved service delivery?’ an indicator may be ‘the area of degraded land’ but what criteria will be used to classify such land? for households what is included in ‘farm income’ and what in ‘non-farm income? NO! Number of expanding and successful parent/teacher associations YES! Number of parent/teacher associations experiencing an annual increase in membership of at least 5%

Criteria to assess indicators Useful to management: Information provided by the measure is critical to decision making. Avoid collecting and reporting information that is not used to support program management decisions. LEVEL OF INSTITUTIONAL CAPACITY NO! Number of computers; number of staff meetings YES! Amount by type of resources mobilized YES! Number by type of critical management systems fully operational

Criteria to assess indicators Direct: An indicator should measure as closely as possible the result it is intended to measure; e.g., yield of green beans per hectare is a direct measure of the result of efforts to increase coffee production. Proxy indicators might be appropriate. INCREASED VARIETY IN AGRICULTURE PRODUCTION NO! Number of types of agriculture seeds distributed YES! Volume of production by type of agricultural product

Formulating indicators a target or other stakeholder group (who?) a descriptive definition of the quality (what?) Q an indication of amount or quantity - percentage, number, ratio (how much?) Q an indication of time (when?) T an indication of location in space (where?) 2.6.13 13

Example of an indicator Hierarchy of Objectives Narrative Summary Indicator Goal In 2015 health and education of citizens of Lowland Region have improved The percentage of citizens in the Lowland Region with adequate nutrition, as defined by WHO (World Health Organisation) standards, has increased from 50% in 2003 to 70% in 2015

Example of presenting an indicator with separate baseline and target values Hierarchy of Objectives Narrative Summary Indicator Baseline Target Goal In 2015 the health and education of citizens of Lowland Region have improved The percentage of citizens in Lowland Region with adequate nutrition, as defined by WHO (World Health Organisation) standards 50% in 2004 70% in 2015

Baseline data information describing the initial starting point or the situation before an intervention has taken place collected either before or in the early stages of an intervention describe the conditions against which comparisons are made at a later stage of the intervention during M&E allows assessing the contribution that an intervention has made to a particular situation in addition to what the situation was like before the intervention 2.6.16 16

Baseline data determine whether baseline values for the indicators exist if not, design the baseline and its implementation process during the intervention’s planning phase include activities and funding necessary to collect the baseline data in the work plan and the budget of the intervention record baseline values to be retrieved later additional reading on baselines in Handout 2.6.9 2.6.17 17

Defining indicators A good definition of an indicator includes: The indicator is stated in clear language, precisely and unambiguous. The method of measurement for the indicator is clearly defined. The data sources or data collection method for the indicator data is clearly stated (means of verification). 2.6.18 18

The logical framework developed during the planning phase of an intervention is usually the first record of the indicators and means of verification/data sources/methods of data collection a tool to summarize the design of the intervention - the intervention’s strategy a tool to summarize the M&E arrangements of the intervention the logical framework is a tool linking planning to M&E 2.6.19 19

A logical framework (logframe) matrix Hierarchy of objectives Narrative summary Objectively verifiable indicators Means of verification/Data sources/Data collection methods Important assumptions Goal In 2015 health and education of citizens of Lowland Region have improved The percentage of citizens in the Lowland Region with adequate nutrition has increased from 50% in 2003 to 70% in 2015 Nutritional survey of a representative sample of the citizens of Lowland Region (baseline in 2004, repeat in 2015) Purpose Outputs Activities Inputs: 2.6.20 20

The logical framework not the M&E plan for the intervention M&E arrangements have to be described in a more detailed M&E plan specifying the data sources and data collection methods the frequency and timing of data collection the responsibilities for data collection, storage and analysis the reporting arrangements 2.6.21 21

A logical framework (logframe) matrix Hierarchy of objectives Narrative summary Objectively verifiable indicators Means of verification/Data sources/Data collection methods Important assumptions Goal In 2015 health and education of citizens of Lowland Region have improved The percentage of citizens in the Lowland Region with adequate nutrition, as defined by WHO standards, has increased from 50% in 2004 to 70% in 2015 Nutritional survey, following WHO guidelines, of a representative sample of the citizens of Lowland Region (baseline in 2004, repeat in 2015) Purpose Outputs Activities Inputs: Thank you! 2.6.22 22