1. Monitoring and Evaluation Systems for NARS Organisations in Papua New Guinea
Day 2. Session 6.
Developing indicators

2. Learning objectives
By the end of this session, participants will be able to:
Explain what indicators are and how they are used in M&E
Describe how quality indicators are defined and formulated.
Use criteria to assess the suitability of indicators to measure the success of an intervention.
Present with appreciation the logical framework as a tool linking planning and M&E.

3. Developing indicators
Activity that overlaps the planning process and the design of the M&E arrangements for an intervention.
Indicators are usually first developed during the planning process for an intervention.
Designers of the M&E system must ensure that the planning process has delivered high quality indicators.
Designing the M&E system requires an assessment of the quality of the indicators, sometimes even developing the indicators.

4. What are indicators?
data or signs that allow the objective verification of the achievement of objectives
signal progress towards the objectives
measures of achievement, change or performance
set the reference point for monitoring
record how the achievement of each objective is measured
(adapted from Bojanic et al. 1995).

5. Developing indicators
Task 1:
Clarify the objectives or results and identify what needs to be measured. What does success look like for the intervention that is being measured?
Task 2:
Brainstorm ideas and prepare a list of possible indicators for the objectives.
Task 3:
Assess each potential indicator from a technical perspective using indicator selection criteria.
Task 4:
Select the most relevant indicators based on the assessment (Task 3)
Task 5:
Define indicators and finalize the indicator selection.
(adapted from Görgens and Kusek 2009)
2.6.5 5

6. Criteria to assess indicators
Measurable:
There must be some practical way to quantify or measure the indicator, either in a quantitative (numerical) way or in a qualitative (descriptive) way.describe in detail the intervention’s strategy
NO! People’s feelings about the elections
YES! Percentage of the population who voted
Note: There must be some practical way to quantify or measure the indicator, either in a quantitative (numerical) way or in a qualitative (descriptive) way.

7. Criteria to assess indicators
Practical:
It must be possible to collect, process and analyse data in time and at reasonable cost
NO! Number of targeted population who understand
their voting rights (census)
YES! Percentage of targeted population who understand
their voting rights (representative sample, through
a poll)

8. Criteria to assess indicators
Reliable:
The indicators should be open to independent validation. The values of the indicators should be reliable and comparable over time when collected using the same methods.
NO! Number of people receiving quality HIV/AIDS care
and support services through workplace programs
YES! Number of people who were tested for HIV at work
in the last 12 months

9. Criteria to assess indicators
Relevant:
Indicators must be appropriate for the objective to be measured. They must be caused by at least in part to the activities of the intervention to be monitored.
NO! Agriculture production yield in the country
YES! Agriculture production yield in the district where the
project is being implemented

10. Criteria to assess indicators
Objective:
Indicators must be unambiguous about 1) what is being measured and 2) what data are being collected.
Ensure everyone agrees on what they mean. For example:
what is meant by ‘improved service delivery?’
an indicator may be ‘the area of degraded land’ but what criteria will be used to classify such land?
for households what is included in ‘farm income’ and what in ‘non-farm income?
NO! Number of expanding and successful parent/teacher associations
YES! Number of parent/teacher associations experiencing an annual increase in membership of at least 5%

11. Criteria to assess indicators
Useful to management:
Information provided by the measure is critical to decision making. Avoid collecting and reporting information that is not used to support program management decisions.
LEVEL OF INSTITUTIONAL CAPACITY
NO! Number of computers; number of staff meetings
YES! Amount by type of resources mobilized
YES! Number by type of critical management systems fully operational

12. Criteria to assess indicators
Direct:
An indicator should measure as closely as possible the result it is intended to measure; e.g., yield of green beans per hectare is a direct measure of the result of efforts to increase coffee production. Proxy indicators might be appropriate.
INCREASED VARIETY IN AGRICULTURE PRODUCTION
NO! Number of types of agriculture seeds distributed
YES! Volume of production by type of agricultural product

13. Formulating indicators
a target or other stakeholder group (who?)
a descriptive definition of the quality (what?) Q
an indication of amount or quantity - percentage, number, ratio (how much?) Q
an indication of time (when?) T
an indication of location in space (where?)
2.6.13 13

14. Example of an indicator
Hierarchy of Objectives
Narrative Summary
Indicator
Goal
In 2015 health and education of citizens of Lowland Region have improved
The percentage of citizens in the Lowland Region with adequate nutrition, as defined by WHO (World Health Organisation) standards, has increased from 50% in 2003 to 70% in 2015

15. Example of presenting an indicator with separate baseline and target values
Hierarchy of Objectives
Narrative Summary
Indicator
Baseline
Target
Goal
In 2015 the health and education of citizens of Lowland Region have improved
The percentage of citizens in Lowland Region with adequate nutrition, as defined by WHO (World Health Organisation) standards
50% in 2004
70% in 2015

16. Baseline data
information describing the initial starting point or the situation before an intervention has taken place
collected either before or in the early stages of an intervention
describe the conditions against which comparisons are made at a later stage of the intervention during M&E
allows assessing the contribution that an intervention has made to a particular situation in addition to what the situation was like before the intervention
2.6.16 16

17. Baseline data
determine whether baseline values for the indicators exist
if not, design the baseline and its implementation process during the intervention’s planning phase
include activities and funding necessary to collect the baseline data in the work plan and the budget of the intervention
record baseline values to be retrieved later
additional reading on baselines in Handout 2.6.9
2.6.17 17

18. Defining indicators A good definition of an indicator includes:
The indicator is stated in clear language, precisely and unambiguous.
The method of measurement for the indicator is clearly defined.
The data sources or data collection method for the indicator data is clearly stated (means of verification).
2.6.18 18

19. The logical framework
developed during the planning phase of an intervention is usually the first record of the indicators and means of verification/data sources/methods of data collection
a tool to summarize the design of the intervention - the intervention’s strategy
a tool to summarize the M&E arrangements of the intervention
the logical framework is a tool linking planning to M&E
2.6.19 19

20. A logical framework (logframe) matrix
Hierarchy of objectives
Narrative summary
Objectively verifiable indicators
Means of verification/Data sources/Data collection methods
Important assumptions
Goal
In 2015 health and education of citizens of Lowland Region have improved
The percentage of citizens in the Lowland Region with adequate nutrition has increased from 50% in 2003 to 70% in 2015
Nutritional survey of a representative sample of the citizens of Lowland Region (baseline in 2004, repeat in 2015)
Purpose
Outputs
Activities
Inputs:
2.6.20 20

21. The logical framework not the M&E plan for the intervention
M&E arrangements have to be described in a more detailed M&E plan
specifying the data sources and data collection methods
the frequency and timing of data collection
the responsibilities for data collection, storage and analysis
the reporting arrangements
2.6.21 21

22. A logical framework (logframe) matrix
Hierarchy of objectives
Narrative summary
Objectively verifiable indicators
Means of verification/Data sources/Data collection methods
Important assumptions
Goal
In 2015 health and education of citizens of Lowland Region have improved
The percentage of citizens in the Lowland Region with adequate nutrition, as defined by WHO standards, has increased from 50% in 2004 to 70% in 2015
Nutritional survey, following WHO guidelines, of a representative sample of the citizens of Lowland Region (baseline in 2004, repeat in 2015)
Purpose
Outputs
Activities
Inputs:
Thank you!
2.6.22 22