1. Indicators Regional Workshop on the
Monitoring and Evaluation of HIV/AIDS Programs
February 14 – 24, 2011
New Delhi, India

2. Session Objectives Understand how indicators are linked to frameworks
Describe how to operationalize indicators
Identify the role of indicators at different levels (national, sub-national, project) and the linkages between them
List sources of indicators that are international standards
Select indicators and define indicators for an M&E plan
Speaker Notes
At the end of the session, participants should be able to critique indicators, identify criteria for selection of sound indicators, understand how indicators are linked to the frameworks covered in the previous module, and select indicators and complete what is known as an Indicator Reference Sheet.

3. An Indicator is… a variable (its value changes)
that measures (objective calculation of value)
key elements of a program or project
Inputs, processes, outputs, outcomes
Speaker Notes
An indicator is a variable that measures one aspect of a program or project. Let’s take a moment to go over each piece of this definition. The purpose of indicators typically is to show that a program activities are carried out as planned or that a program activity has caused a change or difference in something else. Therefore an indicator of that change will be something that we reasonably expect to vary. Its value will change from a given or baseline level at the time the program begins, to another value after the program and its activities have had time to make their impact felt, when the variable, or indicator, is calculated again.
Secondly, an indicator is a measurement. It measures the value of the change in meaningful units for program management: a measurement that can be compared to past and future units and values. A metric is the calculation or formula that the indicator is based on. Calculation of the metric establishes the indicator’s objective value at a point in time. Even if the factor itself is subjective, like attitudes of a target population, the indicator metric calculates its value objectively at a given time.
Thirdly, an indicator focuses on a single aspect of a program or project. It may be an input, an output, or an overarching objective, but its related metric will be narrowly defined in a way that captures that aspect as precisely as possible.
A full, complete, and appropriate set of indicators for a given project or program in a given context with given goals and objectives will include at least one indicator for each significant aspect of program activities.

4. Indicators provide critical M&E data at every level (and stage) of program implementation
Inputs, Process
Was the program carried out as planned?
How well was it carried out?
Outputs, Results
Did the expected change occur?
How much change occurred?
Outcome, Impact
Has the outcome changed in desired direction?
Does the change signal program “success”?
A full, complete, and appropriate set of indicators for a given project or program in a given context with given goals and objectives will include at least one indicator for each significant aspect of program activities.

5. Logic Model Indicators
INPUT
Human and financial resources
Development of training materials
PROCESS
Conduct one PMTCT training workshop in each district for providers
OUTPUT
Providers trained in updated PMTCT service provision
OUTCOME
Increased use of PMTCT services
IMPACT
Reduced perinatal transmission of HIV
Indicator: percent of infants HIV+ born to HIV+ women
Indicator: # of providers who have completed clinical training
Speaker Notes
This example demonstrates how indicators are related to logic models.
Indicator: percent of HIV+ women receiving a complete course of ARV prophylaxis
Indicator: % of pregnant women who are HIV tested

6. Results Framework Example – PEPFAR funded ART Program
SO: Increased utilization of ART services
IR-1: Availability of quality
services
IR-2: Demand for services
IR-1.1: Increase ART sites
IR-2.1: Increase knowledge of
ART
IR-1.2: Supply sites with ARVs
IR-2.2: Increase referral from VCT
IR-1.3: Training for providers

7. Results Framework Indicators PEPFAR funded ART Program
IR-1: Availability of quality
services
# of districts with at least one facility providing ART services in line with national standards
# of ART sites
IR-1.1: Increase ART sites
% of ART delivery points experiencing stock-outs in the preceding six months
IR-1.2: Supply sites with ARTs
IR-1.3: Training for providers
# of health workers trained on ART delivery in accordance with national or international standards

8. Results Framework Indicators PEPFAR funded ART Program
SO: Increased utilization of ART services
# of HIV + persons receiving ART therapy

9. District, Facility, Project
Indicator Pyramid
Number of Indicators
Decreases
Increases
Global
Compare countries Overview world-wide situation
National/Sub-national
Assess effectiveness of response
Reflect goals/objectives of national/sub-national response
Speaker Notes
This is an indicator pyramid, showing that indicators at different levels (global, national or sub-national, or district or health facility) are used for different purposes and typically have different numbers of indicator. Different decision-makers demand different types of information. For example, hospital managers may be interested in knowing what the quality and costs of their services are in order to decide what needs to be done to improve them. District managers, on the other hand, may need data on provision and utilization of health services in order to plan further amendments to the numbers and types of such services within their districts. National agencies, on the other hand, may require assessments of coverage to justify further investments in their program. And finally, international agencies may wish to make global comparisons of coverage and impact to understand global trends in health, for advocacy or to justify continued funding. The selection of indicators needs to be tailored to the specific needs of each level of decision-making. The higher up in the pyramid, the fewer the indicators.
Typically, the indicators at the higher levels are linked to those at the lower levels. Data may be collected at the district level and passed up to the national level and on up to the global level.
District, Facility, Project
Identify progress, problems, and challenges

10. Linkages between levels
Often, indicators at the higher levels in the pyramid are linked to those at the lower levels.
Data may be collected at the district level and passed up to the national level and on up to the global level.
Requires an M&E system to support data flow, compilation, and aggregation
Indicators at the national level are linked to those at the lower levels in two distinct ways. Firstly, indicators collected at the lower level (such as the service delivery level) often capture lower-level results. These should be linked logically to indicators at a higher level that reflect the expected effects of these efforts. Secondly, data may be collected at lower administrative levels and be passed up to the national level where they are aggregated in order to track the national response.

11. Characteristics of good indicators
Valid: accurate measure of a behavior, practice or task
Reliable: consistently measurable in the same way by different observers
Comparable: can be measured in different contexts or time periods
Non-directional: subjective criteria not part of definition
Speaker Notes
What makes a good indicator? Fundamentally, good indicators must be valid and reliable measures of the result. The other desirable characteristics listed here all serve in a sense as aids that help guide the design of indicators and metrics toward this ideal or goal of valid, reliable indicators.
Valid: An indicator is valid when it dictates an accurate measurement the activity, output or outcome of the program.
Reliable: An indicator is reliable when it minimizes measurement error, that is when it is possible to measure it consistently over time, regardless of the observer or respondent.
Comparable: Where possible, indicators should be structured using comparable units, denominators, and in other ways that will enable increased understanding of impact or effectiveness across different population groups or program approaches.
Non-directional: Indicators should be constructed to allow variation in either/any direction, for example: increases and decreases.
Precise: Indicators should be operationalized with clear, well-specified definitions.
Measureable: It must be possible to quantify your indicator using tools and methods that are available.
Timely: Indicators should be measured at appropriate intervals relevant in terms of program goals and activities.
Programmatically important:

12. Characteristics of good indicators (cont.)
Precise: operationally defined in clear terms
Measurable: quantifiable using available tools and methods
Timely: provides a measurement at time intervals relevant and appropriate in terms of program goals and activities
Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact
Speaker Notes
What makes a good indicator? Fundamentally, good indicators must be valid and reliable measures of the result. The other desirable characteristics listed here all serve in a sense as aids that help guide the design of indicators and metrics toward this ideal or goal of valid, reliable indicators.
Valid: An indicator is valid when it dictates an accurate measurement the activity, output or outcome of the program.
Reliable: An indicator is reliable when it minimizes measurement error, that is when it is possible to measure it consistently over time, regardless of the observer or respondent.
Comparable: Where possible, indicators should be structured using comparable units, denominators, and in other ways that will enable increased understanding of impact or effectiveness across different population groups or program approaches.
Non-directional: Indicators should be constructed to allow variation in either/any direction, for example: increases and decreases.
Precise: Indicators should be operationalized with clear, well-specified definitions.
Measureable: It must be possible to quantify your indicator using tools and methods that are available.
Timely: Indicators should be measured at appropriate intervals relevant in terms of program goals and activities.
Programmatically important:

13. Common Indicator Metrics
Counts
# of providers trained
# of condoms distributed
Calculations: percentages, rates, ratios
% of facilities with trained provider
% of FSW who used a condom at last sex with a client
Index, composite measures
Index on infection control prevention
DALY (Disability Adjusted Life Years)
Thresholds
Presence, absence
Pre-determined level or standard
Speaker Notes
Perhaps the most important part of what comprises an indicator is the metric. The metric is the precise explanation of the data and the calculation that will give the measurement or value of the indicator. In other words, it specifies the data that will be used to generate the value, and how the data elements will be manipulated to come up with a value.
Defining good metrics is absolutely crucial to the usefulness of any M&E plan. A good metric clarifies the single dimension of the result that is being measured by the indicator. A good metric does this in such a way that each value measured for the indicator is exactly comparable to values measured at another time.
Indicators can have a number of types of metrics. They can be simple counts of things (for example, the number or providers trained or the number of condoms distributed), or they can involve calculations (for example, the proportion of facilities with a trained provider, maternal mortality ratio, total fertility rate). They can also be more complex, such as an index comprising of the sum of scores on six quality outcomes or the DALY or Disability Adjusted Life Year. The DALY has become a commonly used measurement for the burden of disease to show the total amount of healthy life lost, whether from premature mortality or from some degree of disability, during a period of time. These are just some examples of types of metrics used for indicators.
Additional Speaker Notes
The purpose of this slide is to be sure participants understand what the term “metric” refers to and to help them recognize common formats of indicators in. This slide does not intend to present recommended or good indicators, or to cover all types of metrics used in calculating indicators.

14. Operationalizing indicators
Establish exactly how a given concept / behavior
will be measured
precise definition and metric
how the value will be reliably calculated
(anyone using the same data will arrive at exactly the same indicator value)

15. Always specify the details!
who/what qualifies to be counted, and when
Number of providers trained in PMTCT services
“providers”: any clinician providing direct clinical services to women seeking ANC at government health facilities
“trained”: attended a two-week training course on PMTCT in past year
Speaker Notes
Behind an indicator metric is information important to its correct calculation. This will be addressed later as well, but it is important to understand what goes into an indicator before we move on to discussing how they are selected or developed.
Here are two examples:
In the first example, the indicator is the number of providers trained under a given program aiming to improve PMTCT services. For this example, let’s assume that part of the program includes training of providers. It is important to note that “providers” are defined as any clinician (meaning doctor, nurse or medical assistant) providing direct clinical services to clients seeking antenatal care at public health facilities. Note that any providers working at private facilities are not to be included in this indicator. If the term weren’t defined, the indicator could be counted or interpreted differently than intended. Also note that in order to be counted for this indicator, the clinician has to have attended a two-week training course. This means that this information must be collected at the time of the training workshops.
In the second example, the indicator is the proportion of facilities with a provider trained in PMTCT. To calculate this indicator, you need a numerator and a denominator. (Remember that a numerator is divided by a denominator to carry out a calculation.) The numerator is the number of public facilities with a provider who attended the full five days of the ANC training offered by the program. Note that the numerator specifies that the facilities are public and that the providers must have attended all five days in order to be counted. This information was not included in the indicator itself. The denominator specifies that the number of public facilities offering ANC services be known. In this example, it becomes clear that it is necessary that information about the facility at which the person attending the training works be recorded at the time of the training. If this information is not collected, it will not be possible to calculate this indicator without going to each facility to ask about provider training. It will also be necessary to gather information about total number of public facilities that provide ANC in the target area.
Additional Speaker Notes
The purpose of this slide is to be sure participants understand what goes into a metric and to introduce the terms “numerator” and “denominator” into the discussion. This slide does not intend to present recommended or good indicators.

16. Always specify the details!
How to calculate it
Percent of health facilities with a provider trained in PMTCT services
“Numerator”: Number of public facilities with a provider who attended a two-week PMTCT training course in past year
“Denominator”: Number of public facilities offering ANC services in the past year
Speaker Notes
Behind an indicator metric is information important to its correct calculation. This will be addressed later as well, but it is important to understand what goes into an indicator before we move on to discussing how they are selected or developed.
Here are two examples:
In the first example, the indicator is the number of providers trained under a given program aiming to improve PMTCT services. For this example, let’s assume that part of the program includes training of providers. It is important to note that “providers” are defined as any clinician (meaning doctor, nurse or medical assistant) providing direct clinical services to clients seeking antenatal care at public health facilities. Note that any providers working at private facilities are not to be included in this indicator. If the term weren’t defined, the indicator could be counted or interpreted differently than intended. Also note that in order to be counted for this indicator, the clinician has to have attended a two-week training course. This means that this information must be collected at the time of the training workshops.
In the second example, the indicator is the proportion of facilities with a provider trained in PMTCT. To calculate this indicator, you need a numerator and a denominator. (Remember that a numerator is divided by a denominator to carry out a calculation.) The numerator is the number of public facilities with a provider who attended the full five days of the ANC training offered by the program. Note that the numerator specifies that the facilities are public and that the providers must have attended all five days in order to be counted. This information was not included in the indicator itself. The denominator specifies that the number of public facilities offering ANC services be known. In this example, it becomes clear that it is necessary that information about the facility at which the person attending the training works be recorded at the time of the training. If this information is not collected, it will not be possible to calculate this indicator without going to each facility to ask about provider training. It will also be necessary to gather information about total number of public facilities that provide ANC in the target area.
Additional Speaker Notes
The purpose of this slide is to be sure participants understand what goes into a metric and to introduce the terms “numerator” and “denominator” into the discussion. This slide does not intend to present recommended or good indicators.

17. Always specify the details!
Threshold indicator for VCT program
VCT centers with minimum conditions to provide quality services
Facility provides quality VCT services if the following necessary structural elements are present
Trained staff
Adequate privacy for counseling
Systems for maintaining confidentiality
Directory of services for referral
Adequate conditions for ensuring quality control of specimen tests
IN THIS EXAMPLE, EACH OF THE ELEMENTS OF QUALITY VCT SERVICES IN THE LIST WOULD NEED TO BE SPECIFICALLY DEFINED. “ADEQUATE PRIVACY FOR COUNSELING” MIGHT BE DEFINED AS THE EXISTANCE OF A PLACE FOR DOING COUNSELING WHICH KEEPS PASSERS-BY FROM SEEING OR HEARING WHAT GOES ON INSIDE. OTHER PRIVACY-PROTECTING MEASURES MIGHT ALSO BE INCLUDED IN THE DEFINITION, SUCH AS THE LOCATION OF THE ENTRANCE A TO VCT SERVICE SITE.

18. Limitations All indicators have limitations, even those commonly used:
Blood safety: cannot monitor private facilities adequately
Sexual behavior (e.g. condom use, number of partners): self reporting bias
Sero-surveillance: get biased population (pregnant women and other populations)
Population-based HIV prevalence: refusal bias , sampling bias
Speaker Notes
All indicators can be scrutinized for their strengths and limitations. The strengths are typically that they meet the criteria discussed previously. Although some indicators are better than others, even those most widely used have their drawbacks.
Discussion
Many sexual behavior indicators are used in HIV/AIDS prevention programs, and because of the sensitive nature of the questions, self-presentation bias is an inherent issue. That is, people may not report true information.

19. Frequency of Reporting on Indicators
Input/Process: Continuously
Output: Quarterly, semi-annually, or annually
Outcome: 1-3 years
Impact: 2-5 years

20. Managing indicator systems Basic Indicator Matrix
Data Source
Frequency
Decision Points/Comments
Outputs
# of peer outreach contacts in the past 12 month
Program records
Quarterly
Disaggregate by district
Outcomes
% of FSW who used a condom at last sex with a client
Target group survey
1-3 years
Disaggregate by age and type (brothel, non-brothel based)
Speaker Notes:
There are a couple of sections of M&E plans that are specific to indicators and are very important.
One is the indicator matrix that lists all indicators by level (outputs, intermediate outputs, and outcomes). This matrix should also contain information about the data source and the frequency the indicator will be calculated. Decisions that have yet to be made, as well as the most pertinent comments, should also be listed in the matrix.
This example is very abbreviated since it only shows two indicators, each at a different level. In the handouts you received for this module, you will find an example of an indicator matrix. Indicators are typically organized in a logical order based on the framework used in the M&E plan. In this handout, the “output” or program area is labeled above each section of indicators. If using a results framework, the matrix could be organized by intermediate or sub-intermediate result.

21. The Role of Target Setting
Important element of strategic planning
Tracking progress towards achieving targets assists with resource allocation and improves program management
Often a requirement for performance-based funding

22. Main Steps in Target Setting
Define the populations and subpopulations in need of services (treatment, diagnostic, care, prevention, etc)
Assess existing coverage to identify gaps
Determine potential achievements within the time frame given resource availability, opportunities, and constraints
Set ambitious but realistic targets!

23. Indicator Reference Sheets
Assist in detailed documentation of indicators
Clear definition of indicator, numerator, and denominator
Collection details – data source, frequency, who responsible

24. Content of Indicator reference sheets (see handout)
The linked result
Precise definitions for every term used
Reason for selection, how to interpret
Data requirements, data quality issues
How to measure and calculate
Data source, responsibilities, and frequency
Issues, limitations, significance
Data table (baseline and target values)

25. Guiding Principles to selecting indicators
Ensure that the indicators are linked to the program goals and are able to measure change
Ensure that standard indicators are used to the extent possible
Consider the cost and feasibility of data collection and analysis.
Keep the number of indicators to the minimum and include only those needed for program and management decisions or for reporting.

26. Good indicators: A Synopsis
Provide information useful for program decision-making
Are consistent with international standards and other reporting requirements, as appropriate
Are defined in clear and unambiguous terms
Non-directional, “independent,” and SMART
Have values that are:
Easy to interpret and explain
Precise, valid and reliable measures
Comparable across relevant population groups, geography, other program factors, as needed

27. Indicator Guides
National AIDS Programmes. A guide to monitoring and evaluation. UNAIDS (2000)
National AIDS Programs. A guide to indicators for monitoring and evaluating national HIV/AIDS prevention programs for young people. WHO (2004).
National guide to monitoring and evaluating programmes for the prevention of HIV in infants and young children. UNAIDS/WHO (2004)
National AIDS Programs. A guide to monitoring and evaluating HIV/AIDS care and support. WHO (2004).
National AIDS Programmes. A guide to indicators for monitoring and evaluating national antiretroviral programmes. WHO (2005).
Speaker Notes:
Given all the characteristics a good indicator should have, and since it is not necessary to re-create the wheel for every M&E plan, keep in mind that there are pre-existing sources of indicators.
An obvious source is from past years of a program. Because monitoring program progress over time is a key objective of an M&E system, it is imperative to continue to use the same indicators if they meet the characteristics of a good indicator as described previously. However, improvements to indicators should be made when necessary. If different indicators are used over the life of a program, a decision must be made to either stop collecting/reporting the old indicator and begin with the new, or to continue with the old and add the new, noting the difference in the M&E plan or in reports.
Another source is related or similar programs. The ability to compare your program outcomes with others may be a desirable feature, so when indicators used in other programs would also be useful in yours, you may want to consider those for your M&E plan.
Finally, there are lists of global indicators that should be consulted. Some international organizations produce guides for M&E and include recommended indicators. Often M&E and subject-matter experts take part in working groups led by these organizations to compile lists of indicators, and these guides are typically updated from time to time. In some cases these guides also include data collection instruments that have been field tested.
The Millennium Development Goals are a set of targets for the year 2015 that the 191 UN member states have agreed to. They include goals related to child mortality, maternal health, HIV/AIDS, malaria and other diseases.
UNAIDS and WHO have taken the lead to standardize HIV/AIDS indicators. In 2000 the first guide to program M&E for HIV/AIDS included the first set of indicators.

28. Indicator Guides (cont.)
Guide to monitoring and evaluation of the national response for children orphaned and made vulnerable by HIV/AIDS. UNAIDS/WHO (2005)
A framework for monitoring and evaluating HIV prevention programs for most-at-risk populations (UNAIDS 2007)
Monitoring and Evaluation Toolkit. HIV/AIDS, Tuberculosis, and Malaria. GFTAM (2009).
Monitoring the Declaration of Commitment on HIV/AIDS. Guidelines on construction of core indicators. UNAIDS (2009)
….and others!
Speaker Notes:
Given all the characteristics a good indicator should have, and since it is not necessary to re-create the wheel for every M&E plan, keep in mind that there are pre-existing sources of indicators.
An obvious source is from past years of a program. Because monitoring program progress over time is a key objective of an M&E system, it is imperative to continue to use the same indicators if they meet the characteristics of a good indicator as described previously. However, improvements to indicators should be made when necessary. If different indicators are used over the life of a program, a decision must be made to either stop collecting/reporting the old indicator and begin with the new, or to continue with the old and add the new, noting the difference in the M&E plan or in reports.
Another source is related or similar programs. The ability to compare your program outcomes with others may be a desirable feature, so when indicators used in other programs would also be useful in yours, you may want to consider those for your M&E plan.
Finally, there are lists of global indicators that should be consulted. Some international organizations produce guides for M&E and include recommended indicators. Often M&E and subject-matter experts take part in working groups led by these organizations to compile lists of indicators, and these guides are typically updated from time to time. In some cases these guides also include data collection instruments that have been field tested.
The Millennium Development Goals are a set of targets for the year 2015 that the 191 UN member states have agreed to. They include goals related to child mortality, maternal health, HIV/AIDS, malaria and other diseases.
UNAIDS and WHO have taken the lead to standardize HIV/AIDS indicators. In 2000 the first guide to program M&E for HIV/AIDS included the first set of indicators.

29. Additional References
HIV/AIDS Survey Indicators Database
UNAIDS Indicator Standards: Operational Guidelines for Selecting Indicators for the HIV Response.
UNAIDS Indicator Registry

30. Not everything that can be counted counts, and not everything that counts can be counted.
Albert Einstein

31. Group Activity: Adding Indicators to a Framework
Assemble group-work members
Use the worksheet “Linking Indicators to Frameworks”
Take about one hour to complete sheet
Resulting framework with indicators will to presented back by a group representative (5 minutes for each group)
The selection of indicators will form the basis of your plan, You will have the opportunity to revise this further based on feedback

32. MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for Health,
and Tulane University. Views expressed in this presentation do not
necessarily reflect the views of USAID or the U.S. Government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.