111 CINDI PMER Workshop March, 2007 By Simon Opolot © 2007 Office of the Premier, KwaZulu-Natal Province, Private Bag X9037 Telecom House, 2nd Floor, Room Langalibalele Street, Pietermaritzburg 3200, South Africa All rights reserved March 13, 2007 FEELING THE PULSE OF THE PROJECT FROM INCEPTION TOWARDS MEETINGS ITS OBJECTIVES/GOALS/COMPLETION – TOWARDS IMPACT Feeling the Pulse of the Project as it moves from Inception towards Meeting its Objectives/Goals/Completion – Towards Impact!

222 THE RATIONALE BEHIND INDICATOR USE Because they can be a measurement, a number, a fact, an opinion or a perception that points at a specific condition or situation being targeted by the intervention, and measure changes in that condition or situation over time: Indicators are front-line instruments in monitoring and evaluating HIV/AIDS interventions and other development work Indicators enable us to assess where we stand and where we are going with respect to values and goals, and to evaluate specific programs and determine their impact. Indicators provide a close look at the results of interventions Using indicators enables us to feel the pulse of the intervention/project as it moves from inception towards meeting its objectives/goals/completion – Towards Impact!! March 13, 2007

333 Risk/Enabling Indicators - Experience shows that at all stages of its cycle a project/an intervention may be affected by a variety of risks or enabling features. So by Risk/Enabling indicators we mean those factors external to a project that contribute to the project's success or failure. E.g. in interventions designed to mobilize and strengthen support for OVCs, indicators of risk would be the attitude of local communities. Input indicators (also called Resource Indictors) - relate to the resources devoted to the intervention/ project. E.g. funding, human and non-human resources, infrastructure, institution- building, and other means by which the intervention/project is put into effect. These indicators play an important role in flagging potential problems and identifying their causes. However, input indicators alone will not reveal whether or not the intervention/ project will be a success. USING INDICATORS IN A CHAIN, FROM INPUT THROUGH TO OUTCOME/IMPACT. March 13, 2007

444 Process indicators (also called "throughput" or "activity" indicators) - reflect delivery of resources devoted to the intervention/ project on an on-going basis. As such, they are the best indicators of implementation and are used for project monitoring. However, while they reflect achievement of results, they should not displace measures of distal outcomes (impact). – A process may be successful at the same time as the outcome/impact is a failure, as noted in this piece of cynical folk wisdom, "the operation was a success, but the patient died," Output indicators – Often used in project evaluations, but are less useful than outcome indicators as they do not track distal results. This is because output indicators measure intermediate results concerning products and services that are delivered when a program or project is completed, but not longer-term results. One of the most important tasks in use of indicators is to carry out evaluation at the outcome as well as the output level. USING INDICATORS IN A CHAIN, FROM INPUT THROUGH TO OUTCOME/IMPACT Cont … March 13, 2007

555 USING INDICATORS IN A CHAIN, FROM INPUT THROUGH TO OUTCOME/IMPACT Cont … Outcome indicators - concern the effectiveness, often long-term, of the intervention/project as judged by the measurable change achieved in improving the quality of life of beneficiaries. They are also known as "impact" indicators. In most cases, the primary emphasis in using indicators should be on outcome, because this best measures distal results

666 SOME QUESTIONS TO ASK AND ANSWER What are the various service areas your HIV/AIDS interventions are addressing? How were the indicators you are currently using developed? If they were developed in a non- participatory fashion, using expertise and knowledge from outside – chances are that those indicators have not been properly understood – Intervention/project does have the pulse. Externally developed indicators should be adopted and contextualized to the local environment the intervention is operating in.

777 IN ORDER TO FILL THE PULSE The Test of Reliability – Meaning the indicators used must be accurate and consistent. If multiple uses of the same instrument (e.g. interview, survey) yield the same/or similar result then your indicator is reliable; The Test of Validity – Meaning the information that indicators provide must be close to the reality they are measuring Ways of ensuring an indicator is valid are: 1) common sense; 2) whether the indicator reflects similar findings in different situations; and 3) whether different survey instruments yield or uncover the same indicators. In general, the validity of an indicator can be enhanced by triangulation, or use of multiple sources of information and data. It is in this context that quantitative and qualitative approaches can be fruitfully mixed. Indicators Chosen Must Meet Two Tests:

888 INDICATORS AND YOUR INTERVENTION OBJECTIVES Since indicators are tools to elicit results (and for assessing impact) they are tied to the objectives with which the HIV/AIDS intervention begins: Objectives should be determined in relation to the baseline studies, against which results can be measured. Good objectives serve as “anchors” through which the project cycle in two important ways: – Provide a statement of the principal contributions and intervention will make in impacting a particular condition – Act as a set of “information handle” to assess progress during implement. Good objectives must be realistic, operational and measurable. They must also be tied to a credible implementation plan that links courses of action and intermediate targets to the expected final OUTCOME/IMPACT

999 INDICATORS, TIME-FRAMES AND SUSTAINABILITY So you ask which indicators to use at which stage of the project cycle? Input indicators should be used at or close to the start of the project, at which point base-line data is collected. Process indicators should be used while the project is proceeding, for purposes of monitoring, and until near the end of donor involvement. Output indicators should be used near to the end of donor involvement, and outcome indicators should be used after donor involvement is complete. Process, output and outcome indicators can then be compared against each other and against base-line data in order to determine how far objectives have been met. In some cases, the same indicator will be used to measure process, output and outcome (e.g. disaggregated enrolment figures, literacy rates or the local community's level of satisfaction with the project), the difference coming in that indicators are used at different points in time.