1. MONITORING AND EVALUATION IN TB/HIV PROGRAMS
Presentation for the ProTEST Lessons Learnt workshop, Durban, RSA
By Wilfred Nkhoma
Medical Officer, WHO/AFRO/TB UNIT

2. FOCI OF MONITORING AND EVALUATION IN TB/HIV PROGRAMS
Assessment of the design of interventions
Monitoring of program implementation
Assessment of program utility (effectiveness and efficiency):

3. Monitoring program conceptualisation
Has a problem been appropriately conceptualized ?
is the program designed to meet intended objectives ?)

4. MONITORING PROGRAMME IMPLEMENTATION:
To determine the extent to which a program is reaching the appropriate target population [Program coverage]
To determine whether or not delivery of program elements (services and treatments) is consistent with program design specifications[Program delivery]
To determine what resources are being expended in the conduct of the program [Program costs]

5. Monitoring of Program Implementation questions
There is no point in being concerned with the impact or outcome of a program unless it has indeed taken place and served the appropriate participants in the way intended (as designed). [One not uncommon reason impact evaluations reveal little or no impact is not that a program is ineffective, but rather that its implementation is faulty or incomplete]
Is the program reaching the specified target population (program coverage)?
Are the intervention efforts being conducted as specified in the program design (program delivery)

6. Monitoring programme coverage
Coverage refers to the extend to which the participation by the target population reaches the levels specified in the program design [conclusions include undercoverage, bias and overcoverage].Bias is the degree to which sub-groups of the target population participate differentially
Estimating of efficiency of programme coverage:
Coverage efficiency = 100 *{number Number not }
{in need served - in need served }
{Total number Total number } {in need served }

7. Monitoring delivery of services
Measures the extent to which program specifications
are met in the delivery of the interventions. Common
causes of delivery failures (Rossi, 1978):
No treatment or not enough is delivered: “Nonprograms”. Little evidence that a program existed !
Wrong treatments delivered
Treatment is unstandardised, uncontrolled or
varies across target populations [e.g too much
discretion in the hands of service providers!]
resulting in non-comparability between
sites or communities

8. ASSESSMENT OF PROGRAM UTILITY
Assessing the utility of intervention programmes involves systematic examination of :
i) Programme impacts [extent to which programme produces desired outcomes]
ii) Programme efficiency [the benefits of a programme in relation to its costs]
Thus utility assessments help to distinguish useful from ineffective and inefficient intervention programs/ interventions

9. IMPACT ASSESSMENTS
Undertaken to establish whether or not interventions produce their intended effects (the extent to which a program causes change in the desired direction)
Basic aim is to produce an estimate of the “ net effects” of an intervention free of the influence of other processes and events that may produce the same results [establishing causality].
Comparison of outcomes in targets who have experienced an intervention and those who have not is the underlying common frame of reference for all impact evaluations
Basic requirements:
Well articulated objectives
Specification of measures of goal achievement
Sufficiently well implemented interventions that there is no question that its critical elements have been delivered to appropriate targets.
Data collection plan that allows measures of goal achievement to be derived [commonly quantitative or quantifiable qualitative data]

10. EFFICIENCY ASSESSMENTS
Provide a frame of reference for relating program costs to program results (i.e guages the extent to which a program is producing sufficient benefits to program participants for the costs incurred)
Efficiency assessments employ cost-benefit and cost-effectiveness analysis methodologies
These can be undertaken before or after a program has been in existence but often only possible after program has been in existence

11. Cost-benefit analysis
Program inputs and program outcomes measured in monetary terms
Requires specification, measurement and valuation of all program costs and benefits (costs include both direct and indirect costs of undertaking the program).

12. Cost-effectiveness analysis
Requires program costs in monetary terms and benefits in outcome units
Generally less controversial than cost-benefit analysis as regards the need to express outcomes in monetary terms
Final measure is expressed as the cost of achieving a magnitude of substantive intervention outcome
Amenable to rank ordering of programs in terms of their costs for reaching given goals or the various inputs required for different degrees of goal achievement

13. PROGRAM UTILITY QUESTIONS
Is the programme effective in achieving its intended goals ?
Can the results of the program be explained by some alternative process that does not include the programme ?
What are the costs to deliver services and benefits to program participants ?
Is the programme an efficient use of resources, compared with alternative uses of the resources ?

14. CONCLUSIONS
Monitoring and evaluation are essential elements of the process of implementation of TB/HIV programs
(as for other public health and social programs)
Program implementation and utility with particular focus on coverage, delivery, impact and cost-effectiveness of interventions constitute the cardinal areas in monitoring and evaluating TB/HIV activities (and should be planned for at the design stage of intervention programs).
Meaningful interpretation of monitoring and evaluation efforts entails specification of services: defining each kind of service in terms of the activities that should take place or in terms of participation by target populations and providers [time, costs, procedures, or a product, i.e. Specific simple program elements that can be identified, and reliably meaningfully counted !]