1. SOCIAL SCIENCES &TECHNOLOGY
INTRODUCTION to M&E
Lecturer
D. S. BOWASI
ATD; BED. MAED
EXECUTIVE DEAN
SCHOOL OF EDUCATION,
SOCIAL SCIENCES &TECHNOLOGY

2. 19 February 2015 University of LUSAKA Bpa 450 D.S. BOWASI (DEAN – SEST
Introduction to M&E
19 February 2015
University of LUSAKA
Bpa 450
D.S. BOWASI (DEAN – SEST

3. Session Objectives Define Monitoring Define Evaluation Purposes of M&E
Describe the relationship between monitoring and evaluation
Define Program Components
Identify the different types of M&E

4. Learning Objectives By the end of this session you will be able to:
Define key M&E terms
Identify components of programs to evaluate
Describe different purposes for M&E
State why Monitoring & Evaluation are important in programming
Describe characteristics of good M&E system

5. Defining Programs, Projects & Interventions
Program: organized effort to respond to a broad social problem (typically organized at national level)
Project: specific set of activities with linked objectives, that contribute to the overall objective of a program (typically organized at sub-national level)
Intervention: often used in same was as “project”, sometimes a specific sub-set of program or project activities

6. Program Planning Involves Setting…
Goals & Objectives based on intended Impact
Intended Outcomes
Intended Outputs or Deliverables
Planned Activities (Processes)
Inputs or Resources

7. Inputs What you need to implement the program
e.g. the financial , human and material resources used to implement an HIV counseling and testing program
Trained personnel
Protocols and guidelines
Training materials
HIV test kits and other supplies
Money

8. Activities What the program provides to accomplish its objectives
Training workshops on for personnel and site managers
Providing pre-test and post-test counseling to clients
Curriculum development
Recording & reporting
Dissemination of IEC materials
Supervision

9. Outputs What the program delivers Condoms distributed
Clients receiving pre-test counseling, HIV tests, post-test counseling
Materials distributed
People reached
Intervention sessions completed

10. Outcomes
The results of the program or changes that occur both immediately or some time after activities are completed
Changes in knowledge, behavior, attitudes and skills.
Quality of personnel and managers improved.
. Clients develop and adhere to personalized risk-reduction and treatment strategy

11. Impact The wider effect of the program on long-term results
HIV transmission rates decrease
HIV incidence decreases
Changes in HIV mortality and morbidity
Decrease in deaths due to HIV-related TB

12. Example: Components of an HIV program
Programme level
Population level
Input
Process
Output
Outcome
Impact
Financial resources
Staff
Supplies
Capacity building
BCC interventions
Community mobilization
Policy development
Improved skills of providers
Increased knowledge
Mobilized communities
Sound policies available
Reduced risk behaviors
Declining HIV/AIDS morbidity and mortality
Economic growth

13. So how does this relate to Back to our definitions…
M&E?
Back to our definitions…

14. Set of procedures & analytical tools to examine
What Do We Mean by M&E?
Set of procedures & analytical tools to examine
how programs are conducted (inputs & activities)
their level of performance (outputs)
whether they achieved what they were intended to achieve (outcomes & impact)
Types of evaluation
monitoring (including process evaluation)
evaluation (outcome and impact evaluation)

15. Evaluation: What have we achieved?
Monitoring: What are we doing? Tracking inputs and outputs to assess whether programs are performing according to plans (e.g., people trained, condoms distributed)
Evaluation: What have we achieved?
Assessment of impact of the programme on behaviour or health outcome
(e.g., condom use at last risky sex, HIV prevalence)

16. What is Monitoring?
Is a continuous systematic process of collecting, analyzing and using information to track the efficiency of achieving program goals and objectives
Provides regular feedback that measures change over time in any of the program components such as costs, personnel and program implementation
An unexpected change in monitoring data may trigger the need for a more formal evaluation of activities

17. Key M&E Questions Is the program being implemented as planned?
Are things moving in the right direction?
Did the program achieve its objectives?
Can results be attributed to program efforts?
Which program activities were more (or less) important/effective?
Did the target population benefit from the program?
At what cost?

18. What is Evaluation?
Is a systematic process limited in time of collecting, analyzing and using information to assess the effectiveness, relevance and impact of achieving your program’s goals.
Requires study design; sometimes a control or comparison group; often measurement over time.
Often involves measuring changes in knowledge, attitudes, behaviors, skills, community norms, utilization of health services, & health status at population level
Provides regular feedback that helps programs analyze the consequences, outcomes and results of its actions

19. What is Evaluation? (con’t)
Rigorous research design need for:
Establishing a causal link between program effort and desired outcomes
Isolating program effect from other, non- program influences on the outcome of interest
Not undertaken routinely; usually reserved for specific situations, such as determining the success of a project for scale-up or replication

20. Outcome Evaluation Time With program Attribution “Impact”
Program Indicator:
Without program
Program Start
Program End
Time

21. Impact Evaluation Time With program Attribution “Impact”
Program Indicator:
Fertility
or HIV prevalence
Without program
Program Start
Program End
Time

22. Illustration of Program Evaluation/ Impact
With program
Change in
program
outcome
Without program
Program
impact
Program
start
Program
end
TIME

23. Comparison of Potentially Confusing Terms
Outcome
Impact
Monitoring
Simple Tracking of:
Results that occur both immediately and some time after the activities are completed, such as changes in knowledge, attitudes, beliefs, skills, behaviors, access, policies, and environmental conditions.
Clients reduced risk to HIV infection
Increased prevalence of immediate breastfeeding in hospitals
Decrease in newborn hypothermia
Increased use of modern methods of contraception
Simple Tracking (over multi-year periods) of:
Long-term results of one or more programs over time, such as changes in morbidity, and mortality
HIV prevalence
TB incidence
Maternal morbidity
Infant mortality
Evaluation
Rigorous Scientific Analysis (including attribution of effects): of:
Program outcomes (same definition as above)
Note: Most people consider this to be “Impact Evaluation”
Program impacts (same definition as above)

24. Key M&E Questions Did the program achieve its objectives?
Did the target population benefit from the program?
At what cost?
Can improved health outcomes be attributed to program efforts?
Which program activities were more (or less) important/effective?
What would have happened in the absence of the program?
How can we know or measure this (the counterfactual)?

25. Why Monitor & Evaluate?
To make decisions about project management and service delivery
To ensure effective and efficient use of resources and provide accountability to donors
To assess whether the project has achieved its objectives - has the desired effects
To learn from our activities, and provide information to design future projects
Programme
Improvement
Share
with
Partners
Reporting/
Accountability

26. Purposes of Monitoring and Evaluation
Determine whether a plan or program is on schedule with planned activities
Assess whether a policy, plan or program has produced desired impacts
Generate knowledge:
Identify factors (individual, community, programmatic) that influence health outcomes
Help inform policy, planning or program decisions: new services, resource allocation, corrections, etc.

27. What Decisions are Guided by M&E Results
How much funding should be allocated to a program
What types of programs should be funded
Which programs approaches should be presented as models
New strategic objectives, activities or results packages
Replication and scaling up of successful

28. Reasons to Monitor & Evaluate Different Needs for Different Stakeholders: 1. Funding Agencies & Policy Makers
What M&E Measures
Evidence of achievement of program objective
Program outcome and impact
Program cost-efficiency
Data about the target population
What M&E Results Identify
Priorities for strategic planning
Programs that qualify for donor assistance
Best practices
Impact of donor assistance

29. Reasons to Monitor & Evaluate- Different Needs for Different Stakeholders: 2. Communities & Youth
What M&E Measures
Youth behavior related to reproductive health
Young peoples’ needs
How program funs are being spent
The process and impact of community participation
What M&E Results Identify
Actual and potential benefits of youth programs
Need for new and better youth services
Community resources that can be used to support ARH programs
Need for local support for ARH issues and actions

30. 2. Communities & Youth Cont…
What Decisions are Guided by M&E Results
Degree to which community members and youth should participate in and support the program
How to better coordinate community actions to address identified programs
How many and what type of local resources should be allocated to identified programs

31. 3. Programme Managers & Staff
What M&E Measures
Quality of activities and/or services
Why some sites are more successful
Program coverage
What M&E Results Identify
Priorities for strategic planning
Training and supervision needs
How to improve reporting to funding agency
Feedback from clients
Why program is not accomplishing what it set out to do

32. Replication and scaling up of intervention Fund-raising
What Decisions are Guided by M&E Results (Programme Managers and Staff).
Resource allocation
Replication and scaling up of intervention
Fund-raising
Motivating staff
Policy advocacy

33. What decisions are Guided by M&E Results
Agreement on the scope and objectives of M&E plan with stakeholders
Selection of Indicators
Systematic and consistent collection of information on the selected indicators
Analyze the information gathered
Compare results with program initial goals and objectives
Share results with stakeholders

34. Defining the Scope of M&E Effort
Scope refers to the extent of the activity you will undertake in a M&E effort. Scope is determined by several factors (questions)
What should be monitored and evaluated?
When should health programs be monitored and evaluated?
How much will M&E cost?
Who should be involved in M&E?
Who should carry out the evaluation?
Where should M&E take place?

35. What Should be Monitored & Evaluated?
M&E can measure each stage of your program development: design, system development and functioning, and implementation.
After goals, objectives and activities are developed, decision on about M&E at each stage is needed.
M&E effort can measure each stage to determine how the program is working and its impact on the target population.

36. What Should be Monitored & Evaluated?
Program Design is measured by process evaluation:
Assessing how well the program has been designed
Documenting the success/challenges with program design
System development and functioning is measured by monitoring and process evaluation:
Document the development of support systems and determine if they are actually operating once program implementation begins
Assess the performance of support systems
Measure how effective the preparatory activities are in readying program personal for program implementation
Implementation is measured by monitoring, process evaluation, and outcome/impact evaluation:
Reveal how program implementation is occurring
Determine whether program is achieving its objectives by measuring the changes in outcomes in your target population

37. When Should Programs be Monitored & Evaluated?
Monitoring and process evaluation should occur throughout the life of a program
Outcome and impact evaluations are usually done near the end of a program (baseline gathered to measure change)
Starting M&E at the beginning of a program is ideal
Some activities can still be measured if M&E is started in the middle of a program
Even Fewer activities can be measured if M&E is started towards the end of the program

38. When Should a Program be Monitored & Evaluated?
Stage of Monitoring Process Outcome/Impact
Program
Early
- Set up monitoring system; identify indicators and tools, plan for tracking program, data analysis, and reporting.
- Assess systems development and functioning. Provide early feedback. Assess if program is responsive to target.
- Identify objectives and indicators. Take baseline measurements. Crate an outcome or impact evaluation plan.
Middle
- Assess MIS and data. Modify original system is inadequate. If program is not performing, launch process evaluation.
- Conduct more formal midterm process evaluation to assess quality of program performance. Determine coverage, or whether program is reaching target.
- Take mid-term measurements. Analyze short term outcome measures, such as changes in knowledge, increase in use of programs and changes in contextual factors. Provide feedback to program.
Late
- Analyze data from tracking system to conclude if you conducted the program as planned. Prepare and submit reports.
- Analyze end of program measurements. Determine what was done to improve quality of program implementation. Make recommendations for program replication or expansion.
- Take end of program measurements. Examine evidence of changes in outcomes. Depending on study design conduct impact analysis to conclude whether outcomes are attributable to program activities. Report to donor and other stakeholders.

39. Where are the levers in a working health systems
6 building blocks of health systems strengthening
Service delivery
Health workforce
INFORMATION
Medical products ,Vaccines and technologies
Financing
Leadership and governance
WHO: Nellie Bristol www. globalhealthmagazine.com

40. Reference M&E Websites
MEASURE Evaluation:
Health Metrics Network:
John Snow Inc.:
HIV Global Partners:

41. END OF PRESENTATION