Monitoring and Evaluation: Frameworks For Malaria Programs

Learning Objectives At the end of the session, participants will be able to: 1.Identify and differentiate between conceptual frameworks, results and logical frameworks, and logic models 2.Define goals and objectives for specific intervention programs 3.Design frameworks for specific intervention programs 4.Discuss how frameworks are used for M&E planning

Module Outline Introduction to frameworks o Conceptual Frameworks o Goals and objectives o Results Frameworks o Logical Frameworks o Logic models Exercises developing frameworks Interface between frameworks and M&E planning Work on country group project

Types of Frameworks Conceptual Frameworks Results Frameworks Logical Frameworks Logic Models Why is it important to know about each of these frameworks?

Why are frameworks useful? Designing frameworks assists to develop: Clearly understood program/project goals and measurable, long-term, short-term, and intermediate objectives Clearly defined relationships between program/project: inputs, processes, outputs, and outcomes, and between program/project activities and the external context (environmental factors) Sound implementation and M&E plans

Conceptual Frameworks

Conceptual Frameworks (aka. research or theoretical frameworks) What is a conceptual framework? Diagram that identifies and illustrates the relationships between all relevant systemic, organizational, individual, or other salient factors that may influence program/project operation and the successful achievement of program or project goals M&E Purpose: To show where program fits into wider context To clarify assumptions about causal relationships To show how program components will operate to influence outcomes To guide identification of indicators To guide impact analysis (causal pathways) Similar frameworks: Proximate Determinants

Conceptual Frameworks Individual characteristics Technical inputs Program supply Institutional capacity Program sustainability Health status Healthy practices Service utilization

Conceptual Framework: Malaria Burden Malaria mortality Malaria infection Malaria morbidity Treatment: Early diagnosis & treatment Health care system:  Accessibility  Affordability  Quality of care  Efficiency  Demand/utilization Prevention: LLINs, IRS, IPT Environmental management Malaria knowledge: Cause Prevention methods Early treatment Cultural beliefs Information Program factors: Health policy Antimalarial drug policy Support/partnership National MCP External factors: Environmental (ecological, climate) Socio-economic (economic status, movement, occupation, housing condition, war, population displacement, etc) Demographic ( age, immunity, gender)

Conceptual Framework: Malaria Control and Elimination

Activity Get into country groups for 30 minutes Review your Problem Statement Develop a conceptual framework for your malaria control program Identify factors that influence malaria control and elimination Arrange factors into a conceptual framework

Outputs from Activity Reviewed Problem Statement A list of factors influencing malaria control and elimination A conceptual framework for your malaria control/elimination program

Goals and Objectives

What is the difference between goals and objectives? Goal: a broad statement of a desired, long- term outcome of the program Objectives: statements of desired, specific, realistic and measurable program results

Goals 1.To reduce malaria incidence and to significantly reduce deaths due to malaria by the end of Through the attainment of a 75% reduction, malaria control will ultimately contribute to the reduction of all-cause mortality by 20% in children under five by 2020

SMART Objectives  SMART S pecific: identifies concrete events or actions that will take place M easurable: quantifies the amount of resources, activity, or change to be expended and achieved A ppropriate: logically relates to the overall problem statement and desired effects of the program R ealistic: Provides a realistic dimension that can be achieved with the available resources and plans for implementation T ime-based: specifies a time within which the objective will be achieved

Class Activity: Are these objectives SMART? 1.To ensure that at least 80% of people sleep under LLINs in every district 2.At least 80% of pregnant women have access to the package of interventions to reduce the burden of malaria in pregnancy by December At least 85% of people sleep in sprayed structures in eligible areas of the 36 selected districts by December 2008, an upward revision from the 15 initially planned districts in the National Malaria Strategic Plan 4.100% of malaria patients in all districts are receiving prompt and effective treatment according to the current drug policy within 24 hours of onset of symptoms by next month

Small group work Get into your groups for 30 minutes and begin to develop goals and objectives for your program (First review your problem statement and Conceptual Framework)

Results Frameworks

What is a results framework? Diagram that identifies steps or levels of results and illustrate the causal relationships linking all levels of a program’s objectives. Purposes: Provides a clarified focus on the causal relationships that connect incremental achievement of results to the comprehensive program impact Clarifies project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends Other terms used: Strategic frameworks

Results Frameworks Goal: Improved Health Status Strategic Objective: Improved Use of Health Services or Practices IR1: Access/Availability IR1.1:Commodities/ Facilities IR1.2: Equity IR2: Quality IR2.2: Provider Performance IR2.3: Training/Supervision IR2.4: Information System IR3: Sustainability IR3.1: Policy IR3.2: Health Care Finance IR3.3: Private Sector IR4: Demand IR4.1: Attitude IR4.2: Knowledge IR4.3: Community Support

SO1: Reduced Malaria Burden SO1: Reduced Malaria Burden IR2: Improved malaria epidemic prevention & management IR2: Improved malaria epidemic prevention & management IR3: Increased access to early diagnosis & prompt treatment of malaria IR3: Increased access to early diagnosis & prompt treatment of malaria IR1.1 Access to & coverage by ITNs increased IR1.1 Access to & coverage by ITNs increased IR1.2 Improved access to IPT IR1.2 Improved access to IPT IR1.3 IRS coverage increased in epidemic prone areas IR1.3 IRS coverage increased in epidemic prone areas IR1.4 Use of source reduction/ larviciding increased IR1.4 Use of source reduction/ larviciding increased IR3.4 Access to services improved IR3.4 Access to services improved IR1: Improved malaria prevention IR1: Improved malaria prevention IR2.1 Early detection & appropriate response improved IR2.1 Early detection & appropriate response improved IR2.2 Epidemic preparedness improved IR2.2 Epidemic preparedness improved IR2.3 Surveillance system improved IR2.3 Surveillance system improved IR2.4 Early warning system strengthened IR2.4 Early warning system strengthened IR3.1 Quality of care improved IR3.1 Quality of care improved IR3.2 Efficiency in service delivery improved IR3.2 Efficiency in service delivery improved IR3.3 Utilization of care improved IR3.3 Utilization of care improved Resu lts Framework: Malaria Control Program

Results Frameworks Assist you in Identifying/selecting appropriate interventions to achieve strategic objectives Defining results/intermediate/operational/ outcome objectives Revisit the intervention you have selected, update them and discuss with your Supervisors

Logic Models

What is a Logic Model? Diagram that identifies and illustrates the linear relationships flowing from program inputs, processes, outputs, and outcomes. Inputs or resources affect Processes or activities which produce immediate results or Outputs, ultimately leading to longer term or broader results, or Outcomes. Purposes: Provides a streamlined interpretation of planned use of resources and desired ends Clarifies project/program assumptions about linear relationships between key factors relevant to desired ends Other terms used: M&E Frameworks, Logical Frameworks

Components of Logic Model The logic model has 5 essential components Logic Model Component Description InputsThe resources we invest in an intervention ProcessesThe activities carried out in order to achieve the intervention’s objectives OutputsThe immediate RESULTS achieved by the intervention at the intervention level through the execution of activities OutcomesThe set of results at the (target) population level induced by the interventions. People level effect ImpactsThe long term effects or end results of the intervention eg. changes in health status

Logic Models: ITN Distribution INPUT Human and financial resources ITNs BCC materials OUTPUT ITNs distributed BCC conducted OUTCOME ITN use IMPACT Malaria prevalence malaria PROCESS Establish distribution points for ITNs BCC staff trained

Logic Model: Malaria Control Programs InputsProcessOutputsOutcomesImpact Strategies Policies Guidelines Funding Materials Facilities Commodities Supplies Staff Training Services Education Treatments Interventions Services delivered Knowledge, skills, practice ITNs distributed HH sprayed IPTs delivered Antimalarials delivered RDTs/slides delivered and reads taken IITN/IRS ownership ITN use Treatment based on parasitological confirmation Malaria Incidence/ Prevalence Mortality Socio- economic wellbeing Economic Impact Adapted from: RBM Global Malaria Action Plan. Available at:

Logical Frameworks

What is a logical framework? A table which presents a standardized summary of the project and its logic. Purposes: Summarizes what the project intends to do and how Summarizes key assumptions Summarizes outputs and outcomes that will be monitored and evaluated Other terms used: Logframe matrix

Logical Frameworks Project DescriptionPerformance IndicatorsMeans of Verification Assumptions Goal: The broader development impact to which the project contributes - at a national and sectoral level. Measures of the extent to which a sustainable contribution to the goal has been made. Used during evaluation. Sources of information and methods used to collect and report it. Purpose: The development outcome expected at the end of the project. All components will contribute to this Conditions at the end of the project indicating that the Purpose has been achieved and that benefits are sustainable. Used for project completion and evaluation. Sources of information and methods used to collect and report it. Assumptions concerning the purpose/goal linkage. Component Objectives: The expected outcome of producing each component's outputs. Measures of the extent to which component objectives have been achieved and lead to sustainable benefits. Used during review and evaluation. Sources of information and methods used to collect and report it. Assumptions concerning the component objective/purpose linkage. Outputs: The direct measurable results (goods and services) of the project which are largely under project management's control Measures of the quantity and quality of outputs and the timing of their delivery. Used during monitoring and review. Sources of information and methods used to collect and report it. Assumptions concerning the output/component objective linkage. Activities: The tasks carried out to implement the project and deliver the identified outputs. Implementation/work program targets. Used during monitoring. Sources of information and methods used to collect and report it. Assumptions concerning the activity/output linkage.

Logical Frameworks: Malaria Control Program Project Description Performance indicators Means of verification Assumptions Goal: To reduce malaria morbidity and mortality by 50% by 2015 Malaria prevalence All-cause under –five mortality rate Annual reports Surveys DSS (INDEPTH) DHS Political stability Environmental stability (no natural disasters) Overall Objective: To achieve universal coverage of malaria control interventions % of individuals with access to an LLIN in their HH Annual reports Surveys Record reviews Availability of effective and affordable drugs Component Objective: To achieve universal coverage and increase use of LLINs % of individuals with access to an LLIN in their HH % of individuals who slept under an LLIN the previous night Population- based survey Health facility surveys Community surveys Strong political support Availability of LLINs

Project DescriptionPerformance Indicators Means of Verification Assumptions Outputs: LLINs Distributed to target population LLIN hang up demonstrated to individuals in target population Number of LLINs distributed to target population Number of individuals who observed LLIN hang up demonstration Activity reports/program records Funds available for distribution and communication campaign Community support Activities: LLIN Distribution Campaigns LLIN hang up demonstrations Number of LLIN distribution campaigns Number of LLIN hang up demonstrations Activity reports/program records Funds available for distribution and communication campaign Logical Frameworks: Malaria Control Program

Discussion 1.Does your program use any of these frameworks for planning? How? 2.What are other ways that these frameworks can be used for planning?

Frameworks for M&E Planning Purposes: Clarifying assumptions, goals, and interrelationships between factors relevant to the project or program Defining objectives Selecting activities Defining levels of performance and desired results in terms of planned activities and realistic, objective impacts Monitoring and evaluation plans incorporate: Program managers’ assumptions and objectives, in a given context A schematic design displaying the directional linkages between key program elements and/or planned results, and other relevant factors

Summary of Frameworks Type of Framework Brief DescriptionProgram ManagementBasis for Monitoring and Evaluation ConceptualInteraction of various factors Determine which factors the program will influence No. Can help to explain results ResultsLogically linked program objectives Shows the causal relationship between program objectives Yes – at the objective level LogicalLogically linked program objectives, outputs, and activities Shows the causal relationship between activities and objectives Yes – at the output and objective level Logic modelLogically links inputs, processes, outputs, and outcomes, Shows the causal relationship between inputs and the objectives Yes – at all stages of the program from inputs to process to outputs to outcomes/ objectives

References AusGuide. The Logical Framework Approach. Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project. Global AIDS Program. (2003) Monitoring and Evaluation Capacity Building for Program Improvement Field Guide, Version 1. U.S. Centers for Disease Control and Prevention, Atlanta, GA. Ministry of Health, Zambia National Malaria Control Action Plan: Actions for Scale-up for Impact on Malaria in Zambia. Marsh, David Results Frameworks & Performance Monitoring. A Refresher by David Marsh (ppt) Measurement, learning and evaluation project for the urban reproductive health initiative. Measuring success toolkit. Tsui, Amy Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. Tsui, Amy Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. UNICEF State of the World’s Children.

Group Projects 1.Get into your groups 2.For your malaria control program final project, complete the following for your final presentation: –Problem Statement –Conceptual framework –Goals and objectives with targets –The Logic model, results framework, or Logical Framework

MEASURE Evaluation is a MEASURE program project funded by the U.S. Agency for International Development (USAID) Through Cooperative Agreement GHA-A and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, John Snow, INC., ICF International, Management Sciences for Health, and Tulane University. VISIT US ONLINE AT