Monitoring and Evaluation: Frameworks Speaker Notes Recap prior work – here is some suggested wording, but it needs to be consistent with what is actually presented Yesterday we were introduced to the role of Strategic Information and Decisions Making and the components of an M&E plan. Today we are going to discuss the first part of that plan: the frameworks that are used to describe the factors that influence a public-health problem, used to design intervention studies and programs, and used to to design M&E plans. The speaker should introduce him/herself and if new to the group, have the participants introduce themselves. The speaker’s introduction could include a brief description of his/hers experience with using frameworks. Additional Speaker Notes The handout for this session should be distributed prior to beginning. The handout contains examples of frameworks to which participants can refer later. Some frameworks also appear in the slides, some do so in abbreviated form, and others do not appear in the slides.

Learning Objectives At the end of the session, participants will be able to: Identify and differentiate between conceptual frameworks, results and logical frameworks, and logic models Design goals and objectives for specific intervention programs Design frameworks for specific intervention programs Discuss how frameworks are used for M&E planning Speaker Notes Four types of frameworks are introduced in this module: conceptual frameworks, strategic or results frameworks, logical frameworks and logic models. The learning objectives of this Module on Frameworks include understanding the components of these four basic types of frameworks in order to identify and differentiate between them. Participants will be expected to design frameworks for a specific intervention program and to use these frameworks for developing M&E plans. Another learning objective is to design goals and objectives for specific intervention programs. Goals and objectives are the core of any M&E systems. If the program goals and objectives are written in such a way that they can be easily distinguished from each other and measured, the job of M&E will be much easier. Unfortunately, many times, this is not the case. In this section, we will learn how to write program goals and objectives so that they are easily monitored and evaluated. Additional Background Different organizations tend to prefer a selected type of framework. It is not important to convince participants to use any particular type. The gist of this session, however, is the importance of designing a useful framework for a specific project in a specific context, not the titles or particular appearance of any designated kind of framework.

Module Outline Introduction to frameworks Conceptual Frameworks Goals and objectives Results and Logical Frameworks Logic models Exercises developing frameworks Interface between frameworks and M&E planning Work on group project Speaker Notes We will start by reviewing the different types of frameworks and by learning how to design goals and objectives that will be measurable. Frameworks are probably not new to you; However, people tend to use different terms for different types of frameworks. So we will mention these other terms as we go along. All of you should have received a handout containing examples of frameworks. You may find these handouts helpful for understanding the content and purpose of the different types of frameworks. You may want to keep these handouts for future reference. Please keep in mind that these frameworks are meant to be examples and are not intended to be interpreted as the “correct” framework for a certain program. There will be an opportunity to develop examples of the different types of frameworks. Lastly, the interface between frameworks and M&E planning will explored. The last two hours of the day will be devoted to work on your group projects that will last the duration of the workshop.

Why are Frameworks Useful? Designing frameworks assist developing: Clearly understood program/project goals and measurable, long-term, short-term, and intermediate objectives Clearly defined relationships between program/project inputs, processes, outputs, and outcomes, and between program/project activities and the external context (environmental factors) Sound implementation and M&E plans Speaker Notes Frameworks are best understood as useful tools for understanding and analyzing a program. Designing frameworks is one way to develop a clearer understanding of the goals and objectives at the heart of a project, with emphasis on measurable objectives. Developing frameworks also helps to clearly define the relationships among factors key to the implementation and success of a project. These factors may be internal or external to the program context. The framework design process is crucial for developing sound implementation and M&E plans and serves as the foundation for selecting appropriate and useful M&E indicators. The process of designing frameworks also helps to clarify many of the concepts and assumptions underlying basic project activities. Additional Background While frameworks are obviously useful after their completion, providing a clear structure of the indicators at the heart of M&E plans, one of most useful qualities is that discussing and determining their design serves to clarify many of the concepts and assumptions underlying basic project activities. An M&E plan that does not identify appropriate, useful M&E indicators, metrics, and data systems is an M&E plan that will not contribute to good program management or to an intervention’s success.

Types of Frameworks Conceptual frameworks Results frameworks Logical frameworks Logic models Speaker Notes We will review four types of frameworks: Conceptual Frameworks, Results frameworks, Logical Frameworks and Logic Models. We will discuss each type of framework in terms of its characteristics and uses, and you will get to practice developing these frameworks. Keep in mind that it is not necessary for an M&E plan to design and use all four types of frameworks. However, it is important to know the differences between the types and how they are used.

Conceptual Frameworks (a.k.a., research or theoretical frameworks) Diagram that identifies and illustrates the relationships between all relevant systemic, organizational, individual, or other salient factors that may influence program/project operation and the successful achievement of program or project goals. M&E Purpose: To show where program fits into wider context To clarify assumptions about causal relationships To show how program components will operate to influence outcomes To guide identification of indicators To guide impact analysis (causal pathways) Similar frameworks: Proximate Determinants Speaker Notes Conceptual frameworks are sometimes called “research” or “theoretical” frameworks. A conceptual framework is a useful tool for identifying and illustrating a wide variety of factors and relationships that may affect program success. Conceptual frameworks take a broad view of the program itself in order to clarify the relationship of its activities and its main goals to the context in which it operates. The design of the conceptual framework should show the interrelationships between all factors that are relevant to achieving the program’s goals. These factors can be systems, organizations, government or institutional policies, infrastructure, population characteristics, or other features of the operational landscape that may help or hinder the program’s success. Constructing a conceptual framework thus clarifies the complete context affecting the outcome of an program or intervention. Constructing a conceptual framework also clarifies your assumptions about the causal relationships between significant features of the program context, clarifying aspects that your planned intervention may affect as well as other factors beyond your control. Identifying the variables that factor into program performance, and organizing the explicit ways they interact with each other sets the stage for outlining the objective results you can reasonably expect from your program activities. Clarifying all of these issues is a critical step toward designing valid measures for analyzing, or evaluating, the success of those interventions. Another type of framework, proximate determinants, highlights the factors directly affecting the health outcome. Sometimes the proximate determinants are labeled as such within a conceptual framework. We will not be discussing proximate determinants in this module. Additional Background Conceptual frameworks are used in the sciences to select key variables for analysis. By constructing this kind of analytical framework as the foundation within which your program will design, plan, and implement the Monitoring and Evaluation of program performance, real possibilities and limitations become clearer to everyone involved.

Conceptual Frameworks Individual characteristics Technical inputs Program supply Institutional capacity Program sustainability Health status Healthy practices Service utilization Speaker Notes This is a very general example of a conceptual framework in the health field. It has the characteristics described in the previous slide that it includes a number of factors that affect the desired outcome, including those not under the control of the program (such as individual factors). In each box is a factor related to another factor that ultimately affect a health outcome. Arrows indicate the relationship between factors, which in this case include individual, systemic and program variables. As you look at the framework, notice that there are two outcomes: health status and program sustainability. Following health status backward, we see that healthy practices influence health status. Healthy practices are affected by individual characteristics, service utilization and institutional capacity. You can also see service utilization is influenced by program supply and individual characteristics. Going back to institutional capacity, note that, according to this framework, it is affected by program supply and technical inputs. We just looked at this framework by moving backward from the outcomes of interest through all the influencing factors. You could also read it moving forward, however moving backwards allows you to keep the outcomes of interest in your mind when looking at relationships between different factors. Also, note that that one factor can influence or be influenced by more than one other factor. In this example, you can also see that program sustainability and institutional capacity influence each other. Additional Background During the rest of this workshop, many examples of conceptual frameworks may be provided or developed and discussed by participants. The point at this stage is to show that this kind of framework explicitly incorporates factors external to the program in order to examine the kind of impacts that contextual factors may have on outcomes of interest to the program.

Conceptual Frameworks National Tuberculosis (TB) Program External Factors Demographic: (age, gender) Socio-economic: (education, occupation) Environmental: (war, weather) TB infection Health Systems (DOTS) Availability Access Utilization Case Detection TB Morbidity Prevalence Incidence HIV co-infection MDR-TB TB mortality Speaker Notes This is an example of a more specific conceptual framework for national tuberculosis (TB) programs. Again, factors include those external to a program, such as demographic or socio-economic characteristics. It also includes factors related to the health system and to individuals. You can see that the program is aiming to affect the health system and knowledge of TB which will in turn affect morbidity, which will ultimately affect mortality due to TB. Other health related factors that can affect morbidity are included in this model. Additional Background In the TB framework above, the term DOTS in the Health Systems box refers to “directly observed therapy” which is the internationally recognized treatment involving direct observation by a clinician of short course chemotherapy. Program Factors Political commitment Donor involvement National TB program Co-morbidity HIV Malnutrition Alcoholism Diabetes TB Knowledge Stigma Treatable Symptoms Source: MEASURE Evaluation 2004 (draft)

Conceptual Frameworks Causes of malnutrition in society Insufficient access to food Potential Resources: environment, technology, people Quantity & quality of actual resources - human, economic and organizational – and the way they are controlled Inadequate maternal & child care practices Poor water/sanitation & inadequate health services Child malnutrition, death and disability Inadequate Disease dietary intake Outcomes Immediate causes Underlying causes at household/ family level Speaker Notes This slide shows an example of a nutrition framework. This framework is laid out bottom to top whereas the other two examples were left to right. The orientation is not important – what is important is the causal links between all the factors listed. Notice the labels on the right hand side of the slide. These give additional interpretation to the factors listed in the boxes and clarify their proximity to malnutrition and death. This framework was provided by UNICEF in 1990 and serves as a guide for assessing and analyzing the causes of child malnutrition. The framework highlights the multi-sectoral nature of the causes of malnutrition, which encompass insufficient access to food, inadequate care for mothers and children, and insufficient heath services. The causes of malnutrition are also categorized as immediate, underlying, and basic. Basic causes of malnutrition include inadequate education, the political and ideological superstructure, the economic structure, and the resource base. These factors influence the underlying causes – food, care, and health services – which affect directly, in turn, the two primary proximate determinants of nutritional status and survival, namely dietary intake and disease. Basic causes at societal level Source: UNICEF, State of the World’s Children, 1998

Activity Get into groups of 4 or 5 Develop conceptual framework for utilization of family planning/maternal and child health services Identify factors that influence health service utilization Arrange factors into a conceptual framework After 30 minutes, a member of each group will share the framework with all participants Speaker Notes Arrange yourselves in small groups of up to 5 people. (This does not need to be the group for the group project.) Each group will be asked to develop a conceptual framework for utilization of family planning/maternal and child health services. Use the materials provided to work out a framework. You may want to us a post-it note for each item in your framework, then move the post-it notes around on the large piece of paper until you finish your framework. Finally, copy the framework using the marking pens onto the large paper. One person from the group will be designated to share this framework with the larger group of participants. I will warn groups when there are 10 minutes remaining. Additional Background Monitor the groups to make sure they are progressing and to answer questions. The instructor should have also received a handout with commonly used conceptual frameworks for child health, family planning and quality of care for his/her information. This handout should not be distributed to participants at this time. Activity Recommended materials for this activity are: - Large piece of poster board or flip chart paper, one per group - Post-it notes, one pack per group - Thick marking pens, one or two per group

Goals and Objectives Goal: a broad statement of a desired, long-term outcome of the program Objectives: statements of desired, specific, realistic and measurable program results SMART Specific: identifies concrete events or actions that will take place Measurable: quantifies the amount of resources, activity, or change to be expended and achieved Appropriate: logically relates to the overall problem statement and desired effects of the program Realistic: Provides a realistic dimension that can be achieved with the available resources and plans for implementation Time-based: specifies a time within which the objective will be achieved Speaker Notes Fundamental to the design of any type of program and its accompanying framework, whether it is a results framework or a logical framework or a logic model is to clearly identify the program’s goals and objectives. A goal is a broad statement of a desired, long-term outcome of the program. Whereas the objectives are the statements of desired, specific, realistic, and measurable program results. The acronym “SMART” is useful to remember when developing objectives. (Review SMART as explained on the slide.) SPECFIC: Is the objective specific? Does it cover only one activity rather than multiple activities? MEASURABLE: Is it measurable? Can it be measured or counted in some way? APPROPRIATE: How important is this objective to the work that we are doing? How relevant is it to achieving our goal? REALISTIC: Is the objective actually doable? Can we achieve this goal? TIME-BASED: Does the objective give a timeframe by when the objective will be achieved, or a timeframe during which the activity will occur? ATTAINABLE: Sometimes the letter “A” in the SMART acronym stands for attainable. RELEVANT: Sometimes the letter “R” in the SMART acronym stands for relevant: Source: GAP 2003

Formulating Program Objectives To [action] The [specify what knowledge, attitudes, skills, behaviors] Among [specific population or segment] From – to [from baseline to desired level] or by [ x percent ] or to [specific level] By [time frame] A properly stated objective is action-oriented, starts with the word “to,” and is followed by an action verb. Objectives address questions of “what”, “who”, “how much” and “when,” but not “why” or “how.”

Goals and Objectives National AIDS Committee (NAC) 2005 Strategic Plan Goal: Prevent the spread of HIV epidemic and minimise its impact on nation by 2009 Objective 1: Reduction of HIV prevalence in nation Objective 2: Improved health & quality of life of people infected & affected by HIV/AIDS Objective 3: Strengthened capacity of NAC & stakeholders to respond to the HIV/AIDS epidemic at all levels through improved research, M&E and improved management & coordination Speaker Notes Here is an example of goals and objectives developed as part of a national AIDS program strategy. Notice that the goal is more general and the objectives are more specific, further explaining the overall goal of the program. Using the SMART acronym, we can assess whether the objectives meet the criteria presented in the previous slides: Specific – the objectives do seem to specify the intended outcomes of the program Measurable – HIV prevalence is measurable, health and quality of life can be measured although the metrics will have to be specified in order to be clear, capacity of NAC and stakeholders can also be measured. Appropriate – all objectives relate logically to the stated goal, and most likely the problem statement although that is not presented here. Realistic – Since we don’t see the activities and resources available for the program here, it is difficult to judge whether they are realistic. However, we can say that all three objectives are realistic to achieve within 5 years. Time-based – While a time is not specified for each objective, a time is specified for the goal. It can be assumed that the aim is to achieve these objectives within the 5 years of the program.

Goals and Objectives: Activity Insufficient access to food Potential Resources: environment, technology, people Quantity & quality of actual resources - human, economic and organizational – and the way they are controlled Inadequate maternal & child care practices Poor water/sanitation & inadequate health services Child malnutrition, death and disability Inadequate Disease dietary intake Outcomes Immediate causes Underlying causes at household/ family level Activity With all participants in one large group led by the session facilitator, develop a goal and at least two objectives for a program that addresses this conceptual framework for a nutrition program shown previously. Use a flip chart to write the group’s contributions. After the goal and objectives have been written out, review them against the SMART criteria: specific, measurable, appropriate, realistic, time-based. Refer to previous slides (number 11-12) if necessary. If the SMART criteria are not met, revise goal and objectives. Possible goal and objectives for nutrition framework: Goal: Reduce childhood mortality from malnutrition Objectives: At the end of five years, the proportion of children under 5 years of age classified as malnourished will be reduced. At the end of five years, the proportion of households with insufficient access to food will be reduced. Reduce by half the proportion of people who suffer from hunger (Millennium Development Goals) Reduce by two thirds the mortality rate among children under five (Millennium Development Goals) Basic causes at societal level Source: UNICEF, State of the World’s Children, 1998

Results Frameworks Purposes: Other terms used: Diagrams that identify steps or levels of results and illustrate the causal relationships linking all levels of a program’s objectives. Purposes: Provides a clarified focus on the causal relationships that connect incremental achievement of results to the comprehensive program impact Clarifies project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends Other terms used: Strategic frameworks Speaker Notes Results or strategic frameworks are a useful tool for identifying and illustrating the focal points of a project for monitoring and evaluation: the results. A results framework diagrams the relationships between the incremental results of the key activities all the way up to the overall objective or goal of the intervention. This is the type of framework used by USAID in what they call Performance Monitoring Plans. Developing a framework focused on program results clarifies the points at which results can be monitored and evaluated. In addition, results frameworks clearly depict the causal relationships that the project design assumes will connect. For instance, the clinical training of providers will connect with the provision of a higher quality of care at facilities where (more) trained providers work, and thus ultimately lead to an improvement in health status or health outcomes for the targeted population. The effectiveness of the project’s related activities can be measured at each step along the way.

Results Frameworks Speaker Notes This is a generic example of a typical results framework for a program to improve use of health services or practices. The four themes of access/availability, quality, sustainability and demand are usually addressed. As you will see in subsequent slides, program frameworks contain more information that what appears here in this abbreviated example. Notice that the goal and strategic objective appear at the top of the framework, and under those are intermediate results or IRs. Under each IR are subordinate intermediate results or sub-IRs that relate directly to the intermediate results. For example, under IR2, factors related to the information system, training and supervision of clinicians, and provider performance lead to quality health services. Source: David Marsh, 1999

SO1: Increase tuberculosis case detection rate to 70% Results Frameworks Tuberculosis (TB) Control Programs SO1: Increase tuberculosis case detection rate to 70% IR1: Increased availability of quality services IR2: Increased demand for quality services IR1.1: Services increased IR2.1: Customer knowledge of TB improved Speaker Notes This example is for a tuberculosis control program. Three of the four IRs shown in the previous slide of a generic results framework appear here, two under IR1 (availability of services and quality of services). Notice that in this more developed example, the IRs and sub-IRs are phrased in such a way to indicate the desired results. For example, IR1 is not simply “quality services” as the previous generic example indicated, but “availability of quality services increased”. Also notice that each IR and sub-IR is measurable. That is, indicators can be developed and data collected to calculate those indicators. We will discuss indicators in the next session, but keep this in mind when thinking about results frameworks. IR1.2: Practitioners’ skills and knowledge increased IR2.2: Social support for TB practices increased IR1.3: Improved program management Source: MEASURE Evaluation, 2004 (draft)

Results Frameworks Donor/USAID Reproductive Health Program SO1: Increased Utilization of Family Planning/Reproductive Health Services IR1 Strengthened sustainability of FP/RH Program IR2 Expansion of high quality FP/RH services in the public and private sectors IR1.1 Improved policy environment for the provision of FP/RH services in the public and private sectors IR2.1 Increased availability of postpartum and postabortion FP services Speaker Notes Here is an example from a donor program. You can see that although this strategic objective is similar to that of the previous slide, the wording is slightly different. IR2.2 Increased accurate knowledge of clients about modern methods and FP services IR1.2 Strengthened NGO advocacy for FP program IR2.3 Improved job performance of health providers, trainers, and administrators Source: USAID/Turkey Performance Monitoring Plan, 1998-2001

Results Frameworks FIVE-YEAR GOAL: Foundation Established for Reducing Famine Vulnerability, Hunger and Poverty SO 14: Human capacity and social resiliency increased IR 14.1: Use of high impact health, family planning, and nutrition services, products, and practices increased IR 14.2: HIV/AIDS prevalence reduced and mitigation of the impact of HIV/AIDS increased IR 14.3: Use of quality primary education services enhanced IR 14.2.1: Reduced risk behavior IR 14.3.1: Community participation in the management and delivery of primary education services strengthened IR 14.1.1: Community support for high impact health interventions increased Speaker Notes Here is an example of a results framework that is multisectoral. This example comes from a country-level strategy adopted by a donor. The objective of improving human capacity and social resiliency is dependent on three intermediate results, two involving the health sector and one involving the education sector. It is important to remember that programs can span more than one sector and that this should be reflected in the framework. IR 14.2.2: Mother-to-child transmission of HIV reduced IR 14.1.2: Availability of key health services and products improved IR 14.3.2: Planning, mgmt and monitoring and evaluation for delivery of primary education services strengthened IR 14.2.3: Access to care and treatment for people living with HIV and AIDS increased IR 14.1.3: Quality of key health services improved IR 14.2.4: Care and support for orphans and vulnerable children expanded IR 14.3.3: Quality of primary education improved IR 14.1.4: Health sector resources and systems improved IR 14.2.5: A more supportive environment for responding to HIV/AIDS IR 14.3.4: Equitable primary education services strengthened Source: USAID/Ethiopia Integrated Strategic Plan, February, 2004 (revised draft)

Logical Frameworks Purposes: Other terms used: Present a standardized summary of the project and its logic. Purposes: Summarizes what the project intends to do and how Summarizes key assumptions Summarizes outputs and outcomes that will be monitored and evaluated Other terms used: Logframe matrix Speaker Notes Logical frameworks are similar to results frameworks in that they are program management tools that manage by results. USAID introduced the use of logical frameworks to international development work but is now promoting the results framework. Other donor agencies including Australia AID, DFID, GTZ, and CIDA continue to use logical frameworks. Logical frameworks logically summarize what the project intends to do and how, the key assumptions, and outputs and outcomes that will be monitored and evaluated. A Logframe Matrix is a standardized table that summarizes the important aspects of a project.

Logical Frameworks Project Description Performance Indicators Means of Verification Assumptions Goal: The broader development impact to which the project contributes - at a national and sectoral level. Measures of the extent to which a sustainable contribution to the goal has been made. Used during evaluation. Sources of information and methods used to collect and report it. Purpose: The development outcome expected at the end of the project. All components will contribute to this Conditions at the end of the project indicating that the Purpose has been achieved and that benefits are sustainable. Used for project completion and evaluation. Assumptions concerning the purpose/goal linkage. Component Objectives: The expected outcome of producing each component's outputs. Measures of the extent to which component objectives have been achieved and lead to sustainable benefits. Used during review and evaluation. Assumptions concerning the component objective/purpose linkage. Outputs: The direct measurable results (goods and services) of the project which are largely under project management's control Measures of the quantity and quality of outputs and the timing of their delivery. Used during monitoring and review. Assumptions concerning the output/component objective linkage. Activities: The tasks carried out to implement the project and deliver the identified outputs. Implementation/work program targets. Used during monitoring. Assumptions concerning the activity/output linkage. Speaker Notes: A Logframe Matrix is a standardized table that summarizes the important aspects of a project. There are two examples of logical frameworks matrices in your handouts, and this slide is replicated there as well. Notice that the matrix contains all elements important to a program: goal, purpose, objectives, outputs (meaning the measurable results produced by the program), and activities that will be carried out to achieve the outputs. For each of these, indicators are specified and the means for verifying these is also listed. In the last column, assumptions made for each program element are also described.

Logical Frameworks Taskforce on Communicable Disease Control in the Region X: Tuberculosis GOAL Reduced burden of TB to reach global average levels Further development of multi-drug resistant TB (MDR-TB) prevented   PERFORMANCE INDICATORS Notification rate B-1. Treatment outcome B-2. Prevalence of Multi-drug Resistance in “new” and previously treated TB patients MEANS OF VERIFICATION Annual notification reports (surveillance) B-1. Annual reports on outcome of treatment (cohort analysis) B-2. Periodic reports on surveillance of anti-TB drug resistance  ASSUMPTIONS - A dual HIV/TB epidemic causing increase in TB incidence does not occur - Control of private practitioner and pharmaceutical sectors to prevent MDR - Prevalence of resistance to second line anti-TB drugs low enough at the outset so as not to seriously compromise treatment success ratio Speaker Notes This example of a Logical Framework or LogFrame Matrix is for tuberculosis programs in Region X. Only a portion of the matrix is shown here, however you can see the complete matrix in your handout to see how detailed and complete these matrices are. On this slide, notice there are two goals. Let’s follow the first goal across the matrix. The goal of reducing the burden of TB to reach European average levels has as its indicator the notification rate. This information will come from annual surveillance reports. There are a few assumptions relevant to both goals, including that the HIV epidemic does not increase TB incidence, that multiple drug resistance can be controlled via the health care and pharmaceutical sectors, and that this resistance is low to begin with. Additional Speaker Notes Although this framework appear technical in nature, it is not necessary to understand the clinical elements of tuberculosis programs in order to follow it. MDR stands for multi drug resistance.

Logical Frameworks Taskforce on Communicable Disease Control in the Barents and Baltic Sea Regions: Tuberculosis PURPOSE   [Implementing] cost-effective measures for the prevention and control of TB operating within civil and penitentiary health services in the Task Force area PERFORMANCE INDICATORS   1. Coverage of TB programmes in line with international recommendations. 2. Proportion of patients defaulting out of patients treated. 3. Proportion of previously treated cases among all cases. 4. Proportion of patients on ambulatory treatment out of all patients treated. MEANS OF VERIFICATION Annual reports National / local annual notification reports (surveillance) 4. Annual record reviews during site visits (consecutive series of patients) ASSUMPTIONS - Stable political situation, sustained political commitment and financing - Sufficient numbers of competent health care personnel in the government sector Speaker Notes The LogFrame continues here with the purpose of the program, to implement cost-effective measure to prevent and control TB. The task force identified four indicators and their data sources that correspond with this purpose.

Logical Frameworks Taskforce on Communicable Disease Control in the Barents and Baltic Sea Regions: Tuberculosis (only one output and related activities shown here) OUTPUTS   8. Measures to increase awareness of TB and its treatment among all members of the community developed and tested PERFORMANCE INDICATORS   8.1. Number of pamphlets / posters printed and distributed annually 8.2. Awareness of TB among target groups MEANS OF VERIFICATION 8.1. Material produced / distributed 8.2. KAP or other surveys (before/after) ASSUMPTIONS Relevant persons motivated to participate Professional interest, sufficient financing Target groups interested in (their) health and able to participate  ACTIVITIES   8.1 Identify groups at risk for TB 8.2 Develop advocacy material suitable for all target groups (not only risk groups) 8.3 Organize health education directed at all target groups 8.4 Involve the mass media  INPUTS MEANS OF VERIFICATION Financial management reports ASSUMPTIONS Speaker Notes In this slide, one output from the TB Log Frame Matrix has been selected for illustration. One output is to develop and test methods to increase awareness of TB and its treatment. The indicators include material produced and distributed and awareness among target groups. Again, the data sources and the relevant assumptions are also listed. Then, the four activities related to this output are listed. In this matrix, no inputs or assumptions were listed, however in a complete matrix they would be included.

Activity Return to small groups from previous activity Develop goals and objectives for a program aimed at increasing utilization of FP/MCH health services Develop a results or logical framework for this program In 45 minutes, a member of each group will share the goals, objectives and framework Speaker Notes Get back into the same groups you were in to develop the conceptual frameworks. Your next task will be to come up with goals and objectives for a program aimed at increasing utilization of family planning/maternal and child health services. To do this, each group will need to consider the nature of the activities that comprise the program. Because this is something your group is making up, there is no right answer as to the type of activities involved in this fictitious program. Based on these goals and objectives, develop A results framework, identifying the strategic objective, intermediate results and sub-intermediate results. -OR- (B) A logical framework, identifying the purpose, component objective, outputs, and activities. You do not have to construct indicators for this exercise. You have 45 minutes to complete this exercise. I will let you know when there are 15 minutes remaining. At the end of this exercise, you should have your goals and objectives and framework written out with the marking pens on the large paper. If there is not time to share them with the larger group, we will post them on the walls and you are encouraged to look at them during the break and compare what each group produced. Additional Background Monitor the groups to make sure they are progressing and to answer questions. If a group is not able to agree on goals and objectives, they will not be able to move forward with a framework. It may be necessary to intervene in this case, in order to move things along and ensure all groups have practice with results or logical frameworks. Activity Recommended materials for this activity are: Large piece of poster board or flip chart paper, two per group Post-it notes, each group should have some remaining from the previous activity Thick marking pens, each group should have pens from the previous activity

Logic Models Purposes: Diagrams that identify and illustrate the linear relationships flowing from program inputs, processes, outputs, and outcomes. Inputs or resources affect Processes or activities which produce immediate results or Outputs, ultimately leading to longer term or broader results, or Outcomes. Purposes: Provides a streamlined interpretation of planned use of resources and desired ends Clarifies project/program assumptions about linear relationships between key factors relevant to desired ends Other terms used: M&E Frameworks, Logical Frameworks Speaker Notes Logic models are diagrams that identify and illustrate the linear relationships flowing from program inputs, through processes, outputs, and leading to outcomes. Inputs, or the program’s resources, provide the fuel for Processes, or the program’s activities. These processes produce immediate or direct Outputs or Effects; in other words, the program’s activities yield some intermediate results. These Outputs lead to long-term or broader, overall results, or the program’s Outcomes. A logic model is a useful tool for identifying and illustrating operational factors important to achieving a successful ultimate outcome. For this reason, they are sometimes perceived as the most useful for program monitoring. Since it is the only framework that specifies inputs, it is also the only model that can directly point toward indicators for process monitoring. One of its strengths is that it can help identify linkages and key processes, highlighting the relationship between resource allocation and success, as well as where problems may lie if goals are not being achieved. Logical frameworks provide a streamlined perspective on the most critical processes contributing directly to program outputs and outcomes, and clarify the linear relationships between program decisions, activities, and products. Logic models provide more detail about the activities and the necessary inputs than results frameworks do. Logic models are also sometimes called log frames or logical frameworks. Since we use the term “logical frameworks” to label the matrix presented previously, we use the term “logic model” here. Although terminology regarding frameworks may vary from organization to organization, what is important is to understand the differences between types of frameworks. Additional Background “Logic models” are used in many organizations, and participants may have specific notions about the labels that should be affixed at each level (e.g. outputs versus effects). It is unlikely to be a productive use of time to debate these semantics from the various organizational perspectives. The point here is to explain and illustrate the general idea and usefulness of logic models so that participants see that they can help clarify the linear flow connecting resources, allocation decisions, and program effectiveness.

Logic Models: Training INPUT PROCESS OUTPUT OUTCOME IMPACT Develop clinical training curriculum Conduct training events Practitioners trained in new clinical techniques Increase in clients served by (newly) trained providers Declining morbidity levels in target population Speaker Notes This logic model presents a straightforward view of a project designed to reduce population morbidity due to untrained or unskilled health care providers. As you can see, it does not try to account for all of the factors that may be influencing operation and results like a conceptual model, but instead focuses on the project’s activities and impacts. This narrow focus assists program managers and M&E planners as they clarify the direct relationships among elements of particular interest within a particular program effort. Also, unlike the LogFrame Matrix, no activities are listed here. Sometimes these models do include assumptions between inputs and processes, processes and outputs, outputs and outcomes, and outcomes and impact. Additional Background Logic models can usefully diagram the flow of program operations, while keeping the ultimate goals in view. Process indicators are sometimes useful for monitoring, but may provide most insight if used in the context of an evaluation -- to pinpoint glitches or breakdowns within the project, for instance, if objectives are not being fully met.

Logic Models Voluntary Counseling and Testing (VCT) Problem Statement: HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services. INPUTS PROCESSES OUTPUTS OUTCOMES IMPACTS Financial Resources Supervision and training for VCT personnel Clients (HIV+ and -) develop & adhere to personalized HIV risk-reduction Risk behaviors decreased HIV transmission rates decreased People know their HIV status Demand for Services Provide pre-test counseling Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples Infrastructure HIV+ Clients develop & adhere to personalized HIV care, support, and treatment plans Provide HIV testing HIV+ people are referred to appropriate services HIV incidence decreased VCT MIS** VCT protocols, guidelines, and training documents** Provide post-test counseling Refer HIV+ clients to appropriate services (PMTCT, care and support, treatment) HIV morbidity & mortality decreased Counseling and Testing Personnel* Health outcomes of HIV + improve Speaker Notes Within a program, there are several activities with their own inputs and outputs. Collectively the outputs of the activities contribute to the program outcomes and impacts. In some cases the output of one program activity could be an input for another activity. This is an example of a logic model for a VCT Program. the inputs and processes and outputs of VCT service delivery are presented in a logically connected manner. Arrows are used to show connections. In order to make this easier to follow, in this example, only arrows connecting to the process of HIV testing are depicted. From the inputs column financial resources, demand for services, infrastructure, a VCT information system, VCT protocols and guidelines, testing personnel with skills in phlebotomy and testing are joined with the group of people who have gone through the process of counseling. The output of the VCT service requires another step in the process: people who have been tested need to be counselled about their results. The output of the VCT process is that people know about their status. In some cases programs will develop logic models for each activity. For example, if an activity is to develop guidelines, the output of that activity is the guideline, which is an input in this overall logic model for VCT service delivery. Client records are available Continuity of care is available HIV test kits Maintain patient records Referral system for prevention & Tx services** Service reports are produced Program and services are improved Complete reporting requirements

SO1: Increase tuberculosis case detection rate to 70% Results Framework Tuberculosis (TB) Control Programs SO1: Increase tuberculosis case detection rate to 70% IR1: Increased availability of quality services IR2: Increased demand for quality services IR1.1: Services increased IR2.1: Customer knowledge of TB improved Speaker Notes This slide was presented earlier as an example of a results framework for a tuberculosis program. The next slide shows a logic model for the portion of the framework relating to Intermediate Result 2: Demand for quality services increased and also for the Sub-Intermediate Result of customer knowledge of tuberculosis improved. IR1.2: Practitioners’ skills and knowledge increased IR2.2: Social support for TB practices increased IR1.3: Improved program management Source: MEASURE Evaluation, 2004 (draft)

Logic Model INPUT PROCESS OUTPUT OUTCOME IMPACT Portion of model for tuberculosis control relating to increasing demand for quality services INPUT Human and financial resources to develop and print educational brochure PROCESS Distribute brochure to health facilities Meet with physicians to promote distribution of brochure OUTPUT Brochure distributed to clients of facilities OUTCOME Increased customer knowledge of TB transmission and treatment Increased demand for quality TB services IMPACT Decreased TB infection, morbidity and mortality Speaker Notes This logic model shows how it relates to the results framework for the same program. Each activity related to a results framework could have its own logic model. In this example, the input is the human and financial resources to design and print and educational brochure about tuberculosis. The process, or program activities related to this will be the distribution of the brochure and making sure the physicians know about the brochure and promote it among their clients. The anticipated output is that the brochure is distributed to clients of the facilities. The outcome is where the model relates to the results framework. The aim of these activities are to increase customer knowledge of TB transmission and treatment, the Sub-Intermediate Result, which will lead to an increased demand for quality TB services, the Intermediate Result. The desired public health impact of the program is that TB infection, morbidity and mortality will be decreased as a result.

Activity Return to small groups from previous activities Develop logic models for one of the activities of your program to increase utilization of FP/MCH health services After 20 minutes, a member of each group will share one of the logical frameworks Speaker Notes Get back into your small groups for these framework activities. Now you will practice developing logic models. Select one activity for which you will develop a logic model. Then, follow the inputs involved in that activity through the logic model. You will have 20 minutes for this activity, and if there is time, each group will share one framework with the larger group. I will let you know when 10 minutes remain. At the end of this activity, you should have one logic model written out on the large paper to be put on the wall. As you are completing this activity, be sure to refer back to the conceptual framework and the results or logical framework your group developed and to notice the relationships between the different types of frameworks. Additional Background Monitor the groups to make sure they are progressing and to answer questions. Activity Recommended materials for this activity are: Large piece of poster board or flip chart paper, one per group Thick marking pens, each group should have pens from the previous activity

Frameworks for M&E Planning Purposes: clarifying assumptions, goals, and interrelationships between factors relevant to the project or program defining objectives selecting activities defining levels of performance and desired results in terms of planned activities and realistic, objective impacts Monitoring and evaluation plans incorporate: program managers’ assumptions and objectives, in a given context a schematic design displaying the directional linkages between key program elements and/or planned results, and other relevant factors Speaker Notes Now that we have discussed different types of frameworks, let’s turn back to the purpose of designing M&E frameworks. One main purpose is the clarification of the program’s assumptions, goals, and the known or expected relationships among program and environmental factors that may affect the effectiveness of the activities or the outcome of the intervention. Another important purpose of M&E framework design is to define clear levels of results that should occur as the intervention unfolds. These should be realistic and objective impacts that can be measured and assessed. M&E frameworks rest on the assumptions and objectives of the program within its operating environment. Drawing on those expectations, the M&E framework provides a schematic design showing how various relevant factors, results, and overall outcomes are linked. Additional Background Participants should be clear on several points: that the process of framework design is both useful in itself, to clarify project and partner understandings of the key assumptions and other necessary elements for a successful intervention, and that it is essential in order to develop an appropriate and useful M&E plan. Without developing a consensus on the foundations and context of the program and its success, monitoring and evaluation cannot be managed in a coherent or comprehensive way in order to contribute to effective management and the achievement of that success.

Summary of Frameworks Type of Framework Brief Description Program Management Basis for Monitoring and Evaluation Conceptual Interaction of various factors Determine which factors the program will influence No. Can help to explain results Results Logically linked program objectives Shows the causal relationship between program objectives Yes – at the objective level Logical Logically linked program objectives, outputs, and activities Shows the causal relationship between activities and objectives Yes – at the output and objective level Logic model Logically links inputs, processes, outputs, and outcomes, Shows the causal relationship between inputs and the objectives Yes – at all stages of the program from inputs to process to outputs to outcomes/ objectives Speaker Notes: We have covered four types of frameworks.  These four frameworks are used to inform programming and for monitoring and evaluation of the program that is implemented.  The conceptual framework puts the health problem in to context and either a results or logical framework is used to understand the interrelationships between factors relevant the program. USAID funded programs tend to use results  frameworks but many others, such as DFID and UN programs, use logical frameworks. Programs should use the type of frameworks that best suit them.  Logic models help to show the logical connections between the inputs and processes and outputs of an activity and its link to the programs objectives (outcomes) and goals (impact).

References AusGuide. The Logical Framework Approach. http://www.ausaid.gov.au/ausguide/ausguidelines/1-1-1.cfm Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project. Global AIDS Program. (2003) Monitoring and Evaluation Capacity Building for Program Improvement Field Guide, Version 1. U.S. Centers for Disease Control and Prevention, Atlanta, GA. Marsh, David. 1999. Results Frameworks & Performance Monitoring. A Refresher by David Marsh (ppt) http://www.childsurvival.com/tools/Marsh/sld001.htm Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. UNICEF. 1998. State of the World’s Children.

Group Projects Form groups for your group projects For your project, develop Goals and objectives Conceptual framework One other framework: results, logical or logic model Speaker Notes In the time remaining today, get into the groups for the group project that you will undertake for the entire workshop. What we covered today will be fundamental to your M&E plans. For your project you are asked to develop goals and objectives, a conceptual framework and one other framework of your choice. There is not enough time to finish these tasks today, however there will be daily opportunities to work on your projects.