Assoc. Prof. Dr. Wongsa Laohasiriwong Faculty of Public Health, Khon Kaen University Planning for Measurement, Monitoring and Evaluation the Reform 1

2 What is it?

Monitoring and Evaluation (M&E) Plan a guide what you should evaluatewhat information need who is a guide as to what you should evaluate, what information you need, and who you are evaluating for. 3 Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159

4 Characteristic of M&E Plan Utility Utility - serve practical information needs of intended users. Feasibility Feasibility - be realistic, prudent, diplomatic and frugal. Propriety Propriety - conducted legally, ethically, and with regard to those involved in and affected by the evaluation. Accuracy Accuracy - reveal and convey technically accurate information. Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159

5 Developing & Implementing an M&E Plan: Logic Model Inputs Human resources Understanding of the program Authority and mandate Stakeholders Processes Advocate Assess strategic information needs Assess information systems capabilities Achieve consensus and commitment Develop mechanism for M&E plan review Prepare document for final approval Impacts Improved health status Long-term Evidence-based decisions for improving programs Outcomes Output M&E Plan Document Short-term M&E System for obtaining Strategic Information decision making

Performance Measurement 6 ongoing monitoring and reporting of program accomplishments, towards pre-established goals. is the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals. Source: Stephanie Shipman and Joseph Wholey Performance Measurement and Evaluation: Definitions and Relationships

How to Develop a Monitoring and Evaluation Plan? Step 1. - evaluation audience Step 1. - Identify your evaluation audience. Step 2. - evaluation questions. Step 2. - Define the evaluation questions. Step 3. - monitoring questions. Step 3. - Identify the monitoring questions. Step 4. - indicators and data sources. Step 4. - Identify the indicators and data sources. Step 5. - who is responsible Step 5. - Identify who is responsible for data collection and timelines. Step 6. - who will evaluate the data, how Step 6. - Identify who will evaluate the data, how it will be reported, and when. Step 7. - Review Step 7. - Review the M&E plan. 7 Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159

Characteristics of Good Monitoring Systems Follows the reform Follows the reform process. Cost-effective Cost-effective for the operating unit. stages or events used for rating progress Describes the stages or events used for rating progress. methodology or process Describes the methodology or process used for rating the current stage of the reform. 8 Source: Patricia Vondal. USAID Monitoring the policy reform process

Characteristics of Good Monitoring Systems (cont.) Defines key terms Defines key terms e.g. “operational,” “fully functioning,” or “fully implemented”. activities are most critical to the strategic objective. Specifies which activities are most critical to the strategic objective. 9 Source: Patricia Vondal. USAID Monitoring the policy reform process Provides a rationale Provides a rationale for how future performance targets are set. stakeholders as partners Includes reform stakeholders as partners in reviewing the process of adoption and implementation and in setting future performance targets.

Approaches to Monitoring 10 Source: Patricia Vondal. USAID Monitoring the policy reform process Answer for How much?, How Many? 1) Simple Quantitative Approaches (SQA) Address Why? What was happened? 2) Descriptive Approaches (DA) Using qualitative and quantitative methods Detailed explanation to define each stage in a given index and to describe the related scoring scale. 3) Composite Approaches (CA)

1) Simple Quantitative Approaches (SQA) 11 common way of showing the progress Using a purely quantitative approach is a common way of showing the progress. three-year periodeach year records the number of reforms successfully completed Typically the agency promotes a list of reforms during a three-year period, and each year it records the number of reforms that have been successfully completed. Source: Patricia Vondal. USAID Monitoring the policy reform process

2) Descriptive Approaches (DA) 12 reforms is frequently not readily quantifiable. The progress in adopting and implementing reforms is frequently not readily quantifiable. the key events reform process planned results monitor their achievement It needs clearly define in advance the key events in the reform process and the planned results and then monitor their achievement. actual results throughout the performance period. This permits a relatively objective comparison of planned with actual results throughout the performance period. Source: Patricia Vondal. USAID Monitoring the policy reform process

3) Composite Approaches (CA) (Qualitative and Quantitative Methods) 13 complex indices Based on constructing more complex indices. phases or stages identified as milestone events expressed numerically These indices are frequently (but not necessarily) based on phases or stages identified as milestone events and are expressed numerically. number or value. Each phase or stage in the reform process is assigned a number or value. Missions should provide a detailed explanation define each stage be imprecise and subjective. Missions should provide a detailed explanation to define each stage in a given index and to describe the related scoring scale. If not thoughtfully considered and well documented, can be imprecise and subjective. Source: Patricia Vondal. USAID Monitoring the policy reform process

MonitoringMonitoring 14 collecting, recording, and reporting is collecting, recording, and reporting information concerning performance that tracking of services and programmes performances. Routine Working Routine Working Source: Chapter 10 Project Monitoring and Control.

15 Why do we monitor? we know that things don’t always go according to plan Simply because we know that things don’t always go according to plan (no matter how much we prepare). how well a policy or programme is achieving To better assess how well a policy or programme is achieving its intended target. detect and react changes to plans To detect and react appropriately to deviations and changes to plans Source: Chapter 10 Project Monitoring and Control.

16 What do we monitor ? Men (human resources) Machines Materials Money Technology Space Time Tasks Quality/Technical Performance What we looking for? Progress, Schedule, Costs, Performances: Work begin, the completion What we looking for? Progress, Schedule, Costs, Performances: Work begin, the completion Source: Chapter 10 Project Monitoring and Control.

17 When do we monitor?  End of the Plan  Continuously  Regularly  Logically  While there is still time to react  As soon as possible  At task completion  At pre-planned decision points (milestones) Source: Chapter 10 Project Monitoring and Control.

18 How do we monitor? Through information collection. Through meetings with stakeholders. For schedule – Update Gantt Charts Using Earned Value Analysis Calculate Critical Ratios Milestones Reports Tests, Inspections, Auditing Source: Chapter 10 Project Monitoring and Control.

EvaluationEvaluation systematic process of the merit, value and worth is the systematic process of the merit, value and worth of someone or something.

Evaluation Terms  Efficacy  Efficacy : Can it work? (normally for medicine)  Effectiveness  Effectiveness : Does it work in reality?  Efficiency  Efficiency : Is it worth doing it, compared to other things we could do with the same money? (Cost-effectiveness) Evaluation

EffectivenessEffectiveness the connection between inputs and outputsachievement of objectives Effectiveness refers to the connection between inputs and outputs, the achievement of objectives, means how well the job gets done i.e. the quality of the output.

EfficiencyEfficiency outputs are attained while minimizing production costs, least amount of resources. Efficiency refers essentially to the extent to which outputs are attained while minimizing production costs, achievement of the ends with the least amount of resources. There are 2 types of Efficiency 1. Technical efficiency 2. Allocative efficiency

Cost-Benefit and Cost-Effectiveness Analyses 23 Types of Evaluation Types of Evaluation Outcomes Impacts Inputs Activities Outputs Implementation Results Impact Evaluation Outcome Evaluation Process Evaluation

Define the study question and perspective (1) Define the study question and perspective Describe alternative health programs to be considered determine study perspective (decision making context of the study) Define the study question and perspective (1) Define the study question and perspective Describe alternative health programs to be considered determine study perspective (decision making context of the study) Derive decision rules (4)Derive decision rules Compare cost–benefit ratios, identify other decision criteria address implementation issues Derive decision rules (4)Derive decision rules Compare cost–benefit ratios, identify other decision criteria address implementation issues Identify, measure, and value costs and benefits (2) Identify, measure, and value costs and benefits Identify relevant costs and benefits for each program measure costs and benefits for each program in appropriate physical units value costs and benefits for each program, using market valuation of factor inputs, stated preference valuations of health gains, or other valuation methods Identify, measure, and value costs and benefits (2) Identify, measure, and value costs and benefits Identify relevant costs and benefits for each program measure costs and benefits for each program in appropriate physical units value costs and benefits for each program, using market valuation of factor inputs, stated preference valuations of health gains, or other valuation methods Analyse costs and benefits (3) Analyse costs and benefits Adjust costs and benefits for differential timing (discounting) where appropriate perform incremental analysis of costs and benefits of alternatives perform sensitivity analysis to allow for uncertainty in estimates of costs and consequences Analyse costs and benefits (3) Analyse costs and benefits Adjust costs and benefits for differential timing (discounting) where appropriate perform incremental analysis of costs and benefits of alternatives perform sensitivity analysis to allow for uncertainty in estimates of costs and consequences Four Stages of Evaluation

Once an organization has analyzed its mission, identified all its stakeholders, and defined its goals, it needs a way to measure progress toward those goals. 25 Key Performance Indicators are those measurements.Key Performance Indicators are those measurements. Source: F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm

Key Performance Indicators KPIKey Success Indicators (KSI), Key Performance Indicators, also known as KPI or Key Success Indicators (KSI), help an organization define and measure progress toward organizational goals. 26 Source: F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm

Key Performance Indicators (KPI) “Measurable indicators that will be used to report progress that is chosen to reflect the critical success strategic” 27 Source: John McGerty, Key Performance Indicator

Specific Measurable Acceptable Realistic Time frame Source: Clara Freeman.

Specific make sure the different users interpret it the same way result, come to the same and right conclusions There has to be one widely accepted definition of the KPI to make sure the different users interpret it the same way and, as a result, come to the same and right conclusions which they can act on. Source: John McGerty, Key Performance Indicator

Measurable Measurable define a standard, budget or norm, to make it possible to measure the actual value The KPI has to be measurable to define a standard, budget or norm, to make it possible to measure the actual value and to make the actual value comparable to the budgeted value. Source: John McGerty, Key Performance Indicator

Achievable important for the acceptance of KPI’s and Performance Management in general within the organization that it possible to achieved. It is really important for the acceptance of KPI’s and Performance Management in general within the organization that it possible to achieved. Nothing is more discouraging than striving for a goal that you will never obtain. Source: John McGerty, Key Performance Indicator

Realistic action need to happen. The KPI must give more insight in the performance of the organization in obtaining its goal and strategy. Therefore the action need to happen. Source: John McGerty, Key Performance Indicator

Time Frame Time Frame important to express the value of the KPI in time. It is important to express the value of the KPI in time. The realization and standardization of the KPI therefore has to be time frame. Source: John McGerty, Key Performance Indicator

A business A business – the percentage of income that comes from return customers. 34 A Health Promotion Hospital A Health Promotion Hospital – The percentage of coverage Polio vaccination in children age under 5 years old. A school A school – the graduation rates of students. A Customer Service Department A Customer Service Department – the number of monthly customer complaint.

Benefits of Evaluation Finding Important input for government decision making and prioritization Important input for government decision making and prioritization, particularly in the budget process. revealing the performance of ongoing activities Assists managers by revealing the performance of ongoing activities at the project, program or sector levels. assess the performance of organizations and institutional reform processes. Used to assess the performance of organizations and institutional reform processes. Contribute to accountability mechanisms Contribute to accountability mechanisms, whereby managers and governments can be held accountable for the performance of their activities. 35 Source: Keith Mackay Public Sector Performance-The Critical Role of Evaluation. The World Bank Washington, D.C.

References: John McGerty, Key Performance Indicator F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm Clara Freeman. smarter-means-for-the-freelance-entrepreneur Patricia Vondal. USAID Monitoring the policy reform process. Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159. Stephanie Shipman and Joseph Wholey Performance Measurement and Evaluation: Definitions and Relationships. Keith Mackay Public Sector Performance-The Critical Role of Evaluation. The World Bank Washington, D.C. Chapter 10 Project Monitoring and Control. 36