Results Oriented Monitoring & Evaluation Model (ROMEM) Presented by Shawn Grey Ministry of Transport and Works Jamaica. Model developed by MTW and DPMI Consultants 2007

Performance Monitoring and Evaluation Refers to monitoring and analysis of policies and projects as they are implemented in order to determine how they are performing with regard to their intended objectives. (TRB,2001). This can assist with decision making, problem identification and provide guidance for optimization of resources.

The Results Oriented Monitoring and Evaluation Model (ROMEM) is a set of planning and reporting procedures and processes that integrates both the budget tracking system with the performance of investment projects. The Results Oriented Monitoring and Evaluation Model (ROMEM) is a set of planning and reporting procedures and processes that integrates both the budget tracking system with the performance of investment projects. The aim is to enable the Ministry to monitor and evaluate the performance of the MTW and its Agencies in an effective and efficient manner. The aim is to enable the Ministry to monitor and evaluate the performance of the MTW and its Agencies in an effective and efficient manner. When the ROMEM is fully implemented the MTW will be able to offer real-time response to the Agencies with respect to their performance. When the ROMEM is fully implemented the MTW will be able to offer real-time response to the Agencies with respect to their performance. What is ROMEN

Purpose of the ROMEM To support the improvement data collection, collation, analysis and dissemination To support evidence-based decision making that is influenced by information on the progress, outcome and impact of programmes Ensure transparency and accountability, promote and improve service delivery and compliance with statutory and other requirements To support the adoption of a learning culture within Jamaica’s public sector To support the adoption of a learning culture within Jamaica’s public sector

How to Use the ROMEM The results-oriented M&E model is not a single methodology. It presents a set of guidelines that when taken together presents a range of tools, techniques and approaches which may be adapted to different situations The use of the guidelines requires a comprehensive understanding of results based management, monitoring and evaluation (data collection, aggregation and analysis etc.)

Components of the ROMEM Activities Activities implement strategies Outputs outputs are the short-term logical consequences of activities Therefore, data collection on output achievement should begin shortly after the completion of an activity. Outcome outcomes are the logical consequences of a combination of outputs, and will become evident when a sufficient number and type of outputs have been achieved. Therefore baseline data and periodic measurement should be done in order to adjust the implementation plan so as to enhance the probability of successfully achieving the outcome level results. Impact impact level results are a logical consequence of a combination of outcomes. Outcomes are generally achieved around programme/project termination, therefore evaluation of programme impact is best done at the policy level (e.g. by PIOJ) or by the donor (e.g. IDB). Continuous tracking of expected results. Establish partnerships that build capacity. Monitoring Strategy Prepare and plan evaluation; Implement and share the results. Evaluation Strategy Evaluation Plan: Why, Who, What, Where When, How Logical Results Chain Use information collected during monitoring to inform evaluation. Establish a structured plan for data collection and analysis. Decide on who, what where and when. Performance Measurement Strategy Data Collection Strategy and Plan Prepare and disseminate Monitoring and Evaluation Reports Reporting Strategy

Existing Tools Performance Measurement Strategy Performance Measurement Matrix Indicator Matrix Data Collection Plan Monitoring Strategy Template for monitoring milestones IDB’s Risk Analysis Tool Kit Quarterly Performance Review Reports Electronic Performance Management System Evaluation Strategy Evaluation Plan Ex Ante Evaluation Evaluation Framework Evaluation Matrix Template Reporting Performance Monitoring Reporting Framework Project Cycle Management Accountability Framework (PCMAF) Performance Quarterly Review Reports Performance Reviews of the Units and Public Sector Entities (Performance Quarterly [PQs]). Evaluation Report Template

Critical Tasks: 1. Formulate a model to make the Key Performance Indicators (KPI’s) of the Agencies Analyzable. - Make them Quantifiable - make them - Make them Rank-able activity/results-oriented 2. Formulate a Strategy to Weight the Strategic Objectives of the Ministry of Transport & Works (MTW) & the KPI’s of its Agencies.

The Ranking Model A model was developed in Microsoft Excel to rank and analyze the KPI’s of the Agencies. This entailed (in most cases) formulating three (3) components interpreted / extrapolated from each KPI Statement, namely: * Determining a No-Control Parameter * Deciding a Target and, * Deciding on a Result Parameter

These three parameters will, in essence, allow / ensure that the Agencies do the following: ☺ ☺ Look retrospectively on previous attainments and become aware of tasks outside of their purview that impact their KPI’s. ☺ ☺ Become aware that each KPI is indeed a partial measure of their performance. ☺ ☺ Have a full grasp of exactly what is to be achieved through each KPI. ☺ Know exactly how the KPI’s will be met. ☺ Know when the KPI’s are met.

Weighting Assigned to The Driving & Outcome Indicators

The Ranking Sheet

The Scoresheet

Example: Transport Authority Indicator Weightings

Electronic Performance Monitoring System MTW Jamaica MTW Jamaica Developed in 2007 with the assistance of the IADB PRODEV programme