ROM reviews Saskia Van Crugten NSA-LA Thematic programme - Management of the Project Cycle and Monitoring of Project Implementation Brussels, 17-18 September 2013 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

Origin of ROM reviews May 1999: the Council of the EU invited the EC to establish a performance-based monitoring system with the objective to strengthen Monitoring, Evaluation and Transparency. EuropeAid established the Results-Oriented Monitoring system (ROM). Initially, ROM bridged the gap between EC Task Managers (Brussels) and the project (in the field)… …today however, most EC Project Manager work in Delegations. 2 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

Main features of ROM system OECD DAC evaluation criteria PCM / Logical Framework Approach For on-going and closed projects/programmes (on-going ROM, ex-post ROM, regional ROM) Standard methodology for all projects Selection of projects for ROM Mission procedure Questions guiding the assessment Report formats Grading system External experts Review of the progress made in project implementation result-oriented because not only focused on inputs and outputs but also ontoutcome and impact +/- 1 per year and per country All projects over 1 million euros + selection of smaller Independant experts (external to project management and to EC staff) ROM reviews: a “snapshot” of the project’s quality and performance 3 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

Selection of actions for ROM Interactive process between ROM contractors, EU Del, ROM Task Manager, Headquarters All projects with budgets exceeding 1 million € and a sample of smaller projects All projects that have been implemented for at least six months and at least six months remaining before closure 6 months ROM period Start Implementation End ROM should be coordinated with evaluation activities (e.g. not within six months of a mid term review). Currently, roughly 1/3 of DEVCO projects/programmes are “ROMed” every year. 4 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

Timing of ROM missions The ROM exercise is to be timely and quick. 5 days for field visit of an ongoing project. 1 day preparation 2 days reporting. Ex-post ROM can require more days (up to 10) especially to identify interlocutors. ROM reports available 15 days after the field visit. 5

ROM documents Project Synopsis (PS): a concise overview on project background, intervention logic and administrative data. Background Conclusion Sheet (BCS): guiding questions to assess a project and give grades for each of the criteria ROM Report (MR): findings and grades for ROM criteria, conclusions and recommendations. Response Sheet (RS): comment on report quality, reply to recommendations and feedback from other stakeholders (to be filled in by Task Manager). Cut paper work… Keep the essential… 6 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

ROM assessment criteria - BCS BCS = Excel file with 1 sheet criteria + sub-criteria ROM assessment criteria - BCS Relevance 1.1 For target group 1.2 For partner government 1.3 For E.U. Quality of design 1.1 Intervention logic 1.2 Stakeholder support 1.3 Crosscutting issues in design Efficiency 2.1 Availability of means / inputs 2.2 Implementation of activities 2.3 Output achievement 2.4 Partner contribution Effectiveness 3.1 Output achievement 3.2 Outcome achievement Impact Prospects 4.1 Direct impact 4.2 Indirect impact Potential Sustainability 5.1 Financial/ economic viability 5.2 Target group ownership 5.3 Policy support 5.4 Institutional capacity Horizontal Issues 6.1 Quality Systems, Monitoring and Evaluation 6.2 Technical Cooperation 6.3 EC visibility Cross-cutting issues 7.1 Gender 7.2 Environment 7.3 Good Governance 7.4 Human Rights 7 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

Using the BCS 2. 1. 8 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

Grades strongly reduce the complexity! ROM grading system Gives a quick overview of its performance. Allows for a comparison and aggregation. Meant to be a summary of the assessment. Don’t focus too much on grades at the expense of substance… Grades strongly reduce the complexity! 9 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

ROM performance categories Made to describe the performance of a project as a whole; not quantitative averages of the grades for the performance criteria, but qualitative categorisations. NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1

Why ROM reviews ? For project managers and EC task managers: ROM informs stakeholders on project performance and helps to focus on results. For higher level of EC management: ROM provides statistics on overall portfolio performance and highlights problems and success stories. For knowledge production: ROM can be a source of lessons learnt via meta-analysis/evaluations). For project beneficiaries: opportunity to discuss project progress and outputs. 11

What a ROM review is not ? An Audit A mechanism replacing internal monitoring by EC staff An in-depth analysis of a project/ programme An assessment of EC Task Managers performance

ROM and Quality assurance in the project cycle Ex-Post ROM Programming Ex-Post Project Evaluation Follow-Up Identification Final Project Evaluation --Closure-- oQSG 1 PPCM Formulation oQSG 2 Ongoing ROM Internal Monitoring Project Mgmt DEL managed HQ managed Accountability Implementation Mid term Project Evaluation Lessons Learnt Ongoing ROM 13

ROM reviews for supporting project management Using external experts ROM offers fresh eyes to scrutinize a project: Is the project, with all due flexibility, pursuing its planned course and will it attain its objectives? What can or should be done to improve project performance? Are there issues which could be subject to a more in-depth analysis (evaluation)? Rom reviews have their limits, but the tool exists and can help. Make the best of it! 14 NSA/LA thematic programme - Capitalisation seminar - Day 1 pm 1