Results-oriented Monitoring Comissão Europeia Results-oriented Monitoring Seminário sobre Resultados da Ajuda Externa e Metodologias de Avaliação da CE, em Lisboa 9/3/2007 Alexandra Chambel EuropeAid/ Unidade de Avaliação http://ec.europa.eu/europeaid/evaluation/index.htm

How is it done? Independent experts Ongoing projects Rapid assessment using a strictly structured consistent methodology against internationally agreed criteria: Relevance Efficiency Effectiveness (Potential) Impact (Potential) Sustainability 12/11/2018

Outputs A two page report is uploaded in a database allowing for aggregations, simple statistics and comparisons. It provides, short explanations and recommendations on quality improvement + important complementary information to internal monitoring A response from the field on quality and usefulness of the report is also uploaded, projects synopsis , etc 12/11/2018

The assessment gives a picture of the situation helping : Aims The assessment gives a picture of the situation helping : to improve the individual projects quality performance to have an overview over the performance of the project portfolio; how is it progressing towards results 12/11/2018

4247 projects in 172 Countries have been recorded. Some Numbers In 2006 148 countries visited 1402 projects monitored by ROM with a total value of approximately €9.9 billion were visited. Since inception, in 2000 6721 monitoring reports 4247 projects in 172 Countries have been recorded. 12/11/2018

Grading The basis for the assessment is the programming document of the project => Is the project, with all due flexibility pursuing its planned course and will it attain its objectives? Is it on track? 4 Grades: 4 points (A): Much better than planned , very good 3 points (B): slightly better than planned , good 2 points (C): slightly below plan, minor problems 1 point (D): serious problems Methodological note: The baseline parameter for the scoring scale, 2.5 points, has been defined as “implementation according to plan”. Thus, a project with a score of 2.5 is fully on track and expected to deliver the benefits and impact planned. 12/11/2018

Performance Timeline 2002 – 2006 12/11/2018

Improved Projects (all regions) Rom helps also to improve the quality of project performance See here the ratings for all those projects which were monitored, both, in 2004 and in 2005. 413 projects were monitored in year 2005 and 2004, or 33,3 % of monitored population in 2005 (1244 projects) project performance improved clearly for the projects that were monitored on all criteria. Some projects were monitored by the same person, some projects were monitored by another contractor altogether (change of contractors between 2004 and 2005 for Tacis and ACP). Analysis made showes that there is no correlation between the the scores given and the fact that the monitor is the same person or not. * Average ratings for main criteria for the re-monitored projects in 2006, which had been monitored in 2004 and 2005 12/11/2018

Table: Results per Sector Social infrastructure and services (education, health, water, government and civil society etc.) - performing well and in line with the global picture. Another important sector, economic infrastructure and services with transport, energy and banking, is performing even better with 2.81. 12/11/2018

Scope: The aim is to cover all projects above 1 million EC contribution and a 10% sample of smaller ones For the time being only ongoing projects are monitored But Ex-post monitoring is being tested to get information on impact and sustainability as realized Sector Policy Support Programme methodology under preparation – Pool Finding + Sector Budget Support 12/11/2018

Challenge: best possible access to credible and useful information Project Evaluations Challenge: best possible access to credible and useful information Strategy: global Evaluations + Project performance information (ROM) + in depth project evaluations only where needed The challenge is to provide all stakeholders with the best possible access to credible and useful information. The joint Evaluation unit has developed adapted evaluation methodology on the level of sectors, themes, country, regions and for programming. Concerning projects, an instrument (ROM) has been set up to provide real-time data on performance. In view of the two instruments above, project evaluation has to be rationalised and focussed on those studies which are needed and provide real value added. The project evaluation reports must be made more accessible and a database should be created. . 12/11/2018

Making project evaluations more useful Number of project evaluation limited to relevant ones (value - added to the EC) Focus on quality for devolved environment (delegations) Common methodology and harmonized terms of reference –> comparability Compile reports and draw lessons across the board Project evaluations are important for those who order them, we want : Not to overload the colleagues with unnecessary obligations Give them a hand with clear and simple methodology to achieve quality Use the evaluation reports more largely (beyond project stakeholders), make them better known and draw common lessons. 12/11/2018