Evaluation Working Group Ex-ante Evaluation July 20, 2006 MINISTERUL FINANŢELOR PUBLICE Autoritatea de Management pentru Cadrul de Sprijin Comunitar Unitatea Centrală de Evaluare Evaluation Working Group Ex-ante Evaluation July 20, 2006 Niall McCann, TA project on Programming, Monitoring and Evaluation

Presentation Structure What is it? What should it include? What the Commission says about ex-ante evaluation. Why do ex-ante evaluation? DG Regio’s October 2005 “Draft Working Paper on Ex Ante Evaluation” The ex-ante evaluation contract – what does it say? Timetable on ex-ante evaluation Horizontal objectives (Strategic Environmental Assessment, equal opportunities)

Ex Ante Evaluation – what is it? An independent assessment of whether the proposed programme is the most appropriate and cost-effective way to achieve the identified objectives Whether the identified objectives are the most appropriate to meet the identified needs It involves a critical judgement, the ultimate aim of which is to optimise the allocation of resources and to improve the quality of programming, before it is finalised

The European Commission’s 2000 ‘Communication on Evaluation’ (SEC(2000)1051) Good quality ex-ante evaluation (of a programme) is necessary because: It allows a proper appreciation of whether the proposed level of funding and resources are in accordance with the expected results and impacts, and Reliable ex-post evaluation, and hence accountability for results and impacts, is largely dependent on the quality of the preparation of the intervention at the outset.

EXERCISE – Ex-ante evaluation – what should it include? Each group to list 5 topics that the ex-ante evaluation of the 2007-2013 NSRF Operational Programmes should cover DG Budget’s “Ex-Ante Evaluation – A Practical Guide…” names 7 topics that should be covered in an ex-ante evaluation! (e.g. whether there are more appropriate alternative delivery mechanisms that were not considered)

DG Budget’s “Ex-Ante Evaluation – A Practical Guide for Preparing Proposals for Expenditure Programmes” (Dec 2001) Purpose of ex-ante evaluation is to: Gather information; Carry out analyses that help to define objectives; Ensure that these objectives can be met; Ensure that the instruments used are cost-effective; Ensure that later evaluation will be possible.

The “Communication on Evaluation” stated that the ex-ante evaluation should include information on: The needs and the external environment General and specific objectives, associated results and measures of achievement Assessment of option for intervention including appropriateness and quality of delivery mechanisms Value added by Community intervention Expected levels of results and the associated costs Lessons learned from any past interventions, potential future risks and the ways to reduce these The monitoring and evaluation plan during the course of the intervention.

The Practical Guide for Preparing Proposals for Expenditure Programme says that the standard content of an ex-ante evaluation is… Analysis of how the programmers: Identified the problems and the needs of the economy Set the objectives for the programme Identified alternative delivery mechanisms and assessed the risk Calculated the added value of Community involvement Dealt with lessons from the past planned future monitoring and evaluation Helping to achieve cost-effectiveness

Draft Proposal for a Council Regulation laying down general provisions on the ERDF, the ESF and the Cohesion Fund states that… “the ex-ante evaluation [Article 46(2)] shall identify and appraise: medium and long term needs the goals to be achieved the results expected the quantified targets the coherence, if necessary, of the strategy proposed for the region the Community value-added the extent to which the Community’s priorities have been taken into account, the lessons drawn from previous programming, and the quality of the procedures for implementation, monitoring, evaluation and financial management.”

The New Programming Period, 2007-2013: Methodological Working Papers, Draft Working Paper on Ex Ante Evaluation… …states that of the 5 standard evaluation criteria (Relevance, Effectiveness, Efficiency, Impact, Sustainability), the main concerns at ex-ante stage are: Relevance – of the strategy to the needs identified Effectiveness – whether the objectives of the programme are likely to be achieved Impact – on the wider social, environmental and economic needs. But also important are: Internal and external coherence – financial allocations and linkages to national and Community priorities Quality of implementation systems – and how they may affect achievement of programme objectives. Potential risks – of policy choices made and implementation systems proposed.

Specific evaluation questions – pg. 5 of Working Paper Is the strategy appropriate to meet needs? Are the objectives clear and achievable? Is the strategy coherent with national and Community priorities? Are the indicators appropriate? What will be the impact of the strategy? Are the implementation systems appropriate? What is the Community added value?

Content of the ex-ante evaluation reports Socio-economic analysis and relevance of strategy to needs identified Methods – benchmarking, data analysis, macro-economic modelling, stakeholder consultation, focus groups, surveys, SWOT analysis, issue mapping, concept mapping Rationale of the strategy and its consistency Method – examine strategy against 4 rationale for public intervention – public good, corrective subsidies, targeted schemes, redistribution Coherence of the strategy with national policies/CSG (Annex 2 for ESF/HRD OP) 3 Community Strategic Guidelines Making Europe and its regions more attractive places to invest and work Improving knowledge and innovation for growth More and better jobs.

Content of the ex-ante evaluation reports (continued) 4. Expected results and impacts (particularly employment) Input/output analysis, econometrics, benchmarking, etc. 5. Proposed implementation systems Previous experience, risk appraisal

Contract Terms of Reference Working Paper suggests average 7 months for ex-ante evaluation. Timetable for execution of Romanian contract is as follows…

Timetable of execution of the contract… Activity Date Full evaluation team to arrive 1 Aug Project Steering Committee meeting to discuss the Inception Report 23 Aug Presentation of OP-specific interim reports 29 Sept Informal interim meetings at OP level to discuss OP-specific interim reports 2-6 Oct Informal meetings at OP level to present emerging conclusions 30 Oct-3 Nov Presentation of draft OP ex-ante evaluation reports by evaluators 15 Nov Meetings of PSC at OP level to discuss draft reports Receipt of comments on draft OP ex-ante evaluation reports from key stakeholders 24 Nov-1 Dec Final draft evaluation report 8 Dec

Timetable of execution of the contract… Activity Date Quality controlling of final OP evaluation reports 15 Dec Evaluators present draft PC evaluation reports 22 Dec Meetings of PSC at OP level to discuss draft PC reports 8-12 Jan Receipt of comments on draft PC ex-ante evaluation reports from key stakeholders 15-19 Jan Receipt of final PC ex-ante evaluation reports 26 Jan Quality controlling of final PC evaluation reports 2 Feb Training seminars with relevant stakeholders on results of ex-ante evaluation process 5-23 Feb Submission of 3 ad hoc analyses draft reports 16 Feb Submission of final ad hoc analyses reports 28 Feb

Role of Project Steering Committee The members of the PSC will be the MACSF (including ECU), ECD, plus 2 members of each OP The role of the PSC is to oversee all aspects of the implementation of the contract, including the primary task of determining whether the final ex-ante evaluation report meets the Terms of Reference. The report for each OP will undergo two reviews 1st review – draft report will be circulated to all stakeholders 2nd review – after comments from stakeholders are incorporated, final report is quality controlled by the ECU. Quality control criteria are on pg. 20 of Working Paper

Contract Terms of Reference The evaluator is required to produce an evaluation report for each OP and each PC – so 14 evaluation reports in total. And as per the Working Paper, each OP evaluation report should include… Appraisal of socio-economic analysis Relevance of strategy Rationale of strategy Coherence with national policies and CSG Quantification of results and impacts Proposed implementation system Strategic Environmental Assessment (5 OPs)

Each PC evaluation report should include… Analysis of previous evaluation results Assessment of consistency of operational objectives in PC with global objectives of OP Assessment of quantification Analysis of expected impacts Assessment of quality of implementation and monitoring arrangements

Contract Terms of Reference The ex-ante evaluation is forwarded to the Commission with the programme Romania is including the ex-ante findings in the programme. The Territorial Cooperation Objective OPs are being ex-ante evaluated under a separate contract!

Consultant details The contract was won by a consortium headed by Panteia B.V. (NL), and also including: NEA Transport research and training (NL) Regional Environmental Centre (HU/RO) Provice Professional Services (HU) PRAC Policy Research and Consultancy (DE)

And the relevant Key Experts for each OP are… Operational Programme Ex Ante Evaluator Team Leader/Building Administrative Capacity OP Malcolm Ross (IRL) Increasing Economic Competitiveness OP Vincent Rouwmaat (NL) Transport OP Adrian V. Vasile Vilcan (RO) Environment OP Ausra Jurkeyiciute (LT) Human Resources Development OP Douwe Grijpstra (NL) Regional OP Pieter van Run (NL) Technical Assistance OP Tamas Lunk (HU)

Process issues/tips for managing evaluators… Remember… Ask them to explain their methodology – exactly how they will evaluate your OP Use them as a resource! – ex-ante is a great opportunity to learn about what the OP can realistically achieve. It is difficult to alter the OP after it is agreed! Be prepared to be criticised! – it’s their job to criticise the OP constructively

Process issues/tips for managing evaluators… Some tension may be inevitable! The MA wants the OP agreed The evaluator wants to preserve professional independence Both sides have responsibilities The Managing Authority should Respect evaluator’s independence Be prepared to accept criticism where valid Engage with the evaluator Evaluator should Be critical but constructive Be careful with choice of words Engage with the policy maker

Ex-ante evaluation of horizontal principles ToR states that they must address the horizontal principles of: Environment (this will be tackled through the SEA) Equal opportunities Information society In their proposal they deal with only Environment and Equal Opportunities!

Strategic Environmental Assessment ...is required for five OPs (not required for HRD OP and TA OP) Not to be confused with an Environmental Impact Assessment (EIA)! SEA is done at the level of policies and programmes, while EIA is done at the level of individual projects. Steps in an SEA include: Assessment of the baseline environmental situation Identify environmental objectives, targets and priorities (which Romania should achieve through the programme) Draft development proposal and identification of alternatives (ensure that environmental objectives and priorities are fully integrated into the draft plan/programme) Environmental assessment of the draft proposal (assessment of the environmental implications of the programme and the conformity with relevant national and EU environmental policies and legislation. Identify environmental and sustainability indicators Integration the SEA findings into the final programme.

Equal opportunities NSRF – “Equal opportunities (men and women) will be promoted by interventions under all the OPs…Equality of opportunities will also focus on vulnerable groups, young people, ethnic minorities, especially Roma, the disabled people and people with learning difficulties.” How to ex-ante evaluate whether equal opportunities has been factored into the design of the Operational Programme?

Answer… the management team, who is making the decisions? Is the definition clear? Review programme documentation – is it built into objectives? Has the programme document set out the expected impact the programme/priority will have on equal opportunities? Review relevance Is it factored into programme design and delivery: the management team, who is making the decisions? project application forms project selection procedures Are there indicators measuring progress? How is it reported on? Is target group represented on relevant committees?