4/5 June 2009Challenges of the CMEF & Ongoing Evaluation Common monitoring and evaluation framework for evaluation of rural development programs Jela Tvrdonova, 2012

Outline of the presentation  Strategic approach to RD  Common approach to the evaluation - the CMEF  Monitoring and evaluation –explanatory notes  Intervention logics – the role in the evaluation  Indicators  CMEF structure and content 4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 2

Strategic approach to rural development Focus on limited number of objectives Competitiveness – Environment – Quality of life Axes connected with strategic objectives Strong and dynamic agri-food sector Agriculture and forestry with high added value Employment and growth in rural areas Improvement of the governance in rural areas and mobilisation of the endogenous potential

Common approach to monitoring and evaluation Based on: Exact definition of objectives in Regulation, Strategic Guidelines for RD and rural development programs EU Strategic monitoring a national strategies Definition of baseline indicators at the program start period Suitable combination of output, result and impact indicators, Common evaluation questions for all RD programs (94) Single framework for all program intervetion

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 5 Content of the CMEF  CMEF – Common monitoring and evaluation framework – Handbook, Annexes, Guidance notes  Common and additional/program specific indicators  Common and additional/program specific evaluation questions  Intervetion logic of rural development programs and individual measures

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 6 CMEF – legal base  EC Reg. 1698/2005 and 1974/2006 and its Annex VIII

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 7 Monitoring and evaluation  Monitoring – measurement of inmediate outcomes and results at measure and axis level  Evaluation - measurement of long-term and rural areas effects/impacts of intervention, within the program development context  Comparissons and lessons learned for next interventions

Monitoring  On-going process which monitors the gradual implementation of the program at the level of financial inpiuts, physical outputs and axes results  It subject of annual reporting on the program implementation  Instruments – common and additional indicators of output and results,

Strategic monitoring  Since 2010 and every other two years  Focus of the EC on the progress of national strategic plans and its objectives implementation  Assesment of the contribution of national plans to the EU Rural Development Strategy

Evaluation On-going evaluation is based on CMEF and country´s own methodology to carry on the evaluation during the program implementation Process based on annual reporting of result/impact indicators including periodical exercises:  Ex – ante: evaluation of relevance of the planned intervention and optimizing of the RDP budget  Mid-term and ex-post: for the assessing the progress related to the specific and overall/program objectives,

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 11 Monitoring and evaluation in program logic  Good program is basic ground for the high quality of monitoring and evaluation  The evaluation and the monitoring strats with the program intervention logic

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 12 Intervention logic of RDP Needs Change observed in reality during and/or after intervention Envisioned change before intervention Impact Result Output Inputs of the RDP Measures Overall/horizontal objectives of RDP Specific/Axis objectives of the RDP Operational/measure objectives of the RDP Change Overall baseline socio-economic and environmental situation of rural areas, at baseline period Development context - all intervening factors

Hierarchy o objectives and indicators  5 types of indicators coresponding with the hierarchy of objectives  Hierarchy of objectives: - instrument helping to show how local activities contribute to the overall objectives,  It is composed of: - operational objectives (measure level) - specific objectives (Axes/priority areas level) - overall – cross cutting/strategic/horizontal objectives (at the program level)  Indicators copy objectives in the hieararchy

Indicators in intervention logic of RDP Hierarchy of indicators Hierarchy of objectives Impact ndicator Result indicator Output indicato Input indicators Measures Overall/horizontal objectives of RDP Specific/Axis objectives of the RDP Operational/measure objectives of the RDP Context related baseline indicators Objectives related baseline indicators

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 15 Role of indicators  Indicators – used as tools to assess how far the expected objectives have been achieved by measures or whole programmes –should be specific, measurable, available / achievable in a cost effective way, relevant for the programme, and available in a timely manner (SMART) –Indicators can not always be filled with quantitative statistical data; in some cases, indicators might also include qualitative assessments or logical assumptions

The quality of indicators - SMART approach S – simple, specific M – measurable - data exist A – achievable – available – at low cost R – relevant – selected well for given program T - timely oriented

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 17 Types of indicators  Baseline indicators: they relate to general socio- economic context of the programme area (context- related baseline indicators) and to the state of the economic, social or environmental situation in direct relation with the wider objectives of the programme (objectives-related baseline indicators)  Financial execution (input) indicators: they refer to the budget or other resources allocated to the programmes  Output indicators: measure activities directly realized within programmes

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 18 Types of indicators  Result indicators: measure the direct and immediate effects of the intervention and provide information on changes that have taken place  Impact indicators: refer to the benefits of the programme both at the level of the intervention but also more generally in the programme area. They are linked to the wider objectives of the programme

The process Inputs Activities Result Ouputs Impact Monitoring and Evaluation process

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 20 Common indicators  A common set of baseline, output, result, and impact indicators for the RDPs (Art. 62 Reg. 1974/2006) “shall form the Common Monitoring and Evaluation Framework (CMEF)’’ (Annex VIII lists the common indicators)

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 21 Additional indicators  Since common indicators may not fully capture all effects of programme activities, it is necessary to define additional indicators within the programmes (see Guidance notes A & K)

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 22 Why additional indicators?  The CMEF provides the Common Framework to be applied throughout the EU, thus ensuring comparability  The specificities of each single programme cannot be fully reflected by this Common Framework  It is therefore important to complement the Common Framework by additional indicators in order to capture the full range of intended effects of a given programme  As general rule, a thorough analysis of the programme intervention logic can drive the choice of relevant additional indicators  Considering the limited number of common impact indicators and their broad scope, additional indicators are essential to overcome attribution gaps

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 23 CMEF structure Handbook Annex 1:  Guidance notes - A: Choice and use of indicators, B: On-going evaluation, C: Mid-term evaluation Annex 2:  Guidance notes – D: Hierarchy of objectives, E – Measure Fiches Annex 3:  Guidance notes – F – K, Common indicators Fiches Annex 4  Guidance J – O Other guidances

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 24 CMEF guidance on indicators CMEF provides guidance for MAs in setting up indicators at each level of intervention and baseline indicators Annex 3 of the CMEF Handbook provides detailed description of all common indicators (Guidance notes F – K): –F: COMMON INDICATOR LIST (overview of all common indicators) –G. BASELINE INDICATOR FICHES (detailed description of indicators) –H. OUTPUT INDICATOR FICHES –I. RESULT INDICATOR FICHES –J. IMPACT INDICATOR FICHES

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 25 Indicator Fiches Each indicator fiche contains the following elements:  Type of indicator  Related measures  Measure Codes  Definition of the indicator  Subdivision  Unit of measurement  Level of collection  Responsible actor for collection  Collection method/good practice  Sources  Registration frequency

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 26 Measure Fiches Guidance on practical use of indicators in monitoring and evaluation of RDPs  Guidance note E provides the following on use of indicators within each particular measure: –Measure Code –Rationale of the measure –Target group –Target area –Common indicators –Link rationale of the measure and indicators –Evaluation questions

4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 27 Setting up the evaluation framework at the stage of Ex- ante and use of indicators Evaluation QuestionIndicators Evaluation questions – cross- cutting Evaluation questions Axis/sector specific Output indicators Impact indicators Result indicatorsAxis based baselines, objective related, contextual Baselines and their quantification, Target levels Horizontal baselines, objective related, contextual RDP objectives Overall strategic objectives Axis specific objectives Measure/op erational objectives Baseline values of output indicators Targets for impact indicators Targets for results indicatiors Targets for output indicators Common and program spec. Eval. questions

Thank you for your attention! The Helpdesk of the European Evaluation Network for Rural Development 260, Chaussée Saint – Pierre - Etterbeek Brussels – Belgium Tel: http//enrd.ec.europa.eu/evaluation/