Common monitoring and evaluation framework for evaluation of rural development program Jela Tvrdonova, 2016

Outline of the presentation Strategic approach to RD Common approach to the evaluation - the CMEF Monitoring and evaluation – explanatory notes Evaluation framework Intervention logic Evaluation questions Indicators CMEF structure and content Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Strategic approach to rural development Focus on limited number of objectives Competitiveness – Environment – Quality of life Axes connected with strategic objectives Strong and dynamic agri-food sector Agriculture and forestry with high added value Employment and growth in rural areas Improvement of the governance in rural areas and mobilisation of the endogenous potential

Why do we need then common approach to M&E? Increase the accountability and transparency of interventions financed of EU funds, Demonstrate achievements towards EU national and local objectives, Assess effectiveness, efficiency, results and impacts, and Learn how to design and implement policy better Ensure the comparability of evaluation findings among EU Member states

Evaluation in policy cycle Programme design Ex ante evaluation Mid-term Evaluation Ex post evaluation Policy priorities Will it work? Did it work? Is it working?

What is the legal framework? Community strategic guidelines for rural development Council Regulation (EC) 1698/2005 Commission Regulation 1974/2006 and its Annex VIII CMEF as guidance to implement the legal framework Para 66 of 1698/2005 (revised): “The effectiveness and the impact of actions under the EAFRD also depend on improved evaluation on the basis of the common monitoring and evaluation framework. In particular, the programmes should be evaluated for their preparation, implementation and completion.”

Common approach to monitoring and evaluation Based on: Exact definition of objectives in Regulation, Strategic Guidelines for RD and rural development programs EU Strategic monitoring a national strategies Definition of baseline indicators at the program start period Suitable combination of output, result and impact indicators, Common evaluation questions for all RD programs (94) Single framework for all program intervetion

Content of the CMEF CMEF – Common monitoring and evaluation framework – Handbook, Annexes, Guidance notes Common and additional/program specific indicators Common and additional/program specific evaluation questions Intervetion logic of rural development programs and individual measures Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Monitoring and evaluation Monitoring – measurement of inmediate outcomes and results at measure and axis level Evaluation - measurement of long-term and rural areas effects/impacts of intervention, within the program development context Comparissons and lessons learned for next interventions Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Monitoring On-going process which monitors the gradual implementation of the program at the level of financial inpiuts, physical outputs and axes results It subject of annual reporting on the program implementation Instruments – common and additional indicators of output and results,

Strategic monitoring Since 2010 and every other two years Focus of the EC on the progress of national strategic plans and its objectives implementation Assesment of the contribution of national plans to the EU Rural Development Strategy

Evaluation On-going evaluation is based on CMEF and country´s own methodology to carry on the evaluation during the program implementation Process based on annual reporting of result/impact indicators including periodical exercises: Ex – ante: evaluation of relevance of the planned intervention and optimizing of the RDP budget Mid-term and ex-post: for the assessing the progress related to the specific and overall/program objectives,

Monitoring and evaluation in program logic Good program is basic ground for the high quality of monitoring and evaluation The evaluation and the monitoring strats with the program intervention logic Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Intervention logic What do we mean by intervention logic? EU rural policy and RDP intervention logic and its evolution over the time (e.g. Health check), What are major challenges? Complexity of composition: overall, axis and operational objectives, measures, activities, expected outputs, results and impacts, Looking beyond the visible - various types of IL (economy, public goods/bads, quality of life), What shall be assessed? Relevance, coherence, unintended effects, efficiency, How shall the intervention logic be assessed? Two stages in assessment: desk scrutiny, facilitated workshops. Specificities with respect to TA and NRN

RDP intervention logic - hierarchy of objectives Overall RDP objectives – EU common and RDP specific Axis 1 objectives Axis 2 objectives Axis 3 objectives Axis 4 objectives Competitiveness Environment Quality of life and diversification Leader Measures Measures Measures Measures

RDP intervention logic - hierarchy of objectives Conditions for measures aimed at promoting knowledge and improving human potential Conditions for measures aimed at restructuring and developing physical potential and promoting innovation Conditions for measures aiming to improve the quality of agricultural production and products Conditions for transitional measures Conditions for measures targeting the sustainable use of agricultural land Conditions for measures targeting the sustainable use of forestry land Conditions governing the measures to diversify the rural economy Conditions governing the measures to improve the quality of life in rural areas Training, skills acquisition and animation To implement the Leader approach in mainstream rural development programming (49) Axis 1: Improving competitiveness of the agricultural sector Axis 2: : Improving the environment and the countryside Axis 3: The quality of life in rural areas and diversification of the rural economy Axis 4: Leader

Intervention logic of RDP EU policy objectives Intervention logic of RDP RDP Intervention logic Complementarity Context , its description SWOT and needs assessment Relevance Overall objectives EU/MS Programme level Impacts Coherence Specific objectives EU/MS Axis level Results Effectiveness Operational objectives EU/MS Measure level Outputs Measures, projects and their management and implementation Inputs Efficiency

Monitoring and evaluation system Focus of evaluation Evaluation questions Attribution of impacts Basis of evaluation Evaluation methods Intervention logic Measurement tools Data Indicators Collection of evidence

Evaluation questions Specificities with respect to TA and NRN The role of evaluation questions in ex post evaluation Common and programme specific evaluation questions Evaluation question, judgment criteria, links to indicators: triangular consistency Revised set of common evaluation questions Programme specific evaluation questions Role of programme specific evaluation questions Development of programme specific evaluation questions and judgment criteria, links to indicators How to use evaluation questions in ex post evaluation of RDP? In structuring, observing, analysing, judging, reporting Specificities with respect to TA and NRN New!

Revised set of common evaluation questions New! Revised set of common evaluation questions Original set of 150 common evaluation questions of CMEF has been simplified and reduced to the essential demand for knowledge from the European perspective. 3 groups of CEQs Programme-related: Community strategic priorities Health Check objectives 7 impact indicators TA and NRN Efficiency of RDP resources Measure-related: Contribution of the measure to its axis objectives Other measure effects and contribution to other axes objectives Leader approach-related: Community strategic priorities: Employment, diversification and governance Leader approach LAG contribution to LDS and RDP

Hierarchy of objectives and indicators 5 types of indicators coresponding with the hierarchy of objectives Hierarchy of objectives: - instrument helping to show how local activities contribute to the overall objectives, It is composed of: - operational objectives (measure level) - specific objectives (Axes/priority areas level) - overall – cross cutting/strategic/horizontal objectives (at the program level) Indicators copy objectives in the hieararchy

Role of indicators Indicators used as tools to assess how far the expected objectives have been achieved by measures or whole programmes should be specific, measurable, available / achievable in a cost effective way, relevant for the programme, and available in a timely manner (SMART) Indicators can not always be filled with quantitative statistical data; in some cases, indicators might also include qualitative assessments or logical assumptions Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Indicators The CMEF indicators in ex post evaluation Definitions, role and types of indicators (baseline, input, output, result and impact) Values for indicators Programme-specific indicators Existing indicators and development of new indicators How to use indicators in evaluation of RDPs Balancing importance and measurability Coherence with intervention logic, links to evaluation questions, methods and data, Interpretation of values, Specificities with respect to Leader Examples of additional indicators linked to revised CEQ Specificities with respect to TA and NRN New!

The quality of indicators - SMART approach S – simple, specific M – measurable - data exist A – achievable – available – at low cost R – relevant – selected well for given program T - timely oriented

Types of indicators Baseline indicators: they relate to general socio- economic context of the programme area (context- related baseline indicators) and to the state of the economic, social or environmental situation in direct relation with the wider objectives of the programme (objectives-related baseline indicators) Financial execution (input) indicators: they refer to the budget or other resources allocated to the programmes Output indicators: measure activities directly realized within programmes Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Types of indicators Result indicators: measure the direct and immediate effects of the intervention and provide information on changes that have taken place Impact indicators: refer to the benefits of the programme both at the level of the intervention but also more generally in the programme area. They are linked to the wider objectives of the programme Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

The process Monitoring and Evaluation process Impact Result Ouputs Inputs Activities Result Ouputs Impact Monitoring and Evaluation process

Common indicators A common set of baseline, output, result, and impact indicators for the RDPs (Art. 62 Reg. 1974/2006) “shall form the Common Monitoring and Evaluation Framework (CMEF)’’ (Annex VIII lists the common indicators) Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Additional indicators Since common indicators may not fully capture all effects of programme activities, it is necessary to define additional indicators within the programmes (see Guidance notes A & K) Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Why additional indicators? The CMEF provides the Common Framework to be applied throughout the EU, thus ensuring comparability The specificities of each single programme cannot be fully reflected by this Common Framework It is therefore important to complement the Common Framework by additional indicators in order to capture the full range of intended effects of a given programme As general rule, a thorough analysis of the programme intervention logic can drive the choice of relevant additional indicators Considering the limited number of common impact indicators and their broad scope, additional indicators are essential to overcome attribution gaps Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Methods Addressing methodological challenges New structure and more guidance Methods Methodological challenges in assessing programme effects Addressing methodological challenges Distinction between programme results and impacts Identification of main programme effects – direct and indirect (leverage, deadweight, multiplier, displacement, substitution effects) Identifying programme net effects Distinction of positive and negative externalities, expected and unexpected programme effects Identifying factors distorting programme effects (confounding factors, selection bias, self-selection bias, homogeneous and heterogeneous treatment effects, etc.) Using counterfactual in addressing methodological challenges Evaluation designs (randomised, quasi- experimental, non-experimental) Key approaches to evaluation (Theory based, quantitative, qualitative, mixed) – advantages, disadvantages, when to be used, various techniques to apply etc. How to select appropriate evaluation design (quality standards, considering limits and constrains, e.g. resources) Suggested methods in relation to the assessment of individual axes, TA, NRN and answering related evaluation questions

Data Major challenges in data management and collection during the evaluation of RDP Data needed for evaluation of RDP results at beneficiary level Data needed for evaluation of RDP results and impacts at RDP level Economic Environmental Social Specificities with respect TA and NRN

CMEF structure Handbook Annex 1: Guidance notes - A: Choice and use of indicators, B: On-going evaluation , C: Mid-term evaluation Annex 2: Guidance notes – D: Hierarchy of objectives, E – Measure Fiches Annex 3: Guidance notes – F – K, Common indicators Fiches Annex 4 Guidance J – O Other guidances Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

CMEF guidance on indicators CMEF provides guidance for MAs in setting up indicators at each level of intervention and baseline indicators Annex 3 of the CMEF Handbook provides detailed description of all common indicators (Guidance notes F – K): F: COMMON INDICATOR LIST (overview of all common indicators) G. BASELINE INDICATOR FICHES (detailed description of indicators) H. OUTPUT INDICATOR FICHES I. RESULT INDICATOR FICHES J. IMPACT INDICATOR FICHES Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Indicator Fiches Each indicator fiche contains the following elements: Type of indicator Related measures Measure Codes Definition of the indicator Subdivision Unit of measurement Level of collection Responsible actor for collection Collection method/good practice Sources Registration frequency Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Measure Fiches Guidance on practical use of indicators in monitoring and evaluation of RDPs Guidance note E provides the following on use of indicators within each particular measure: Measure Code Rationale of the measure Target group Target area Common indicators Link rationale of the measure and indicators Evaluation questions Challenges of the CMEF & Ongoing Evaluation 4/5 June 2009

Thank you for your attention! jelatvrdonova@gmail.com