agriregionieuropa Exploring the perspectives of a mixed case study approach for the evaluation of the EU Rural Development Policy Ida Terluin and Petra Berkhout Agricultural Economics Research Institute LEI 122 nd European Association of Agricultural Economists Seminar Evidence-Based Agricultural and Rural Policy Making Methodological and Empirical Challenges of Policy Evaluation February 17 th – 18 th, 2011, Ancona (Italy) associazioneAlessandroBartola studi e ricerche di economia e di politica agraria Centro Studi Sulle Politiche Economiche, Rurali e Ambientali Università Politecnica delle Marche

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Common Monitoring and Evaluation Framework (CMEF)  CMEF is designed by the EU Commission for the evaluation of the EU Rural Development Policy  Comprehensive approach: about 160 indicators and 140 common evaluation questions

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Concerns on the CMEF  large number of indicators is experienced as a cumbersome requirement of Brussels; data are not always available  simplification of the CAP: wish to have a simpler and more popular kind of evaluation framework  evaluation questions are not always relevant in Member States/regions  emphasis on what has happened and not how or why it happened

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Aim of our paper  To explore whether there is an alternative approach to the CMEF for the evaluation of the EU Rural Development Policy and to asssess the perspectives of this alternative for EU wide use

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Plan of this presentation 1.Comparative analysis of evaluation methods of rural development policy 2.Identification of an alternative approach for the CMEF 3.Testing of this alternative approach in the midterm evaluation of the RDP in The Netherlands 4.Assessment of the perspectives of this alternative approach for EU wide use

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Comparative analysis of evaluation methods of rural development policy  Literature study: 22 recent evaluation methods  Classification of these methods according to applied methodology into five goups: 1.the CMEF type approach: hierarchy of indicators combined with evaluation questions 2.the tally approach: counting whether a quantified objective has been achieved 3.the econometric approach: use of econometric methods 4.the modelling approach: use of models 5.the mixed case study approach: use of quantitative and qualitative analyses, often in case studies

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Main properties of evaluation methods Approach CMEF type tallyecono- metric model -ling mixed case study What happened in quantitative terms xxxx What happened in qualitative termsx How and why did it happenx Unintended effects of policyx Why actors participate in policyx Easy to apply for evaluatorxx

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Identification of an alternative approach to the CMEF  Starting point for identification: different properties of the groups of evaluation methods  Given differences of CMEF and mixed case study approach, this last approach might be a promising alternative for the CMEF

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) The mixed case study approach  analysis consists of different stages, in which different methods are applied  by combining evidence from these stages, it is tried to find exploring patterns  approach has been applied for the ex post evaluation of the EU Rural Development Policy in Wales by Midmore et al. (2008): 1. analysis of secondary data of the case study region 2.in-depth interviews with representatives of different interest groups

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Design of a protocol with instructions for the various steps in the evaluation 1.Analysis of the baseline situation in the case study region 2.Analysis of the objectives of the EU Rural Development Programme (RDP) in the case study region 3.Analysis of the financial input and the output for each rural development measure 4.Questions for in-depth interviews with representatives of the different interest groups 5.A template for writing a report

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Testing alternative in midterm evaluation RDP in 2 Dutch provinces:(1) Zeeland and (2) Gelderland

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Experiences in case study in Zeeland (1)  Baseline situation in the case study region: easy to describe, as data and literature were readily available  Objectives of the RDP: difficult to assess, due to complex way the RDP is implemented in The Netherlands –there is one RDP for the whole country –national government is responsible for axes 1 and 2 –province is responsible for axes 3 and 4 and has integrated them with other rural policies

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Experiences in case study in Zeeland (2)  Data on financial input and output indicators for each rural development measure are collected by two institutes, which have different data systems  Data for input and output indicators for measures under axes 1 and 2 are only available at national level, and it is very time-consuming to extract regional data  Data on result and impact indicators in Zeeland are not collected

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Experiences in case study in Zeeland (3) P rogress in the absorption of the EAFRD budget in Zeeland: Group 3 focuses on tourism activities, basic services, village renewal and rural heritage, measures which fit rather well into the rural development needs in Zeeland 1Axes 1 and 2no information 2Measures 311, 312, 41moderate 3Measures 313, 321, 322 and 323 substantial

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Experiences in case study in Zeeland (4) According to the six interviewed persons:  the landscape and quality of life have been improved in qualitative terms by the RDP  these projects fit in the long tourist tradition in Zeeland as well as the need to maintain or improve the supply of basic services in rural areas  economic diversification has been boosted  they had no idea of the impact of the measures of axis 1 on the development of the agricultural sector

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Experiences in case study in Zeeland (5)  actors participate in the RDP measures as this generates funds for their projects  most of the projects would also have been conducted without EAFRD financing, but often at a smaller scale  the requirement of 50% national cofinancing for each project is not always feasible  interviewees appreciated the easy communication and the openness of the province of Zeeland, which stimulates the implementation of the measures of axes 3 and 4

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Concluding remarks (1) 1.Based on a comparative analysis of 22 evaluation methods of rural development policy, we have identified the mixed case study approach als an alternative to the CMEF 2.The results of testign this alternative in the province of Zeeland are promising: the analysis of inputs and outputs of the measures in combination with interviews provided sufficient information to get a good overview of the performance of the RDP

agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Concluding remarks (2) 3.The interviews yielded useful insights into processes within the region 4. When experiences with the mixed case study approach in other regions are also satisfying, then it could be considered to use it EU wide as an alternative to the CMEF