Evaluation Plan Workshop for Cross-border Programmes February 2016 | Thessaloniki, Greece Interact is co-financed by the European Regional Development Fund (ERDF)

Agenda 09:30 – 10:00 Introduction and objectives of the workshop 10:00 – 11:00Update on the state of play of the EP Presentation by David Alba, Evaluation Unit, DG Regio 11:00 – 11:20Coffee Break 11:20 – 13:00First and second part of an EP working groups Overview of the approved EP 13:00 – 14:00Lunch 14:00 – 16:00 Practical example: EP DE-NL Presentation and Q&A by Julia Wengert, DE-NL Communication Strategy for evaluation findings Presentation and Q&A by Interact Presentation and Q&A by Linda Talve, Central Baltic 2 Introduction

Agenda 09:15 – 11:00 Third part of an EP Introduction to impact evaluation, Presentation by David Alba, Evaluation Unit, DG Regio Practical example of the EP RO-BG Presentation and Q&A by Anca Simion, RO-BG 11: :15 Coffee Break 11:15 – 12:30 Continuation of discussion of third part of the EP 12: :45 Wrap up and overview of upcoming Interact activities 12:45-13:45Farewell Lunch 3 Introduction

An introduction to the plan setting out its main objectives Coverage of the evaluation plan Analysis of relevant evidence available Mechanisms of coordination within a MS and exchange between MAs First part of the EP Evaluation Plan 4

Evaluation functions with clearly defined responsibility Description of the evaluation process The involvement of partners in evaluation The source for evaluation expertise (internal - external) Possibly a training programme An overall timetable showing how the evaluations will feed into implementation and the programmes reports The overall budget for implementation of the EP A quality management strategy for the evaluation process A strategy to ensure use & communication of evaluations Second part of the EP Evaluation Plan 5

Process and responsiblities and quality management Communication of evaluation findings Link evaluation – reporting (timetable and budget) Group discussions Evaluation Plan 6

13 CBC programmes adopted in 2014: ABH, AT-DE Bavaria, BE-NL, Central Baltic, DE (Bavaria)- CZ, DE-DK, DE-NL, FR-CH, ÖKS, SE-NO, SE-FI-NO (Botnia- Atlantia), SE-FI-NO Nord, Upper Rhine (FR-DE-CH) All these programmes submitted their EP to the MC but not all these EPs adopted RO-BG* (adopted in , but EP approved) Overview of approved evaluation plan (EP) Evaluation Plan 7

adequate planning of evaluations, including as regards data availability and drafting of detailed ToRs, setting out clear award criteria and quality requirements; wide advertising of tender; appointing a selection committee responsible for evaluating the bids against the criteria set out in the ToR; including in the evaluation contract a procedure for the early termination of the contract conditional on the quality of the work provided; organising a kick-off meeting with the contractor to clarify all aspects of the Terms of Reference and Technical Offer; requesting an inception report and monthly progress reports; involving the Evaluation Steering Committee in endorsing draft reports and approving final reports; using a thorough quality assessment grid for assessing the quality of final deliverables. Quality Management Strategy (EP Romania-Bulgaria) Analysis of the approved EP 8

- Evaluation outcomes give insight about the programme implementation and if necessary indication for improvements (programme implementation and intervention logic) - Evidence and justification of the programme result and impact (added value of cooperation, accountability for invested funds, etc.) - Evaluation outcomes will generate an opportunity for learning -> good example: EP NPA (p.20) What will be done with the evaluation outcomes? Analysis of the approved EP 9

Link reporting-evaluation (EP Upper Rhine, p.6) Analysis of the approved EP 10

Theme, scope, subject and rationale, including the background, the coverage, the main approach (process or impact evaluation) and the main guiding evaluation questions. Methods to be used and their data requirements Duration and a tentative date Third part of the EP Evaluation Plan 11

- Impact evaluations assess how the ERDF funding contributed to the objectives of each priority of the programme. - The impact evaluations should outline what change the programme achieved through its interventions. Therefore the programme should start with reflecting on the intervention logic. - As the evaluation approach might differ per priority, there will be one or more impact evaluations per Interreg programme Impact evaluations Evaluation Plan 12

focus on the set of questions which are devoted to quantifying “whether a given intervention produces the desired effects on some pre-established dimension of interest. The overarching goal is to answer a “does it make a difference” question by identifying and estimating casual effects through counterfactual methods.” The core element of a counterfactual impact evaluation is to compare two groups/areas to see what has been the change in the group/area with the intervention and in the group/area without the intervention. Counterfactual based impact evaluations Evaluation Plan 13

are based on establishing the theory behind an intervention (the theory of change) and assessing whether it has been implemented according to that theory in order to judge the contribution of the intervention to the observed effects. The theory based impact evaluations deals with ‘why it works’, ‘did things work as expected to produce the desired change’. Theory based impact evaluations Evaluation Plan 14

“The Theory of Change is a programme theory approach concerned with going beyond input output descriptions and seeking to understand the theories of actors with regard to programme interventions and why they should work”. (EVALSED: The resource for the evaluation of Socio-Economic Development, Sept. 2013) Theory of change Evaluation Plan 15

As a planning tool Theory of Change helps to come up with the logic of the programme (to reconstruct the chain that links objectives of the programme, the interventions funded, the outputs achieved and the contribution of the interventions to these results). Begin with a situation analysis (current situation) Needs analsyis (intended situation) Develop a theory how to get from the current situation to the intended situation Clarify which aspect of the problem the intervention will address: define activities, outcomes, result and impact Theory of Change as used for planning and evaluation Evaluation Plan 16

When preparing the evaluation plan the programme should start with the reconstruction of the theory of change. This will be useful for identifying the right questions, data that needs to be collected. During the evaluation the progromme assesses the ability of the programme to contribute to its intended outcomes. Theory of Change as used for planning and evaluation Evaluation Plan 17

Evaluation Plan 18

Tips: The issues of greatest concern should be addressed by specific evaluation questions Start with reflecting on the intervention logic and theory of change Be specific The questions should be answerable How to formulate good impact evaluation(s) questions? Evaluation Plan 19

1. Reflect on the selected part of the Intervention Logic: Which change would the programme like to achieve? What are possible gaps and concerns? 2. Identify what should be answered in the evaluation 3. What are the challenges when defining the evaluation questions? 4. What method do you consider theory based or counterfactual evaluation? 5. Which data is needed and available? 6. What are the challenges related to method and data? Exercise: What are our evaluation questions? Evaluation Plan 20

- What will be the objective of the impact evaluation(s) and how deeply will the impact be evaluated? - Which evaluation questions should the impact evaluations answer? - Which method(s) is/are the most suitable, considering the objective of the evaluation and data availability? - Which data needs to be available to conduct the impact evaluation according to its aims? Is this data available (already collected) or if not, how to get this data? - When should the impact evaluations take place? - Which budget and resources are realistically needed for the impact evaluations? Are they available? What information should be included on impact evaluation in the evaluation plan Evaluation Plan 21

Evaluation Activities Basecamp group ‘Results and Evaluation‘ share examples of EP Q&A Evaluation E-learning tool Upcoming events for 2016 Impact evaluations, June, Data collection, venue and date tbc Upcoming evaluation activities and events 22

Interact Programme Interact is co-financed by the European Regional Development Fund (ERDF) Thank you for your attention Please do not hesitate to contact us for any further information or visit