1. Guidelines on the Mid-term Evaluation
John Grieve & Hannes Wimmer
Evaluation Helpdesk
23 June 2009
Evaluation Expert Committee Meeting

2. Evaluation Expert Committee Meeting
Content
Overview of the Mid-Term Evaluation (MTE)
Main steps of the MTE
Tools to support the MTE
23 June 2009
Evaluation Expert Committee Meeting

3. Overview: legal provisions
Article 86 of Council Regulation 1698/2005 specifies the management and the functions relating to the MTE as follows:
“In 2010, ongoing evaluation shall take the form of a separate mid-term evaluation report.
That mid-term evaluation shall propose measures to improve the quality of programmes and their implementation.
A summary of the mid-term evaluation reports shall be undertaken on the initiative of the Commission.”
drafting of Terms of Reference?
selection of evaluators?
organisation of tender procedures?
use of evaluation methodologies?
23 June 2009
Evaluation Expert Committee Meeting

4. Purpose of the guidelines
support Member States and the relevant national or regional authorities to frame the work of their independent evaluators
providing further guidance on key aspects of the MTE
23 June 2009
Evaluation Expert Committee Meeting

5. Overview: main characteristics (1/2)
is a specific part of ongoing evaluation activities (=combination of ongoing + MTE-specific activities)
has a critical role as formal mid-point review to inform on adjustments of the current programmes  importance of results and impact level!
23 June 2009
Evaluation Expert Committee Meeting

6. Overview: main characteristics (2/2)
draws on ongoing evaluation activities
intervention logic and target levels from ex-ante
data collection and monitoring
thematic studies, Community guidance documents, Synthesis ex-ante, previous programmes
contributes to the ex-post
 The MTE is embedded into the continuous process of ongoing evaluation, which provides the solid ground for it!
23 June 2009
Evaluation Expert Committee Meeting

7. Evaluation Expert Committee Meeting
Challenges of the MTE
RDPs at an early stage in implementation
availability of data
impacts emerge late
timing and organisation
The MTE should not be reduced to issues concerning data-availability: The QUESTIONS that MAs would like answered are crucial!
23 June 2009
Evaluation Expert Committee Meeting

8. Evaluation Expert Committee Meeting
Main steps of the MTE
PREPARATION
review of evaluation questions & related indicators
information gathering
setting up of the steering group 
establishment of the evaluation mandate 
establishment of the ToR
... recommended step!
IMPLEMENTATION
independent and interactive MTE process
management and feedback procedures, MA and evaluator
assessment of quality of the MTE report
DISSEMINATION
dissemination and use of results
- follow-up of MTE
23 June 2009
Evaluation Expert Committee Meeting

9. STEP 1: Review of evaluation questions and indicators
CMEF:
For setting-up the evaluation system, it is inevitable for the managing authority to review the common and programme-specific evaluation questions and the related indicators in order to assess what needs to be done in terms of information gathering and analysis in order to answer these questions in a meaningful and appropriate manner. […]
Guidance note B, chapter 5.1.3, p.9
23 June 2009
Evaluation Expert Committee Meeting

10. STEP 1: Review of evaluation questions and indicators
Key questions:
Do the CEQ and indicators capture the full range of results and impacts or is there a need for further fine-tuning?
Are they sufficiently consistent with the CMEF to allow a European synthesis?
Are the programme specific evaluation question and indicators well defined to address the RDP’s specificities?
Use the programme’s ex-ante evaluation findings regarding the intervention logic and measures.
Justify any variations from the CMEF.
Define further additional questions or indicators where you identify gaps!
23 June 2009
Evaluation Expert Committee Meeting

11. STEP 2: Information gathering
CMEF
MA ensures that evaluators have sufficient data on
general trends
outputs
results
in order to enable evaluators to assess the impacts
23 June 2009
Evaluation Expert Committee Meeting

12. STEP 2: Information gathering
Use evaluation questions and indicators as checklist!
Key questions:
- Does the MA provide evaluators with sufficient data on general trends, outputs, results to enable them to assess the impacts?
Has it been clarified what data is collected by the MA and what by evaluators?
Have the additional data-needs in light of the programmes’ specificities been considered?
Is all the relevant information from ongoing evaluation activities readily available?
Clarify where data is recorded and the accessibility/usability of information. .
Ensure support for evaluators who undertake interviews or surveys!
Clarify accessibility and usability of information.
Consider date of availability of studies, data from third parties etc.
23 June 2009
Evaluation Expert Committee Meeting

13. (STEP 3): Setting up of steering group 
CMEF:
In order to ensure a high quality of the evaluation, a regular consultation of stakeholders should be ensured.
The setting up of a steering group which accompanies the evaluation process and which involves representatives of different departments is advisable.
The steering group should contribute to the preparation of the terms of reference.  evaluation mandate!
The members of the steering group can provide access to additional information; they shall support and monitor the work of the evaluator.
Guidance note B, p. 9
23 June 2009
Evaluation Expert Committee Meeting

14. (STEP 3): Setting up of steering group 
Adds value to evaluation findings and their follow-up
Key questions:
Has a steering group been set-up for the purpose of the MTE?
What is the composition of the steering group?
Does the composition of the SG (i.e. the field of expertise of the members) properly reflect the focus of the programme ?
Have the roles and activities of the SG been clarified?
MA, paying agency, measure responsibles, evaluation unit, sub-regional authorities, LAGs…(N.B. no final beneficiaries)
Ensure that technical expertise reflects content of RDP !
parties concerned with RDP planning: sub-regional authorities; LAGs etc.)
parties concerned with RDP implementation: MA, paying agency, measure responsibles
evaluation unit, experts for M&E, national networks, social partners
23 June 2009
Evaluation Expert Committee Meeting

15. (STEP 4): Establishment of the evaluation mandate 
CMEF
[…] In order to allow a target-oriented preparation of the terms of reference in the steering group, it is advisable to establish an evaluation mandate, giving an overview on context, scope, timing, and objectives of the evaluation in question.
Guidance note B, p.9
23 June 2009
Evaluation Expert Committee Meeting

16. (STEP 4): Establishment of the evaluation mandate 
Use evaluation mandate as guide for the preparation of ToR and the work of the SG.
Key questions?
Has the MA or SG specified the context, scope, timing, and objectives of the MTE in the form of an “evaluation mandate”?
Who has been involved in the preparation of the evaluation mandate?
Has the evaluation mandate been acknowledged by all involved bodies? (MA, SG, evaluators etc.)
MA or evaluation unit should initiate preparation of mandate.
Involve the SG in order to ensure consistency.
23 June 2009
Evaluation Expert Committee Meeting

17. STEP 5: Establishment of the ToR
CMEF
The terms of reference serve the purpose of establishing the framework for evaluation activities during the different phases of ongoing evaluation (ex-ante evaluation, mid-term evaluation, ex-post evaluation).
In addition, they form the basis for calls for tenders or the establishment of service requests within framework contracts.
Guidance note B, chapter , p.9
23 June 2009
Evaluation Expert Committee Meeting

18. STEP 5: Establishment of the ToR
Ensure that previous steps are undertaken (review of questions, information gathering etc.)
Key questions:
Has a proper framework for the MTE (incl. as part of the ongoing evaluation activities) been established?
Have results from ongoing evaluation activities been properly fed into the ToR?
Has an appropriate timetable been defined (incl. deadlines, meetings etc.)?
Do the ToR properly reflect the evaluation requirements?
Ensure that findings from ongoing evaluation fed into ToR.
Cross check with outline-structure .
23 June 2009
Evaluation Expert Committee Meeting

19. STEP 5: Establishment of the ToR
framework, legal requirements, purpose
Indicative OUTLINE for technical specifications for the Terms of Reference:
(a) Context of the MTE
(b) Scope of the MTE
(c) Objectives of the evaluation
(d) Evaluation questions
(e) Tasks to be performed
(f) Timing and content of deliverables
(g) Organisation of work
(h) Sources and documentation
description of measures, geographical area, period, specific focus
common + programme specific objectives
structuring, observing, analysing, judging
budget, responsibilities, interaction with SG
23 June 2009
Evaluation Expert Committee Meeting

20. STEP 5: Further considerations
Ensure that ToR reflect scale and scope of the programme and the required evaluation activities.
Key questions:
Are ongoing evaluation activities (and MTE) adequately resourced to be effective?
Is the concept of the independence of evaluators respected?
Is a regular consultation with stakeholders ensured, are the interactions clarified?
Does the timing for launching/closing the procedures allow time for submitting the final report to the EC within the deadlines?
Bodies without direct involvement in the implementation, management and financing of the programmes.
Ensure sufficient time for preparation, primary research, interaction with the steering group….
CMEF gives limited guidance on PROCEDURAL issues ( responsibility of MAs!):
23 June 2009
Evaluation Expert Committee Meeting

21. Evaluation Expert Committee Meeting
IMPLEMENTING the MTE
Provide for an effective inception process with an accurate project management plan.
Key questions:
What does an independent but interactive evaluation process look like?
Is an ongoing dialogue between evaluators, steering group and MA ensured?
How can the quality of the final MTE Report be guaranteed?
Can the final MTE-Report be submitted in time to the MC and the EC?
Do not leave the evaluator alone!
Provide for iterative and regular exchanges with the steering group.
Anticipate a quality assessment of the final MTE-report by the steering group.
CMEF gives limited guidance on PROCEDURAL issues ( responsibility of MAs!):
23 June 2009
Evaluation Expert Committee Meeting

22. DISSEMINATING the results
Key questions:
How can an effective dissemination process be planned?
What are appropriate means for dissemination?
Define from the outset the purpose, target audiences, dissemination tools, timing and resources.
Targeted presentations,
conference contrib., publications, case studies, extracts, summaries, thematic papers, newsletters, websites, discussion forums.
CMEF gives limited guidance on PROCEDURAL issues ( responsibility of MAs!):
23 June 2009
Evaluation Expert Committee Meeting

23. Timeline …
2010
2009
2011
Review Common & Specific Evaluation Questions
Establishing the steering group
Data Collection and Information Gathering
Evaluation Mandate
Drafting the Terms of Reference
PREPARATION
IMPLEMENTATION
Independent and Interactive Evaluation Process: structuring, observing
analysing, judging
Ongoing Dialogue between Evaluator, SG, MA
Quality Assessment of Final Report by SG
Submission Final MTE- Report to MC & EC
DISSEMINATION
Dissemination of Evaluation Results through various means (presentations, contributions to publications, articles, websites..) to different Target Groups.
23 June 2009
Evaluation Expert Committee Meeting

24. Helpdesk-Tools for the MTE
Survey on preparation of MTE-results
Guidelines on preparation of the MTE
Collection of FAQ on Mid-Term-Evaluation
Good Practices on MTE-preparation process etc.
PPT on specific issues (e.g. ongoing evaluation, MTE-preparation etc.)
23 June 2009
Evaluation Expert Committee Meeting

25. Thank you for your attention!
23 June 2009
Evaluation Expert Committee Meeting