Evaluation Helpdesk Peer reviews of Cohesion policy evaluations Terry Ward Applica ESF Evaluation Partnership Meeting Brussels, 28 September 2017 ISMERI EUROPA

Helpdesk activities Three main activities of Helpdesk in relation to ESF: to report on all evaluations of Cohesion policy programmes undertaken in current programming period and provide short reviews of these to review evaluation plans of MAs for current period and revisions of these to undertake peer reviews by experts of selected evaluations of Cohesion policy programmes

Peer reviews Aim: to subject selected evaluations to critical appraisal by leading evaluation experts to highlight strengths and weaknesses and ways of improving Purpose: to encourage MAs to undertake good quality evaluations and to demonstrate how to do so From evaluations reviewed, shortcomings evident as discussed at Evaluation Partnership meeting in Milan in October last year: evaluation questions – too many, too imprecise, too ambitious methods specified not appropriate to address evaluation questions inadequate supervision of evaluation process Failures originating partly from deficiencies in evaluation process itself: ToRs poorly framed restrictive procurement regulation weight of price in selection criteria disproportional

Peer reviews – extended approach Implication: to improve evaluations, need to review whole evaluation process to identify reasons for shortcomings and how to correct them – i.e. not only to advise evaluators but also MAs employing them To subject ‘evaluation dossiers’ rather than just final evaluation reports to critical review, covering: ToRs Selection of evaluators Budget Inception and interim reports as well as final Other documents relating to process Requires cooperation of MAs to have access to reports, documents and details of evaluation process To open up peer review meetings to MAs (and possibly evaluators) so that they can learn directly from experts’ comments and advice and enter into discussion with them

Putting these steps into practice Three extended peer reviews in 2017: First in Malta on ERDF programmes in May Second in Berlin on ESF programmes in July Third will take place in Tallinn in October Two dossiers subject of reviews at each meeting: Completed programme: Dossier of ex post evaluation relating to 2007-2013 period Ongoing programme: Dossier of planned evaluation and ToR (Malta) and of mid-term evaluation of 2014-2020 programme (Berlin, Tallinn) Both dossiers subjected to critical appraisal by 3-4 leading evaluation experts with MA present at meeting (plus evaluators in Berlin) Written assessments by experts on various elements in dossiers circulated to participants ahead of meeting

Zoom on peer review meeting in Berlin 14 July One-day meeting: 17 participants, 2 evaluation dossiers, discussion in German and English (simultaneous translation) Meeting was a pilot exercise: first time that evaluators participated Specific points from discussion: Two step tendering process: participation competition followed by award competition Selection criteria in participation competition: experience and professional qualification Successful tenderers in participation competition, invited to prepare proposal and make presentation. Details discussed at presentation. Offer revised and fine tuned. Price is part of tender competition: budget not mentioned in ToR Conflict between political and programme agenda: pressure to present results earlier than available – evaluation too early to capture effects entirely Ex post evaluation logic conflicts with timely planning of data needed for randomisation approaches and other innovative methods

Results of the extended peer reviews – first observations Discussion at meeting of elements in evaluation dossier helped to better understand process and identify strong and weak points in cycle In one case, national procurement regulations found to hinder selection of evaluators – need for change Expert guidance on ways of improving evaluation more tangible and to the point because of understanding of underlying process and MA aims Criteria used to assess evaluations sent before meeting useful for: highlighting minimum set of standards which evaluations should comply with demystifying peer review process by making approach transparent Extended peer reviews require more effort from both peer reviewers and MAs, but result in more useful outcomes – MAs learn as well as evaluators

Evaluation Helpdesk is ready and willing to support MAs in similar way How widen effects? As emphasised by Gábor TÓTH at last ESF Evaluation Partnership meeting in March this year: Follow an evaluation from its inception and advise on each part of process before put into effect so as to improve it To use such evaluations as ‘showcases’ so that other MAs can learn from them Invite other MAs to participate in meetings and learn from discussion Evaluation Helpdesk is ready and willing to support MAs in similar way

Thank you for your attention