Evaluation of SF in Romania Creating A Procedural Manual for Structural Fund Evaluations Ann Hyde CSF MA - Twinning Project Bucharest, Romania May 2006

What is the purpose of a guidance manual for evaluation? Guidance for each evaluation unit. Source of information. Guarantee a uniformity of approach. Step by step guidance about the whole process of evaluation. Training and capacity building tool.

When will an evaluation manual be used? When deciding When an evaluation should take place? What should be evaluated? Who should be involved in the evaluation? How the evaluation should be carried out.

Main Sections Introduction – Managing evaluations concepts and definitions Managing evaluations Systems, structures Preparing for evaluations Procedures, kick off Contracting evaluations Procurement and managing the contract Disseminating the results of evaluations Debriefing, publishing and distribution

Concepts and Definitions What is evaluation – definition of aims What evaluation is not. Types of evaluations relevant for SF? What can be evaluated? The key evaluation issues. The uses of evaluation

Managing SF evaluation Defining the operational structure for evaluation – proposed model Setting the Evaluation standards. Identifying (agreeing) the flow of information across the organisational structures. Establishing an evaluation steering committee.

Preparing for an evaluation Identifying the aim and purpose of the evaluation? Defining the scope of the evaluation Defining the key evaluation questions. Formulating the analytical requirements Taking stock of available information? Defining the uses and users of the evaluation

Preparing for an evaluation Describing the programmes to be evaluated Describing the methodologies to be followed in data collection and analysis Identify stakeholders Prepare an indicative evaluation work plan and budget. Elaborating a project concept for the evaluation.

Contracting and Procurement Preparing the terms of reference Deciding on type of tender Preparing advert for tender Issuing tender/ call for proposals Evaluating tender Drafting contract Signing contract Scheduling kick off meeting

Evaluation Steering Group Define purpose of the evaluation steering group; Prepare evaluation steering group timetable. Arrange steering group meetings. Define Quality Assurance and compliance checks

Kick Off Meeting Define purpose of kick off meeting. Prepare evaluation stakeholders for the evaluation kick off Prepare contractor for the kick off Ensure ToR, evaluation questions and timetable are up to date. Arrange kick off meeting.

Managing the evaluation period Define flow of information between contractor – evaluation unit – steering group. Define reporting procedures. Set up system for commenting on draft evaluation report (s) Establish QA for report.

Evaluation complete Prepare stakeholders for debrief; Prepare contractor for debrief; Prepare for publication of evaluation report; Define parameters for information dissemination; Arrange full distribution.

Evaluation Follow Up Prepare evaluation follow up procedures; Prepare action plan for implementation of recommendations; Prepare timetable for follow up; Establish QA procedures for evaluation procedures.

The first evaluation In the beginning God created the heaven and the earth. And God saw everything that He made. “Behold,” God said, “it is very good.” And the evening and the morning were the sixth day. And on the seventh day God rested from all His work. His archangel came then unto Him asking, “God, how do you know that what you have created is ‘very good’? What are your criteria? On what data do you base your judgement? Aren’t you a little close to the situation to make a fair and unbiased evaluation?” God thought about these questions all that day and His rest was greatly disturbed. On the eight day God said, “Lucifer, go to hell.”