Guide for Terms of Reference A checklist European Commission Centre for Research on Impact Evaluation ESF Partnership Meeting 15 March 2017

A. Who do we address? Managing Authorities in charge of European Social Funds may outsource the evaluations foreseen in their Evaluation Plans For this, a Terms of Reference document (ToR) needs to be prepared

B. What is a ToR? The TOR defines the working arrangement with the external evaluators More precisely it sets: The (boundaries of the) design of the evaluation Access/collection of micro-data The award criteria What to do in case of unforeseen hurdles («Plan B»)

C. What should be included? The ToR should contain the following sections: Introduction and description of the intervention 1 Scopes and objectives of the evaluation 2 Data availability 3 (Alternative foreseen) methodological approaches 4 Award criteria 5 List of deliverables and time schedule 6

1 Present why the intervention was chosen for evaluation Introduction and description of the intervention 1 Present why the intervention was chosen for evaluation Say what change the intervention intended to induce (outcome(s) of interest Y) Describe the intervention implementation: eligibility criteria, voluntariness of participation (reference population) Precise what other factors in addition to participation are likely to affect Y

Introduction and description of the intervention 1 Risks If there are too many outcomes of interest Y, it is unlikely that the evaluation may address them all If the intervention is not well described and the reference population is not well defined then the design of evaluation will likely be incorrect If other factors affect the outcome of interest, they may confound the effect of the policy intervention !

1 Remedies Introduction and description of the intervention Select just a few outcome(s) of interest Y Choose the ones that are closer to the intervention in a “theory of change” expected causal chain Define the reference population Are there individuals in the reference population that did not participate in the intervention? These individuals could define the comparison or control group Provide for collecting data on other important factors affecting the outcome of interest

Introduction and description of the intervention 1 Example: Training program for unemployed youth (26 or less) introduced in some regions only Reference population: young unemployed (aged 26 or less) Control group: Unemployed individuals (aged 26 or less) living in a region not participating in the training program

2 Scopes and objectives of the evaluation Define the evaluation questions Describe the “theory of change” linking the intervention to the outcomes of interest This theory of change is a list of Q&A such as: did the training increase the likelihood of the trained unemployed person to search for a job? did the increased search for a job lead to an increase in the likelihood to be employed 6 months after the training? Where can the evaluator look for data on outcomes of interest and other factors for the reference population?

! 2 Scopes and objectives of the evaluation Risks Unclear evaluation questions will lead to unclear answers Lack of a “theory of change” renders the evaluation not credible Lack of available data may render the evaluation impossible, even in presence of a sound “theory of change” !

2 Scopes and objectives of the evaluation Remedies Specify few (at most 5) well defined evaluation questions such as: Has the training increased search efforts? Has the training increased the likelihood to find a job? State the theory of change explicitly in the ToR Training is expected to increase search efforts, which in turn should help increase (together with the acquired new skills) the likelihood to find a job within 6 months after the training Do not anticipate evaluation results: the theory of change expresses the rationale for the intervention / intended causal chain; the evaluation will say what the effect was in reality

Data availability 3 The ToR should anticipate where the data can be found, and state the rules of the game for the contractor Is administrative data available to the contractor (possibly in anonymised form)? Shall the contractor collect data itself? How? Through surveys? In this case where can the contractor find a registry of the reference population?

! 3 Data availability Risks If no detailed provision on data is made in the ToR, the contractor may not be able to carry out the work! If the data is provided to the contractor, the cost of the contract will be much lower than otherwise Shifting all responsability of the data collection to the contractor will almost certainly lead the contractor to select the most inexpensive solution, which is usually the one of the lowest quality

3 Data availability Remedies Describe the data sources that can be made available to the contractor Ensure that data is available or will be collected both on participants and non participants of the intervention Try to give access to administrative data for better quality evaluations Specify the data protection requirements (e.g. anonymisation procedures, data protection officer)

4 Methodological approaches The available data allows for the application of a range of counterfactual methodologies; these should be explicitly foreseen The set of counterfactual methodologies shall be consistent with the evaluation questions and available data Leave room to the contractor to propose innovative solutions within the range of counterfactual methodologies

! 4 Methodological approaches Risks Not all methods can be applied at once in a given context! The contractor may adopt a trial and error approach in order to select the methodology that gives evaluation results closest to expectations; the chosen methods need not be sustainable or appropriate in the given context No sophisticated method can compensate for poor data quality

4 Methodological approaches Remedies Specify the category of methods (CIE) to be used Ask the contractor to explain why the chosen methodologies are appropriate in the given context Ask the contractor to provide supporting evidence that the assumptions underlying each method are valid in the given context

5 Selection and Award criteria Set minimum requirements on: the size and experience profile of the evaluation team (publications on CIE to be preferred) the qualifications of the principal investigator: who is she/he? Which publications indicate expertise in CIE? the necessary human and technical resources the distribution of responsibilities

! 5 Selection and Award criteria Risks Usually 2 award criteria are used: a) price and b) quality of the project. If quality of the project is evaluated close to uniformly (i.e. with little variation among projects), then price is the effective award criteria When price is the effective award criterion, poor evaluation teams are usually awarded the contract You can save on the cost of the contract by providing access to data, which would lower the cost without affecting quality

5 Selection and Award criteria Remedies Invest in the quality assessment of proposals by: Including CIE experts in the selection panel (you can hire an independent consultant) Requiring bidders to state in their proposal the principal investigator for CIE as well as all the staff of the evaluation team Asking your CIE expert in the selection panel to evaluate publications of the CIE principal investigator and evaluation team Foreseing precise quality control arrangements throughout the evaluation process Providing data access to the contractors

6 List of deliverables and time schedule Require quality checks throughout the process Ask bidders to: provide a detailed timetable of the tasks and subtasks describe their planned quality control arrangements identify potential risks/challenges linked to the evaluation assignment detail how they would manage and overcome those risks/challenges hand-in both the evaluation study and the underlying data. This will allow to replicate the analysis

6 List of deliverables and time schedule List of deliverables with deadlines: Inception report Intermediate report Final report Reports shall be presented in public and possibly discussed by invited CIE experts Allow some extra time for unforeseen hurdles, often associated with access to data

! 6 List of deliverables and time schedule Risks Unfeasibly short turnaround times will generate reports of bad quality Excessively long turnaround times with no checks are useless Checks made at the wrong time or by non-experts will probably not help

6 List of deliverables and time schedule Remedies Set a reasonable timeline: Each phase (inception, intermediate, final reports) shall take 3-4 months The collection of data may even take longer than this. Use intermediate deliverables to ensure that the evaluation meets the expectations, using help of experts Involve external CIE experts in order to check the quality of the ongoing evaluation overtime

Do not hesitate to contact DG EMPL for additional support you might need on ToR, using the different channels at your disposal (Yammer, email, CoP, CRIE website)

Stay in touch CRIE website: crie.jrc.ec.europa.eu email: crie@jrc.ec.europa.eu YouTube: JRC CRIE Yammer: https://www.yammer.com/counterfactualimpactevaluationnetworkcie-net/#/home