Supporting material for CIE Video tutorials standard introductory e-learning tools to learn about CIE: outsourcing, application, use of results, choice of data to add subtitles to share them with the consultants

Tips and tricks on how to plan intervention design in view of future counterfactual impact evaluations European Commission Centre for Research on Impact Evaluation

A What’s inside the magic box? Tricks Some practical precautions that can make CIE easer to be carried on. Tips Hints on the most suitable CIE method to be applied in each situation. Typical situation For each trick, the situation in which it best applies.

Main message of the video B Always keep in mind the importance of creating a proper comparison group!

Example 1

Example 1 Launch of “National Supported Work Demonstration” by Manpower Demonstration Research Corporation. Aim of the intervention: to offer job training programs to particular worker categories in need. To participate, people need to apply.

Example 1 Treated group: random selection among applicants. Control group: not selected applicants. Intervention: 9 to 18 months of payed supported work. To disentangle the average treatment effect (ATE) : compare average employment rate of the treated and non-treated group 6 months after the end of the training.

Example 1 Trick Randomise treatment: randomisation creates two similar groups, except for one of them being treated. Tip The average treatment effect is given by the difference between the average employment rate of the treated and non treated. Typical situation In case you have more applications than available slots, allocate them trough randomisation.

Example 3

Example 3 As in example 1, the intervention consists in offering job training programs. Priority is given to poorer applicants. Participation (treatment) will be decided upon a threshold calculated with respect to a yearly income indicator.

Example 3 Treated group: all individuals with income indicator below the threshold level of 250 €. Control group: all individuals with income indicator above the threshold level (excluded from the intervention). The effect of the intervention will be computed by comparing the average employment rate of treated people just below the income threshold level with that of untreated workers just above the same threshold.

Example 3 Trick Allocate treatment on the basis of a score (forcing variable) with a threshold. The threshold must fall in the middle of the distribution of the forcing variable: ensure having both treated units and controls! Need to collect data both for the treated and the controls. The RDD works as a local randomisation around the threshold.

Example 3 Tips Regression Discontinuity Design (RDD). Typical situation Treatment assignment is a function of some variable (income, age).

Example 2 Example 4 Example 5 Example 6 Example 7

Summing-up Take-home message Use foreseen (and unforeseen) characteristics of the policy intervention to create an appropriate comparison group!

Summing-up Several typical situations imply specific class of CIE methods:

Summing-up Not all methods are applicable in a single intervention. Different methods estimate different kinds of impact: RDD estimates the impact for individuals at the threshold PSM estimates ATT DiD estimates ATE IV estimates LATE  Choose the most appropriate impact for your intervention

Summing-up When designing the intervention, pave the way for the application of 2 or 3 methods Save money Have reliable results CIE methods can be combined and refined to fit special situations and purposes  ask to a CIE expert (https:\\crie.jrc.ec.europa.eu/?q=content/cie- market)!

Stay in touch CRIE website: crie.jrc.ec.europa.eu email: jrc-crie@ec.europa.eu YouTube: JRC CRIE Yammer: https://www.yammer.com/counterfactualimpactevaluationnetworkcie-net/#/home