1. 1 The Need for Control: Learning what ESF achieves Robert Walker

2. Identifying impact Time Outcome Control or counterfactual Policy success? Introduction of policy

3. Impact Defining the counterfactual Time Outcome Introduction of policy Control group Action group (Treatment group)

4. Defining control groups Random assignment Random encouragement designs Matching areas Individual matching Propensity score matching Difference in difference Regression discontinuity designs

5. Defining the control group Random assignment 1 2 6 9 5 3 4 10 12 8 7 11 Action group Control group Random number =

6. Defining the control group Random assignment 1 26 9 5 3 4 10 8 7 11 Action group Control group 12 Random = 7 = Control Number Group

7. Advantages of random assignment Action group and controls same (with large numbers) with respect to: –Observed characteristics –Unobserved characteristics Unbiased estimate of mean impact Possible to state degree of confidence that estimate is a true measure of impact

8. Non- participants Eligible individuals General population Participants Control group Programme group Random assignment Determine eligibility Ineligible individuals Random assignment Adapted from Orr (1999)

9. Non- participants Eligible individuals Jobseekers Finland Participants Control group Jobsearch training Random assignment Determine eligibility Ineligible individuals Jobsearch training - Finland Malmberg-Heimonen and Vuori (2005) If staff thought would benefit & Did not express a preference for training 677 19 employment offices 338 430247 20.3%19.5% find employment Seek informed consent 18% asked by post, 28% asked face to face agreed

10. Jobcentre Plus office New Deal 25+ New Deal for Lone Parents Working Tax Credit ERAD Employment Retention and Advancement Demonstration PC RA Mandatory WFI Voluntary JC+ visit PC RA PC JB+ initiated contact

11. Participants Target population Non- participants Encourage to participate No encourage -ment Random assignment Participants Non- participants Randomised encouragement design IMPACT

12. Defining the control group Geographic areas (Limited equivalence) Population

13. Area matching UK - Educational Maintenance Allowance

14. Propensity Score Matching Programme Control Criteria for inclusion in control Have the combination of characteristics making them likely to be included in the programme group

15. PSM Norway, Rehab and Activation Rønsen and Skarðhamar (2009) Months to finding work Participants 0 5101520 Controls (Social security recipients) Months until 25% find work Controls20 Participants11

16. Difference in difference design Outcome Time T1T1 T2T2 Programme group Control group Counterfactual Difference at T 2 (D 2 ) Difference at T 1 (D 1 ) IMPACT (D 2 - D 1 )

17. Regression discontinuity design Measure of need/eligibility (at time T 1 ) Outcome (at time T 2 ) (Measure of need) High Low

18. Regression discontinuity design Measure of need/eligibility (at time T 1 ) Outcome (at time T 2 ) (Measure of need) High Low Threshold

19. Regression discontinuity design Measure of need/eligibility (at time T 1 ) Outcome (at time T 2 ) (Measure of need) High Low Threshold Non-participantsParticipants IMPACT

20. Regression discontinuity design: Extended UB entitlement, Austria IMPACT 14.8 weeks extra unemployment Unemployment duration Age Lalive (2008)

21. CharacteristicRandom assignmen t Random encouragement MatchingRegression Discontinuity Individual level Area control s Propensity score matching Characteristics of policy intervention (1) Measure systemic effects No (Yes, for Cluster RA) No YesNoYes Multiple target groups Possible for a few Difficult but possible for a few Possible for a few Not easy Yes Multiple policy components Possible for a few Difficult but possible for a few Not easy Competing policy designs Possible for a few Difficult but possible for a few Not easy

22. CharacteristicRandom assignmnt Random encouragement MatchingRegression Discontinuity Individual level Area control s Propensity score matching Characteristics of policy intervention (2) Need to measure second order effects No (or only by using quasi experimental methods) Yes (but only by further supplementary individual level matching, i.e. further quasi-experimental methods) Targeted on hard to reach group DifficultNot easyDifficultNot easy DifficultNot easy Targeted at areas No (Yes, for Cluster RA) Not easyYes

23. CharacteristcRandom assignme nt Random encouragemnt MatchingRegression Discontin-ty Individual level Area control Propnsty score matching Ease of implementation Disruption of existing policy YesNoYesNo Requirement for informed consent YesNoYesNo Ethical concerns Real for all prospective evaluation since people are placed at risk of harm due to possible negative effects of previously untested policy for the common good Scale: sample sizes Small Very large Mod- erate LargeModerate Data requirements Low ModerateMod- erate HighModerate

24. CharacteristicRandom assignmnt Random encouragement MatchingRegression Discontinuity Individual level Area control s Propensity score matching Quality of impact estimates Unbiased impact Estimate Yes No (yes at margin) Contamintion of control group Moderate Low Control for unobservable variables Yes No Overall control Excellent Poor Moderate External validity Limited ModerateLimitedHigher

25. Conclusion No evaluation method is perfect Evaluation without a counterfactual is very, very likely to wrong and misleading Only random assignment (and regression discontinuity designs) produce unbiased estimates It is impossible to be confident that controls are adequate Control groups are generally better than no control group – they force evaluators to think about the characteristics of a good counterfactual