Measuring Impact – what can we learn from recent DWP evaluations? Simon Marlow

Overview What support has been/is available to Jobcentre Plus claimants? Focus on ‘work experience’ and Future Jobs Fund Impacts Cost Benefit Analysis of Future Jobs Fund Future thoughts Focus on WE and FJF – number of other programmes for young people WP

Claimant journey for 18-24 year olds on JSA Post mid-2011 New claim Maximum of 9 months JCP offer Includes: 1) Core interventions – signing etc. 2) Flexible interventions eg. work experience, Mandatory Work Activity 3) Back to work support 4) Flexible Support Fund Work Programme – Mandatory for 2 years. Early access at 3 months for disadvantage groups and with long benefit history Post Work Programme - trial Length of JSA claim 2009 - mid-2011 New claim Flexible New Deal Phase 1 areas - 1 year Flexible New Deal – Mandatory for 1 year Early Access for some groups The flagship programme in all of this is the WP – where support for the LTU is contracted out of the Department and to external providers The Key principle of the Work Programme being sustained job outcomes The Key features are: Greater emphasis on payment by results Promotion of competition and everything that can bring – Competitive tendering, Cost-effectiveness, increased outcomes etc. Promoting innovation, both within JCP and external contractors – for example greater use of the 3rd sector Establish longer contracts and provide stability for contractors and ultimately greater opportunities to learn and increase effectiveness Pre-work Programme initiatives Post Work Programme initiatives Initiatives can mandatory or voluntary and flexibility/adviser discretion is promoted Starting to evaluate the new intiatives Young Person’s Guarantee (includes Future Jobs Fund) Flexible New Deal Phase 2 – 6 months New Deal for Young People – Mandatory Early Access for some groups

Work experience (WE) and Future Jobs Fund (FJF) WE – January 2011 - present Who it is for? Mainly 18-24yrs old (a few 25+) Guidance - 13wks + JSA claim duration JCP adviser discretion to refer Aim Help people gain practical experience they need to secure a job What is it? 2-8 week placement with employer Employers expected to allow time for job search where possible Conditionality Voluntary scheme once referred Continue signing; claiming benefit No sanctions for non-completion Cost - £325/per participant FJF – October 2009 – March 2011 Who was it for? Mainly 18-24 (a few 25+ in hotspots) Guidance - 26 weeks JSA duration Voluntary Aim Build skills/work experience for disadvantaged jobseekers to help them find unsubsidised employment What was it? 6m subsidised placement, 25 hrs/wk at least min wage Additional jobs, benefit local communities Organisations mainly from public sector and third sector Conditionality Voluntary, but from Apr 10 if refused, had to do ‘Community Task Fund’ (another strand of the Young Person’s Guarantee) Stop claiming benefit Cost - Employers were paid £6,500/per placement – about £650m total

WE and FJF reach similar levels of take up By end of Nov 11 WE starts reach 6k/month By Apr 10 FJF starts reach 6k/month FJF reached 105,000 So far WE has reached xxx

Methodology Want to understand programme impact on benefit receipt and employment Isolate impact by trying to establish a plausible counterfactual In absence of an RCT, Propensity Score Matching used: Regress on observable characteristics from DWP data to try and get probability of participation (controlling for selection) Match to non-participant group (JSA claimants from same period without a start on the programme) Pseudo starts given to non-participant group so that outcomes can be compared Main assumption: hope that observable characteristics can explain participation (ie needs to explain participant ‘motivation’)

Matching quality on observables is very good Before matching After matching

Work Experience – 1,300 individuals, Jan – March 2013 Benefit Impacts – 6 ppts at 21 weeks/5 fewer days Benefit receipt rate . Employment rate Employment Impact – 8ppts at 21 weeks/8 additional days

FJF impacts on Benefit/Welfare support – first 12,000 individuals from Oct 2009 – April 2010 Benefit impacts Over 6m - 127 fewer days 18-24m – -7 ppts/ 41 fewer days Benefit receipt rate ‘Welfare support’ (benefit or FJF job) impacts Over 6m - 33 additional days 18- 24m – average -7 ppts/ 41 fewer days ‘Welfare support’ (benefit or FJF job) receipt rate Baseline sensitivities – broadly similar results

FJF impacts on employment/unsubsidised employment (caveat – using HMRC P45 data) Employment impacts 0-6m – 99 additional days 18-24m – average 10 ppts/ 57 additional days Employment receipt rate Unsubsidised impacts 0- 6m – 45 fewer days 18-24m – average 10 ppts/ 57 additional days Unsubsidised employment (job other than FJF job) receipt rate

Summary of Impacts Caveats WE - by 21 weeks: Benefit Impact 6 ppts off benefit – overall 5 fewer days. Employment Impact 8 ppts in employment – overall 8 additional days FJF – over 2 years Benefit/Welfare Impact 0-6m – 127 fewer days off ben/33 more days on welfare support 18-24m – average 7 ppts off ben; 41 fewer days off ben Employment/Unsub employment impact 0-6m – 99 add days in employment/45 fewer days in unsub employment 18-24m – 10 ppts in employment; 57 additional days in employment Caveats Work experience first early cohort – only 1,300 starts; only 21 weeks More likely than not to be overestimates for both HMRC data poor quality so employment impacts are less robust Compare with New Deal for Young People – 64 fewer days off JSA over 4 years International job creation scheme evidence poor

Costs & benefits of the Future Jobs Fund FJF programme impact Cost (-) / benefit (+) per participant (£) Participants Participants’ employers Exchequer Society Increase in output + 4,400 Increase in wages + 4,300 - 4,300 FJF employer payments + 6,850 - 6,850 Reduction in operational costs + 900 Reduction in benefits - 1,450 + 1,450 Increase in taxes - 1,000 - 100 + 1,100 Increase in travel & childcare costs - 250 Reduction in healthcare costs + 300 Redistributive costs & benefits + 2,400 Social cost of exchequer finance Net Present Value (NPV) + 4,000 - 3,100 + 7,750 4 perspectives Lots of caveats around this If cohort is representative, then net cost to exchequer ~ £330m (against gross cost of programme - £720m)

Sensitivity to assumptions Wide range of uncertainty

Future Developing our ways to measure impacts and cost benefit analysis of programmes Extend work experience analysis Day one support trial Work Programme evaluation

References Early work experience impacts http://statistics.dwp.gov.uk/asd/asd1/adhoc_analysis/2012/early_impacts_of_work_experience.pdf Future Jobs Fund impacts, costs and benefits http://statistics.dwp.gov.uk/asd/asd1/adhoc_analysis/2012/impacts_costs_benefits_fjf.pdf Two main sources of guidance for CBA: The HM Treasury Green Book http://www.hm-treasury.gov.uk/d/green_book_complete.pdf The DWP Social-Cost Benefit Analysis Framework http://research.dwp.gov.uk/asd/asd5/WP86.pdf