1. Heads you’re in; tails you’re out: How RCTs have evolved in DWP Jane Hall jane.hall@dwp.gsi.gov.uk

2. Background  Increasing use of RCTs  Emphasis on evidence-based policy  Limited UK experience in social policy arena  Practical lessons not theoretical debate

3. Overview  Chronology of RCTs in DWP  Site selection and preparation  Identifying the eligible population  Dealing with resistance  Performing the random assignment  Monitoring take-up

4. Chronology of RCTs  Restart  Various New Deals  Employment Zones  JRRP  ERA  JSA Intervention Pilots  ND50+ Mandatory IAP

5. Site Selection and Preparation  Need commitment from the top  All parties need to buy-in  Set-up is resource intensive  Personal visits  Pilot the approach

6. Identifying and recruiting the eligible population  Can they be easily identified  Self-selection  Suitability of the population  Monitor P & C Group characteristics  Selling techniques  Sample sizes: Sub-group analysis

7. Dealing with resistance  Busting the myth  Significant investment in training at all levels  Aides and FAQs  Scripts

8. Performing the Random Allocation  Needs to be sophisticated  Not open to sabotage/gaming  Block allocation: Maintain P:C ratio  Different techniques –NINO –Call Centre –On-line algorithm –Random numbers

9. Monitoring take-up  Keep track of P & C Group  Ensure only P Group receive the treatment  Monitor key characteristics of P & C Group  Be prepared to redesign the random allocation

10. Expect the Unexpected  Results may not be what you anticipate  A fair allocation of resources?  Participation rates can be disappointing

11. Operational Challenges The ERA Experience Jenny Carrino

12. Overview  The ERA Policy  Key Challenges –Random Assignment (RA) Process –Customer Understanding of RA –Creation of ‘Informal’ Refusers –Jobcentre Plus Target Structure –Technical Assistance

13. The ERA Policy  To test interventions to improve retention and advancement –Adviser support –Funding for training –Financial Incentives  6 Jobcentre Plus districts  Three customer groups NDLP ND25+ WTC  To test the effectiveness of using RA to evaluate social policy in the UK

14. Random Assignment 1  Issue: The random allocation process  Lessons Learnt –Importance of transparency in the allocation process –Avoiding contamination  Outcomes –Most customers and staff viewed random assignment as fair and justified

15. Random Assignment 2  Issue: The Informed Consent Process  Lessons Learnt –Standardisation - adviser scripts and leaflets  Outcomes –Not everyone fully understood what they had signed up for –Too much information at initial interviews – conduct RA as a stand alone interview

16. Random Assignment 3  Issue: Creation of a group of ‘informal’ refusers

17. What do we mean by Informal Refusers? ELIGIBLE POPULATION CUSTOMERS RA’D OR ON SYSTEM FORMAL REFUSERS INFORMAL REFUSERS SIGNIFICANTLY DIFFERENT

18. Informal Refusers  Why this happened –The decision to use RA –RA to ERA was voluntary –Influences from both advisers and customers  Outcomes –Creation of a ‘third’ group –Analysis to identify whether this group are different to the ERA population  Lesson Learnt –If possible monitor intake closely against eligible population

19. Ensuring a Treatment - Targets  Issue: The Jobcentre Plus Target Structure –Some adviser behaviour negatively affected –Senior management buy-in affected  Lessons Learnt –Policies need to reflect the organisations reward system –Need to be able to monitor and feedback to implementation managers

20. Technical Assistance  Issue: Ensuring the effective delivery of RA  Lessons Learnt –US model of on-site RA assistance –Avoiding contamination –Monitoring Performance  Outcomes –Advisers felt supported during the RA period –Initial confusion over the role of TAs –Some districts deferred responsibility of ERA implementation

21. Summary of Key Challenges  RA Process  Informed Consent  Creation of informal refusers  Ensuring a Treatment  Providing effective support to delivery agents

22. Jobseekers Allowance (JSA) Intervention Pilots Jayne Middlemas

23. JSA Intervention Pilots  JSA Intervention Regime  The Pilots  Evaluation  Random assignment  The data  Did Random Assignment work?  Results

24. JSA Intervention Regime  First Contact  New Jobseeker Interview (NJI)  Financial Assessor Interview  Fortnightly Jobsearch Review (FJR)  13 week review

25. The Pilots  Introduced in January 2005  108 Jobcentres in 10 Districts took part  Each Jobcentre took part in a single pilot  Aim to deliver resource savings on the FJR without reducing unemployment off-flow rates.

26. The pilots (cont) Five different approaches:  Excusal of signing for first 13 wks of claim  Excusal of signing for first 7 wks of claim  Telephone signing  Shortened FJR  Group signing

27. The pilots (cont) Some groups excluded:  Part-time workers  16 and 17 year olds  People with no fixed abode  People known to have had a fraudulent claim in the past

28. Evaluation  Customers randomly allocated  Work study to record resources used  Comparison of off-flows  Qualitative evaluation

29. Random Assignment 50% programme, 50% control ORC International Call Centre Two methods:  Adviser calls immediately prior to each NJI  Jobcentre calls at start of day with details of all clients due to attend an NJI that day ORC also provided random call-in date

30. Data  Data collected during random assignment  JUVOS data – derived from the Jobseekers Allowance Payment System (JSAPS)  HMRC Employment Data

31. Did Random Assignment Work?  66,600 randomly assigned  33,100 programme & 33,400 control  All pilots and Districts close to 50/50 split Was everyone assigned?  Difficult to answer precisely  Number randomly assigned around 90% of total new claims. Excluded groups likely to account for 8 to 12%

32. Were People Wrongly Assigned?  19% had no new JSA claim during the pilot  Incorrect NI numbers may mean we can’t find some claims  Jobcentres didn’t always inform us of those who failed to attend  Can’t identify excluded groups in the data

33. Internal Validity  Compared characteristics for programme and control groups  Very little difference was found by gender, age or ethnic origin  Concluded that the control group is well suited to providing a counterfactual for the programme group

34. External Validity  Pilot Jobcentres account for small proportion of all new JSA claims across the country  Gender, age & ethnicity of new claimants in pilot areas different to country as a whole  Some difference in local unemployment rates  Weighted results to take account of differences

35. Results 13 week excusal pilot

36. % still claiming after x weeks

37. Length of Claim

38. Results  Average length of claim is 5.9 days greater in programme group than in control group  Weighting the results to be representative nationally suggests an increase in 6.1 days in average length of claim  No difference in the proportions who moved into work

39. Results (cont) 4 reasons for difference in length of claim:  Some people take longer to find work  Some people take longer to tell us they have found work  Some control group customers fail to attend and have to start a new claim  Some people fail to sign off for other reasons

40. Results (cont)  Work study provided estimates of savings  Extra benefits paid as a result of increase in average length of claim exceed savings  Qualitative evaluation suggested that the pilot was implemented well  Customers were happy not to have to attend every fortnight

41. More information  DWP Research Report 300: The Qualitative Evaluation of the JSA Intervention Regime Pilots  DWP Research Report 382: Jobseeker’s Allowance Intervention Pilots Quantitative Evaluation  Available on DWP Website: http://www.dwp.gov.uk/asd/asd5/rrs-index.asp http://www.dwp.gov.uk/asd/asd5/rrs-index.asp

42. Job Retention and Rehabilitation Pilot Lessons learnt in running an RCT James Holland

43. Structure 1.Background to JRRP 2.Results of the trial 3.Hypotheses 4.Conclusions: Importance of complementary methodologies

44. Design  4 - way trial  To test the effectiveness of a person centred case management approach and increased range of treatments in helping people retain work –Health care focused –Workplace focused –Combined health care and/or workplace focused –Control group  Four service providers in six parts of the country  Participants were people off work sick and unlikely to return to work without help

45. Routes through the trial Project Marketing Approach Contact Centre Contact Centre Explanation Eligibility Screening Decline Ineligible Screened out Screened in: Randomisation Health 25% Work 25%Combined 25% Control 25% Providers make contact Written Consent Assessment and Intervention Return to work Out of work Surveys

46. Evaluation Design  Impact and process evaluation  Cost benefit Analysis  Components of the evaluation –Survey of those screened out and the control group –Outcome survey –Panel study –Focus studies –Database of contacts and treatments –Costs exercise

47. Impact Measures  Primary impact measure – 13 week return to work  Secondary impact measures –Health –Household income  Costs and benefits  Operation of JRRP as a RCT

48. Results 13 Week Return to Work Intervention group% Health43.5 Workplace45.1 Combined44.4 All interventions44.4 Control44.7

49. Results 6 Week Return to Work Intervention group% Health55.7 Workplace56.4 Combined56.5 All interventions56.2 Control53.0

50. Results 2 Week Return to Work Intervention group% Health61.5 Workplace61.4 Combined62.1 All interventions61.7 Control59.3

51. Results  Positive impact among those off work because of an injury  Negative impact among those off work because of a mental health condition  Unaffected return to work rates for those with other health conditions  Positive impact on health, particularly mild depression  A RCT can work in a voluntary labour market setting

52. Hypotheses for findings 1.The interventions were too weak 2.The interventions were delivered in an unhelpful way 3.Too many external barriers 4.Withdrawal rate too high 5.The self-selecting participants were the ‘wrong group’

53. Missing evidence  Evidence of problems mostly drawn from qual research  This generates hypotheses/explanations but does not allow for quantification  Biggest gap is (quant) understanding of behaviour of control group  In retrospect, needed data on self-motivation and better understanding of participant/provider interaction

54. Thoughts on how to do it better  Set out possible scenarios at start  Early qual research on behaviours of participants and control group  Early impact estimates so that later research can be adapted

55. Conclusions  RCTs are the gold standard programme evaluation  But a number of problems, practically and methodologically  Need supplementing with a good quality process evaluation

56. Contact Details James Holland Disability and Work Division, DWP James.holland1@dwp.gsi.gov.uk 0114 209 8280 Reports available via www.dwp.gsi.gov.uk/asd/asd5