Using propensity score matching to understand what works in reducing re-offending GSS Methodology Symposium Sarah French & Aidan Mews, Ministry of Justice 1st July2015

What will be covered What is propensity score matching (PSM)? History of using PSM in MoJ Justice Data Lab aims, history and how it works Methodology Key outcomes Developments

3 We find individuals who were not treated, but who were very similar at the point treatment started. TREATMENT CONTROL ? Observational Studies

4 What Propensity Scores Are, and Why They Are Important Propensity Score = Pr (Treated | background info) Mimics an RCT in that the treatment is purely random for individuals with similar values of the background variables Groups of subjects with similar propensity scores can be expected to have similar values of all of the background information, in the aggregate.

Propensity Score Matching Scale of propensity score 0 1 Control offenders Treatment offenders Key Matched offenders Control offenders are matched to treatment offenders if the control offender’s propensity score are within a specified range away from the treatment offender’s propensity score.

History of propensity score matching in MoJ PSM first used in 2010 Compendium of re-offending to compare the effectiveness of two sentences Since then used:.. for further comparisons of sentences & sentence requirements.. to look at the impact of various interventions to reduce re-offending.. to evaluate the relationship between employment and re-offending

Aim of the Justice Data Lab Launched in April to improve the evidence base on successful rehabilitation....by giving organisations working with offenders secure and legal access to aggregate re-offending data..enabling them to better assess the impact of their work on re-offending

Why do we have the Justice Data Lab? In 2012 we identified that charitable organisations in particular found it difficult to access re-offending data on their clients… … this meant that they could not understand how effective their services were at rehabilitating offenders… … and they were therefore unable to understand how their services could be improved, or have the evidence for further funding It soon became clear that there was intense interest in this initiative from both public and private sector organisations too

How does the Justice Data Lab work? Individual level data sent securely to MoJ Provider organisation MoJ Analysis and Matching Aggregate data return

Process overview Data upload template (60 minimum) Match to MoJ/DWP data to get treatment group –find sentences for offenders from PNC –match to DWP/HMRC data Matched control group via PSM (30 minimum) –Many to one matching, radius matching with replacement Assess quality of matched control group by analysing standardised mean differences Significance testing on re-offending measures compared between treatment and control groups

Key characteristics of ‘research reports’ Binary re-offending rate (overall, resulting in custody, severe, 1 year, 2 year) Frequency of re-offending Survival (or death) analysis Representativeness of matched treatment group Quality of matching Sensitivity analysis Caveats (e.g. unobserved characteristics not included in matching process)

‘Death’ analysis Re-offending rate

What is provided to Justice Data Lab users? One year re-offending rate Frequency of re-offending Time to re-offending Information on characteristics of both the treatment and control groups The best estimates for the one year proven re-offending rate for offenders who received an intervention from WYJS, and a matched control group.

Key Justice Data Lab outcomes Of the 125 reports published so far: 29 reports indicated statistically significant reductions in re-offending on the one year proven re-offending rate 89 reports indicated insufficient evidence to draw a conclusion about the effect on the one year proven re-offending rate Of these 89, 11 reports detail statistically significant reductions in the frequency of re-offending 7 reports indicated a statistically significant increase in re-offending on the one year proven re-offending rate

Developments in PSM use within MoJ Move away from 1-1 to 1-many matching Use of more information in matching process More thinking about methodological improvements to reduce bias Wider range of outcome measures Post matching regression analysis

Justice Data Lab Developments in Progress Providing additional information on the re-offending outcomes, such as severity of re-offending Enhancing understanding of the criminogenic needs of individuals – through the use of Offender Assessment (OASys) data Understanding more about individuals that are not matched in an analysis Official Statistics methodology review Supporting other government departments on potential Data Labs

Contact Details Accessing the Justice Data Lab service: data-lab Published reports: pilot-statistics