Justice Data Lab: Facing the Third Sector How can we develop the capacity of third sector organisations to engage with data? Scottish Universities Insight Institute Georgina Eaton & Tillie Paul, Ministry of Justice 4 th February 2015

What will be covered Aims and history of the Justice Data Lab How does the Justice Data Lab work and how can it be accessed? Findings to date Receptions of the service Challenges and successes

Aim of the Justice Data Lab Launched in April to improve the evidence base on successful rehabilitation....by giving organisations working with offenders secure and legal access to aggregate re-offending data..enabling them to better assess the impact of their work on re-offending

Why are we doing the Justice Data Lab? In 2012 we identified that charitable organisations in particular found it difficult to access re-offending data on their clients… … this meant that they could not understand how effective their services were at rehabilitating offenders… … and they were therefore unable to understand how their services could be improved, or have the evidence for further funding It soon became clear that there was intense interest in this initiative from both public and private sector organisations too

Project timeline April 2013: One year pilot goes live Justice Data Lab Pilot December 2012: Announcement that Justice Data Lab would be piloted October 2013: First publication of Justice Data Lab products January 2012: Ministerial Approval to look into the feasibility of the Justice Data Lab 2012: Feasibility Spring 2014: Pilot extended for further year and announcement of improvements to the service December 2011: NPC approach MoJ about Data Lab idea 2012

How does the Justice Data Lab work? Individual level data sent to MoJ Provider organisation MoJ Analysis and Matching Aggregate data return

Accessing the Justice Data Lab service Data upload template requirements: Sample size must be at least 60 needed for robust analysis Matching variables including: – full name – date of birth – gender – intervention start date Thorough information on the service they provide Send securely (CJSM or gsi account)

What does the Justice Data Lab not do? X We won’t disclose individual level data X We currently only give re-offending related outcomes X There are ethical and practical considerations for assessing interventions aimed at certain groups of offenders (e.g. sex offenders, vulnerable persons) that mean that a Justice Data Lab analysis is not appropriate

Processing and matching the data Match to Police National Computer for demographics, criminal histories Find correct sentence and re-offending follow-up period in MoJ administrative datasets Link to employment and benefits data - Data share with DWP/HMRC Create matched control group of similar offenders who have not had the intervention from MoJ administrative datasets

Processing and matching the data Propensity Score Matching to match individuals in the two groups to each other Test differences in re-offending for the groups Statistical significance testing to determine whether there is a true difference between the groups

What is provided to Justice Data Lab users? One year re-offending rate Frequency of re-offending Time to re-offending Information on characteristics of both the treatment and control groups The best estimates for the one year proven re-offending rate for offenders who received an intervention from WYJS, and a matched control group.

Cumulative findings to date Of the 121 reports published so far: 27 reports indicated statistically significant reductions in re-offending on the one year proven re-offending rate 87 reports indicated insufficient evidence to draw a conclusion about the effect on the one year proven re-offending rate Of these 87, 11 reports detail statistically significant reductions in the frequency of re-offending 7 reports indicated a statistically significant increase in re-offending on the one year proven re-offending rate

Specific reports – Blue Sky Short term, full-time employment contracts. Aims to move them into onward full-time employment elsewhere. The one year proven re-offending rate for 72 offenders employed by Blue Sky was 31%, compared with 43% for a matched control group of offenders with similar characteristics. A reduction in re-offending between 1 and 23 percentage points. One year proven re-offending rate The best estimates for the one year proven re-offending rate for offenders who received an intervention from Blue Sky, and a matched control group.

Specific reports – Prisoners Education Trust Prisoners Education Trust submitted data relating to offenders who had a grant for Open University, distance learning courses, or Art and Hobby materials between 2002 and We carried out one overall analysis and four sub-analyses, the results are in the table below.

Specific reports – Prisoners Education Trust “We have already changed our approach to funding different course types as a result of the findings.” “Our charity and our funders know that learning in prison works – but now we have the evidence to prove it with this robust, hard-edged report carried out by MoJ statisticians.” Rod Clark, Chief Executive of Prisoners Education Trust

Reception of reports Survey issued to organisations who have used the Justice Data Lab service showed that: The expectations around the Data Lab had been met, although our customer service could be variable The Justice Data Lab is a useful service for the third sector, helping to provide information on re-offending and impact More information on outcomes (severity, re-incarceration rates, employment and benefits) would be helpful Results had been used to understand / demonstrate impact internally and externally

Challenges First time the sector has had transparency about their effectiveness Claims about success vs. evidence Understanding technical aspects / statistical literacy Availability and quality of internal and external data Resources for MoJ and organisations Legalities for organisations 40% ↓ re-offending

Successes Positive feedback from users Organisations using results to change their services Continual demand for service Engaging with users Service developments Passing on experience to others

Award Winners! Government Finance Insight Award 2014 “Judges were impressed by the use of data in an innovative way, and the presentation of complex data in a way that is understood by all.” Royal Statistical Society Excellence in Official Statistics Award 2014 “Judges were impressed by the use of statistical techniques to assess success (or failure) in a critical area and by the exceptionally close way MoJ statisticians had worked with their users, mainly non- statisticians.”

How can we develop the capacity of third sector organisations to engage with data? Data collection Central data system – Charity Log? Data manager/analyst Access to data through government/local authorities Open data Asking for help – Academics, Government Statisticians and Think Tanks Don’t be scared of data!!

Contact Details Accessing the Justice Data Lab service: Published reports: statistics Government Statistical Service:

Propensity Score Matching Scale of propensity score 01 Control offenders Treatment offenders Key Matched offenders Control offenders are matched to treatment offenders if the control offender’s propensity score are within a specified range away from the treatment offender’s propensity score.