1. Justice Data Lab: Facing the Third Sector
How can we develop the capacity of third sector organisations to engage with data?
Scottish Universities Insight Institute
Georgina Eaton & Tillie Paul, Ministry of Justice
4th February 2015
Introduce myself and Tillie – work at MoJ in JDL team

2. What will be covered Aims and history of the Justice Data Lab
How does the Justice Data Lab work and how can it be accessed?
Findings to date
Receptions of the service
Challenges and successes
What we are going to talk about in this presentation- Tillie is going to talk about the aims and give a brief history of the Justice Data Lab, explain to you how it works and tell you about our findings to date. I’m going to show you some specific examples of our findings, tell you how our service has been received and tell you about the challenges and successes.
Happy to answer questions as we go through this presentation but there will be time at the end to ask questions as well.

3. Aim of the Justice Data Lab
Launched in April 2013
..to improve the evidence base on successful rehabilitation..
..by giving organisations working with offenders secure and legal access to aggregate re-offending data
..enabling them to better assess the impact of their work on re-offending
What Data Lab is
Service initially set up as a one year pilot in April 2013…
Been working ever since to build this evidence

4. Why are we doing the Justice Data Lab?
In 2012 we identified that charitable organisations in particular found it difficult to access re-offending data on their clients…
… this meant that they could not understand how effective their services were at rehabilitating offenders…
… and they were therefore unable to understand how their services could be improved, or have the evidence for further funding
It soon became clear that there was intense interest in this initiative from both public and private sector organisations too
Surveys showed that the majority of people rarely gained access to re-offending data, with 46% finding it difficult when they did
Showing impact of the work they do we know is a difficulty faced by charities working across all sectors, not just Justice

5. Project timeline Spring 2014:
Pilot extended for further year and announcement of improvements to the service
December 2011:
NPC approach MoJ about Data Lab idea
December 2012: Announcement that Justice Data Lab would be piloted
2012
2013
2014
2012: Feasibility
Justice Data Lab Pilot
April 2013:
One year pilot goes live
October 2013:
First publication of Justice Data Lab products
January 2012:
Ministerial Approval to look into the feasibility of the Justice Data Lab
NPC – New Philanthropy Capital. Think tank organisation that works to support evaluation in the Voluntary, Charity Sector (VCS)
NPC approached MoJ end of reasonably clear idea about the problems facing the third sector in terms of evaluating the impact of programmes aimed at offenders / ex-offenders – huge problem having access to re-offending data- explore with us how we can make it easier
Our Chief Stato / Director of ASD really liked the ideas that NPC came with, partly because of the wins to do with transparency agenda (making data open and accessible to users) – and quickly got ministerial approval for the feasibility of looking into a Data Lab- engagement with organisations
So summer 2012 – series of roundtable events with third sector, private and public sector organisations, academics – developed a much better idea of the scope and demand for such a service- there was clear demand and that it was feasible for users to access it.
December 2012 – ministerial agreement to proceed with the service that we have today and an announcement was made
Since launching – continued engagement with third sector – through expert panel and networking (events, presentations, general user engagement) – we’re gaining increasing knowledge of how the third sector works, their challenges etc
Pilot extended for another year in April 2014 with an announcement of improvements to the service

6. How does the Justice Data Lab work?
Provider organisation
Individual level data
sent to MoJ
Aggregate data return
MoJ
Orgs. send as much data as possible (will be previously collected)
We match to our data sources and analyse
Return results in standardised report
Publish findings monthly
Analysis
and Matching

7. Accessing the Justice Data Lab service
Data upload template requirements:
Sample size must be at least 60 needed for robust analysis
Matching variables including:
full name
date of birth
gender
intervention start date
Thorough information on the service they provide
Send securely (CJSM or gsi account)
Data Requirements:
Sample size must be at least 60 needed for robust analysis – worked with between 2002 and 2012 (2012 latest re-off data available to us)
Matching variables including: full name, date of birth, gender, intervention start date
Basic data useful for all charities to collect/hold
Thorough information about the intervention
When collecting info. should always think about purpose of data and the aims of the work- will the data be able to provide you with answers on the impact of the work
Data Protection and Security in place to ensure provider data is protected
Sensitive data must be handled appropriately on both ends- if asking for consent need to be upfront about what the data will be used for, for example

8. What does the Justice Data Lab not do?
X We won’t disclose individual level data
X We currently only give re-offending related outcomes
X There are ethical and practical considerations for assessing interventions aimed at certain groups of offenders (e.g. sex offenders, vulnerable persons) that mean that a Justice Data Lab analysis is not appropriate
Bearing in mind privacy restrictions
Sensitive data/issues – considering appropriateness of different analysis always important

9. Processing and matching the data
Match to Police National Computer for demographics, criminal histories
Find correct sentence and re-offending follow-up period in MoJ administrative datasets
Link to employment and benefits data - Data share with DWP/HMRC
Create matched control group of similar offenders who have not had the intervention from MoJ administrative datasets
The PNC contains details and criminal histories for all recordable offences committed in England and Wales
We have a data sharing agreement with the Department for Work and Pensions, who are able to supply us with employment and benefits data
- Example of where linking data is helpful and makes best possible use of data- doesn’t necessarily have to be on a large scale just linking from various sources to maximise use of data

10. Processing and matching the data
Propensity Score Matching to match individuals in the two groups to each other
Test differences in re-offending for the groups
Statistical significance testing to determine whether there is a true difference between the groups
Actual stats comes in- might be surprised but takes the least time
Statistical significance important- not always initially considered but for more advanced, complex analyses is greatly important for evidencing
Analysis doesn’t have to be of this complexity- what resource and knowledge allows for

11. What is provided to Justice Data Lab users?
The best estimates for the one year proven re-offending rate for offenders who received an intervention from WYJS, and a matched control group.
One year re-offending rate
Frequency of re-offending
Time to re-offending
Information on characteristics of both the treatment and control groups
standard report designed to be accessible to non-statistical audiences- this will be published
Key re-offending rates for both groups plus a statistical comparison of the results (deciding whether result is stat. sig or not)
We provide blunt measures- some might want softer, more descriptive measures or may only have the resource for this
Characteristics- WHO actually working with- important to understand the demographic, may look different to how you would think
Explanation of how to interpret the results and graphical display- helping to present and explain results in different ways is always good
Current work on further metrics
When analysing/commissioning work want to think about WHAT want to evidence, how realistic it will be and how this will be shown in the measures

13. Cumulative findings to date
Of the 121 reports published so far:
27 reports indicated statistically significant reductions in re-offending on the one year proven re-offending rate
87 reports indicated insufficient evidence to draw a conclusion about the effect on the one year proven re-offending rate
Of these 87, 11 reports detail statistically significant reductions in the frequency of re-offending
7 reports indicated a statistically significant increase in re-offending on the one year proven re-offending rate
So far the Justice Data Lab has received 152 requests for re-offending information
121 reports have been published previously
17 requests could not be processed as the minimum criteria for analyses through the Data Lab had not been met- lack of data
1 request was withdrawn by the submitting organisation
The remaining requests are currently in progress and will be published in future monthly releases of these statistics

15. Drive for developing capacity of all sectors to engage with data- not just third sector alone

16. Specific reports – Blue Sky
The best estimates for the one year proven re-offending rate for offenders who received an intervention from Blue Sky, and a matched control group.
Short term, full-time employment contracts. Aims to move them into onward full-time employment elsewhere.
The one year proven re-offending rate for 72 offenders employed by Blue Sky was 31%, compared with 43% for a matched control group of offenders with similar characteristics.
A reduction in re-offending between 1 and 23 percentage points.
One year proven re-offending rate
One of our first analyses that was published in October 2013.
Blue Sky’s intervention provides short term, full time employment contracts to offenders in the community. This analysis looked at 72 offenders that received the intervention after release from prison. Looked at those they worked with up to 6 months after their release.
Statistically significant
This analysis indicates that individuals receiving short-term, full time employment with Blue Sky within 6 months of leaving custody experienced a reduction in re-offending between 1 and 23 percentage points.
One of our best results yet but at the time they were reluctant to publish as they thought they reduced re-offending by a lot more than what our results showed. Sometimes orgs don’t always understand impact and have unrealistic views and publish these wild claims on their website without getting real evidence.

17. Specific reports – Prisoners Education Trust
Prisoners Education Trust submitted data relating to offenders who had a grant for Open University, distance learning courses, or Art and Hobby materials between 2002 and We carried out one overall analysis and four sub-analyses, the results are in the table below.
PET – provide grants to offenders in prison throughout England for a OU, distance learning courses or to purchase arts and hobby materials.
Separate analyses for different course types as well as overall – all results shown in table.
About 50% of participants were know to have received one of four indicated grant types for (i) courses currently accredited1 and (ii) unaccredited1 – these were funded by PET through grants from the Department for Business, Innovation and Skills, (iii) Open University courses and (iiii) Arts and Hobby grants.
The overall does not sum to the individual treatment groups as the course types were unknown for the rest of the group
All show reduction in re-offending and frequency of re-offending apart from accredited courses.

18. Specific reports – Prisoners Education Trust
“We have already changed our approach to funding different course types as a result of the findings.”
“Our charity and our funders know that learning in prison works – but now we have the evidence to prove it with this robust, hard-edged report carried out by MoJ statisticians.”
Rod Clark, Chief Executive of Prisoners Education Trust
These results made it onto the BBC News website – we were really excited about this
PET used our results to change the policy of awarding grants – some quotes the Chief Executive of PET:
Additionally PET now have evidence to show how effective they are at reducing re-offending which will help them when talking to funders and commissioners.

19. Reception of reports
Survey issued to organisations who have used the Justice Data Lab service showed that:
The expectations around the Data Lab had been met, although our customer service could be variable
The Justice Data Lab is a useful service for the third sector, helping to provide information on re-offending and impact
More information on outcomes (severity, re-incarceration rates, employment and benefits) would be helpful
Results had been used to understand / demonstrate impact internally and externally
We send a short survey to all our customers following the release of our report.
The expectations around the Data Lab have been met to what has been set out in the User Journey document. Customer service variable – at the start we may not have got back to customers swiftly due to resources within the team and learning how to carry out the analysis. Have improved on this by communicating with organisations more and trying to find out more about the service they provide.
Useful service for VCS as opportunity to find out more information about the people they have been working with and their impact on re-offending.
We’ve had feedback that organisations would like a more flexible service by providing more outcomes such as severity, re-incarceration rates, employments and benefits that would be helpful. We have made a commitment to bring improvements to the service throughout the last year and are doing some feasibility work on this.
The majority of organisations that responded to this survey had plans to alert external organisations as to their results and impact. Results have made organisations think about delivery of their services, especially those with negative results.

20. Challenges 40% ↓ re-offending
First time the sector has had transparency about their effectiveness
Claims about success vs. evidence
Understanding technical aspects / statistical literacy
Availability and quality of internal and external data
Resources for MoJ and organisations
Legalities for organisations
40% ↓ re-offending
At first we were worried we’d be like the McDonalds salad (which never worked out as not much demand for it)- the third sector asked for a service like Data Lab but there wasn’t as much demand for it as originally thought once set up. However, we have had continual demand unlike the McDonald’s salad which fizzled out. There is mixed opinion about what we do but those who do use our service agree we’re useful.
Wild claims about their impact – lack of understanding about how impossible their claims are- e.g. one org. we analysed claimed previously to reduce re-offending by 40-50%, in reality was actually 12% for reasonably small sample size- this is an impressive impact however org. reluctant at first to accept this and publish.
Technical competence- more aware of the need for them to collect data, re-offending data held administratively, and fact that it’s retrospective- we can’t analyse work they just carried out in nervous at first as come from such a statistical perspective but been received better than expected- saying issue is that re-offending data takes at least 18 months to follow up- orgs. find it difficult to understand this we still have some challenging queries. Explaining statistical concepts to people with no background of using statistics. As we use blunt measures this is another challenge as we are restricted in what we can give back to organisations but its good to find out what’s possible if you’re looking to have some analysis on your data.
Availability and quality of internal and external data– Quality of our data can be a challenge. Some organisations don’t collect anything on the individuals they work with or very limited information so this has prevented some orgs from accessing our service as they don’t have all of the appropriate variables we need.
Resources for MoJ and organisations – IT resources, staffing resources (MoJ) and orgs may not have time to keep data records or to send data to us.
Some organisations unsure where they stand on legal issues e.g. Data Protection so have been reluctant to access the service because of this.

21. Successes Positive feedback from users
Organisations using results to change their services
Continual demand for service
Engaging with users
Service developments
Passing on experience to others
Positive feedback from survey
Use results to change funding for services e.g. PET
Still receiving requests for analysis
Engaging with users by meeting with them face to face, attending events and keeping in contact with them throughout the process from data submission to publishing the final report
We have managed to deliver the majority of developments that we agreed to make over the past year from the extension of the service in April 2014.
We have also been able to pass on our experience to others – working with Third Sector organisations and talking to other gov. dpts about setting up their own Data Labs

22. Award Winners! Government Finance Insight Award 2014
“Judges were impressed by the use of data in an innovative way, and the presentation of complex data in a way that is understood by all.”
Royal Statistical Society Excellence in Official Statistics Award 2014
“Judges were impressed by the use of statistical techniques to assess success (or failure) in a critical area and by the exceptionally close way MoJ statisticians had worked with their users, mainly non-statisticians.”
The Justice Data Lab team have won 2 awards in the last year – Government Finance Insight Award 2014 and Royal Statistical Society Excellence in Official Statistics award. It was a great honour to get appreciation for our work and it’s great that our efforts in engaging with our users has been recognised.
Government Finance Award:
Using data in an innovative way for key decision makers
Presentation of complex data in a way that is understood by all
Data systems have been used to support the work of the wider departmental family
RSS Award:
- Judges were impressed by the use of statistical techniques to assess success (or failure) in a critical area and by the exceptionally close way MoJ statisticians had worked with their users, mainly non-statisticians

23. How can we develop the capacity of third sector organisations to engage with data?
Data collection
Central data system – Charity Log?
Data manager/analyst
Access to data through government/local authorities
Open data
Asking for help – Academics, Government Statisticians and Think Tanks
Don’t be scared of data!!
TP:
Sharing best practice, encouraging use of data, prioritising, understanding/stressing importance, becoming the norm
Collect data- bearing in mind purpose of intervention and how would want to analyse, want would want out of analysis
If possible central system- – cloud based database for charities- may be others out there- or as simple as just central spreadsheet that is accessed and updated if that’s all resource allows for
Data manager, researcher etc- make part of someone’s job role- priority
GE:
You can get access to open data online – can be hard to find it but possible
You don’t have to engage with data by yourself – you can always ask for help from experts such as academics and Government Statisticians. Currently a Government Statistical Service voluntary placement schemes where government. statisticians can help your organisation with data collection or analysis. Additionally you can discuss data with other organisations through networking events like today or through forums.
Think tank - NPC exists to make charities and funders more successful in achieving their missions. Through rigorous analysis, practical advice and innovative thinking, we make their money and energy go further. Our consulting services help charities and funders find solutions to the challenges they face, whether they are trying to work more effectively, exploring new ways of working, or setting out to prove their worth.
All our advice is backed up by ten years of research, and draws on our think tank work to transform the charity sector.
On a final note, don’t be scared of data – it’s only there to help you. I hope our presentation today will inspire you to engage with data in the future.

24. Contact Details Email: justice.datalab@justice.gsi.gov.uk
Accessing the Justice Data Lab service:
Published reports:
Government Statistical Service:
Contact details and websites
Thank you for listening and we open up to questions

25. Propensity Score Matching Scale of propensity score1 -1
Key
Control offenders are matched to treatment offenders if the control offender’s propensity score are within a specified range away from the treatment offender’s propensity score.
Treatment offenders
Control offenders
Matched offenders