1. Evaluating methodological quality in the criminal justice system literature

2. Dr Amanda Perry
Centre for Criminal Justice Economics and Psychology, University of York.

3. Overview of the session:
Background to RCT in criminal justice
Quality assessment
The Maryland Scientific Methods Scale (SMS)
Problems associated with the SMS
Where next…….

4. Background Few RCT conducted in the UK
Systematic review revealed 125 conducted between
(Farrington & Welsh, 2005)
Concurrent findings from UK systematic review between
(Perry, McDougall & Farrington, 2005)
8 before 1982 in the UK – EXCLUDED STUDIES OF LESS THAN 100 PARTICIPANTS IN TOTAL
Perry et al found 2 trials in the UK - consisting of 14 intervention 13 in no treatment group
Total of 27 participants (Baldwin et al.,
Second trial contained 60 individuals – 50% non-completion rate – affecting validity of the overall trial.

5. Resistance to RCT… Historical resistance Ethical and moral reasons
Practical difficulties
BUT………... New era
Campbell Collaboration (2000)
Department of Health/Home Office (2000)

6. New age of RCT…
Feasibility studies
(Farrington & Joliffe, 2002)
(Farrington et al., 2002)
Matrix (2006) – Criteria for assessing feasibility
Current RCTs
Evaluation of Cognitive behavioural skills programmes (McDougall, Bowles, Perry & Clarbour, ongoing).
Evaluation of Restorative Justice Programmes (Strang & Sherman, 2006)

7. Quality assessment…
Reviews of quality assessment tools: The medical field
Moher et al. (1995) identified 25 scales devised up to 1993.
Juni et al (1999) compared 25 scales for purpose of inter-rater reliability (r=.72).

8. Quality assessment… Reviews of quality assessment tools: The
Social Sciences
Gibbs (1989) – social work
STROBE: STrengthening the Reporting of OBservational studies in Epidemiology
Maryland Scientific Methods Scale – Criminal Justice (Sherman et al., 2002)

9. Quality assessment… The Maryland Scientific Methods Scale (SMS)
(Sherman et al., 2002, based on work by Cook & Campbell, 1979)
Purpose of the SMS:
Simple (measuring internal validity)
To provide policy makers with information about the evidence
Aim to classify all programmes into 1 of 4 categories
The main aim of the SMS was to communicate to scholars policy makers and practitioners the simplest possible way that studies evaluating the effects of criminological interventions differ in methodological quality

10. The SMS quality assessment..
Five point scale:
Rating 1-5
Rating 1: Correlational study
Rating 2: Pre and post test study
Rating 3: Observational cohort with comparable group
Rating 4: Quasi-experimental/controlled trial
Rating 5: Randomised controlled trial
                               
5 point scale rather than a summation of scores.
Correlational study
[This design fails to rule out many threats to internal validity and also fails to establish causal order]
Level 2
This design establishes causal order but fails to rule out many threats to internal validity level 1 and 2 are considered inadequate by cook and Campbell
Level 3 minimum interpretable design that is adequate for drawing conclusions about what works – the main problems centre of selection effects and regression to the mean
Level 4
Better statistical control of extra influences on the outcome and deals with selection and regression threats more adequately.

11. The SMS quality assessment..
Statistical conclusion validity
Was the statistical analysis appropriate?
Did the study have low statistical power to detect effects because of small samples?
Was there a low response rate or differential attrition?
Construct validity
What was the reliability and validity of measurement of the outcome?
                               
If there was a problem on what of these issues or more a RCT could be down graded to a level 4 – for eg a RCT with serious attrition problems might receive a rating of level 4 than 5

12. Evidence and use of the SMS:
What Works: Evidence from 2 or more studies reporting positive results scoring 3 or above on the SMS showing statistical significance and desirable effects and the preponderance of all available evidence showing effectiveness.
What Doesn’t Work: Evidence from 2 or more studies reporting negative results scoring 3 or above on the SMS…

13. Evidence and use of the SMS:
What ‘s Promising: Reporting evidence from 1 study reporting positive results scoring 3 or above on the SMS….
What Unknown: Evidence from 1 study with a negative or inconclusive result scoring 3 or above on the SMS…..

14. Use of the SMS: ‘What works – an example’ CCTV in car parks
(evidence from Welsh and Farrington, 2002)
Street lighting
(evidence from Painter & Farrington, 1997; 1999b;2001b; Farrington & Welsh, 2002)
Burglary reduction schemes
(evidence from Ekblom, 1996a; 1996b)

15. Limitations of the SMS…
Not fully assessing all threats to the validity of a study
Categorising study designs
Does not take into consideration at What cost?

16. Limitations of the SMS…
Is designed to apply equally to all experimental units
Does not embrace all study designs
Method of drawing conclusions on what works based on statistical significance rather than effect size

17. Improving the SMS…. Farrington (2003) Based on five key criteria:
Internal validity
Descriptive validity
Statistical conclusion/validity
Construct validity
External validity

18. Information for policy makers
Improving the SMS …
Information for policy makers
3 five point scales
Design
(internal validity)
Execution
(construct validity/statistical conclusion validity /sampling elements of external validity)
Reporting of the trial

19. Beyond the SMS….. Assessing the cost of an intervention
Adequate and standardised follow-up periods for outcome measures
(e.g., reconviction rates)
Encouraging journal editors to use a new scale/standard similar to CONSORT statement

20. Beyond the SMS…..
Development of a specific quality assessment tool evaluating RCT in the criminal justice
To incorporate all elements of validity
To include an assessment of cost/cost effectiveness/cost-benefits of an intervention
To include guidance on adequate and standardised follow-up periods for outcome measures
(e.g., reconviction rates)

21. SUMMARY RCT rarely used in UK criminal justice system
Use of the SMS as quality measure
Room for improvement
Development of a new scale….

22. Centre for Criminal Justice Economics and Psychology
CONTACT:
Amanda Perry
Centre for Criminal Justice Economics and Psychology
University of York