1 The effect of regression towards the mean in assessing crime reduction interventions using non-randomised trials Campbell Collaboration Colloquium London May 2007 Paul Marchant Innovation North, Leeds Metropolitan University Paul Baxter Department of Statistics, University of Leeds

2 The basic point If non-RCTs are used, we need a sound understanding of the system being studied and a quantitative model to work out what is lost and what the effect is. If non-RCTs are used, we need a sound understanding of the system being studied and a quantitative model to work out what is lost and what the effect is. The effects being sought may be small so impact of small systematic errors can be important. The effects being sought may be small so impact of small systematic errors can be important. Need rigorous scientific evaluation of the implementation of policy. Need rigorous scientific evaluation of the implementation of policy.

3 Scientific Methods Scale 5 point crime research ‘Scientific Methods Scale’ which orders trial designs. (RCT is the top ) 5 point crime research ‘Scientific Methods Scale’ which orders trial designs. (RCT is the top ) While the ordering may be fine there seems little formal indication of what is lost by using ‘a 4’ rather than ‘a 5’. While the ordering may be fine there seems little formal indication of what is lost by using ‘a 4’ rather than ‘a 5’. The potential exists to draw false inference. The potential exists to draw false inference.

4 The Randomised Controlled Trial (A truly marvellous scientific invention) Note to avoid bias: Register trial / protocol. Register trial / protocol. Allocation is best made tamper-proof. Allocation is best made tamper-proof. (e.g. use ‘concealment’) Use multiple blinding of: Use multiple blinding of: patients, patients, physicians, physicians, assessors, assessors, analysts … analysts … Population Take Sample Randomise to 2 groups Old Treatment Compare outcomes (averages) recognising that these are sample results and subject to sampling variation when applying back to the population New Treatment

5 Counts of those cured and not cured under the two treatments CuredNot Cured New Treatmentab Control (Standard treatment) cd By comparing the ratios of numbers ‘cured’ to ‘not cured’ in the 2 arms of the trial, the Cross Product Ratio (CPR)= (ad)/(cb), it is possible to tell if the new treatment is better.

6 Confidence Interval for estimate of treatment effect However there is sampling variability, because we don’t study everybody of interest; just our random sample. There is uncertainty in the estimate. However there is sampling variability, because we don’t study everybody of interest; just our random sample. There is uncertainty in the estimate. Need to know how to calculate the CI appropriately. This can be done under assumptions, which seem reasonable for the case of a clinical RCT and leads to a simple expression for the approximate CI. Need to know how to calculate the CI appropriately. This can be done under assumptions, which seem reasonable for the case of a clinical RCT and leads to a simple expression for the approximate CI. So should be able to obtain a valid estimate of treatment effect,….. providing there is no trouble from biases (e.g. differential drop out, publication bias …). So should be able to obtain a valid estimate of treatment effect,….. providing there is no trouble from biases (e.g. differential drop out, publication bias …).

7 Crime counts before and after in two areas one gets a Crime Reduction Intervention CRI (e.g. 3 on the Methods Scale) A similar table results. But this is not the same as the RCT set up as: 1 Not randomised, so no statistical equivalence exists at the start. 2 The unit is area, rather than crime event. BeforeAfter Treatment Area (Intervention is introduced between the 2 periods ) ab Comparison Area (Nothing is changed) cd

8 This creates 2 problems 1 The areas are not statistically equivalent at the outset. 1 The areas are not statistically equivalent at the outset. 2 The natural variation will be different (overdispersed) from the simple expression appropriate in an individually randomised trial with statistically independent outcomes. 2 The natural variation will be different (overdispersed) from the simple expression appropriate in an individually randomised trial with statistically independent outcomes.

9 The lack of equivalence between areas It is the most crime-prone area that gets the intervention, whereas the relatively crime-free comparison area does not receive anything new. It is the most crime-prone area that gets the intervention, whereas the relatively crime-free comparison area does not receive anything new. Lack of equivalence at the start allows ‘regression towards the mean’ (RTM) to operate. Lack of equivalence at the start allows ‘regression towards the mean’ (RTM) to operate. The name ‘Control Area’ is a misnomer. ‘Comparison Area’ is a better name. The name ‘Control Area’ is a misnomer. ‘Comparison Area’ is a better name.

10 Regression towards the mean First discovered by Francis Galton (1880s) who found that: First discovered by Francis Galton (1880s) who found that: Tall parents tend to have tall offspring but not as tall as themselves (on average); i.e. offspring tend to be less tall than their parents. Tall parents tend to have tall offspring but not as tall as themselves (on average); i.e. offspring tend to be less tall than their parents. Short parents tend to have short offspring but not as short as themselves (on average); i.e. offspring tend to be less short than their parents. Short parents tend to have short offspring but not as short as themselves (on average); i.e. offspring tend to be less short than their parents. This tendency for both to become more average is ‘regression towards the mean’ RTM. This tendency for both to become more average is ‘regression towards the mean’ RTM.

11 Regression towards the mean X The before measurement Y The after measurement Cloud of Data Points Line of Equality Line of mean of Y for a given X (The conditional Mean)

12 RTM needs to be accounted for in making comparisons Imagine a thought experiment for a treatment to reduce the height of the next generation! Imagine a thought experiment for a treatment to reduce the height of the next generation! Problematic if the treatment is given only to tall parents and the results (height of offspring) are compared with those of short parents, who receive placebo, (because of RTM, as in Galton’s discovery). Problematic if the treatment is given only to tall parents and the results (height of offspring) are compared with those of short parents, who receive placebo, (because of RTM, as in Galton’s discovery).

13 Crime reduction assessment RTM will be a problem in assessing crime levels changing from one period to the next. Is any reduction seen due to an intervention? RTM will be a problem in assessing crime levels changing from one period to the next. Is any reduction seen due to an intervention? However what is the bivariate distribution for ‘Before to After’? However what is the bivariate distribution for ‘Before to After’? Log Normal seems a reasonable candidate. Log Normal seems a reasonable candidate.

14 Burglary counts in successive periods: data from Tilley et al.

15 Log Normal Probability Plot for Burglary count data (from Tilley)

16 Estimating the effect of RTM On the basis of a log normal crime distribution it can be shown that if the intervention has no effect, the Expected ( ln CPR ) = (1-ρσ y /σ x ) ln x 1 /x 2 x 1 /x 2 is the crime ratio; σ x, σ y the standard deviations, before and after, on the log scale and ρ the correlation also on the log scale. Var( ln CPR ) = 2 σ 2 (1-ρ 2 ) Var( ln CPR ) = 2 σ y 2 (1-ρ 2 )

17 LnCPR with limits

18 Can work with rates instead (still lognormal) Probability plot of the natural log of burglary rates (burglaries per household) in successive periods, using the same Tilley data Rates will give the same CPR as counts in any one study if the denominator does not change.

19 Log burglary rates (no. per household) in successive periods

20 On the larger geographical scale Have also examined year to year data from Crime and Disorder Partnership CDRP Areas (376 covering all England and Wales). In particular burglary counts and rates. Have also examined year to year data from Crime and Disorder Partnership CDRP Areas (376 covering all England and Wales). In particular burglary counts and rates. Lognormal seems OK for this too (correlation higher than for smaller area data). Lognormal seems OK for this too (correlation higher than for smaller area data).

21 Not just of ‘mathematical’ interest It is claimed that a study of the effect of lighting is not threatened by RTM. Examined using successive years in Police Basic Command Unit Areas (of similar size to CDRPs). CPRs, formed from comparing quintile bands of crime rate, show the effect is a only a few percent. It is claimed that a study of the effect of lighting is not threatened by RTM. Examined using successive years in Police Basic Command Unit Areas (of similar size to CDRPs). CPRs, formed from comparing quintile bands of crime rate, show the effect is a only a few percent. This value is what our simple lognormal model predicts from the parameters of the data. This value is what our simple lognormal model predicts from the parameters of the data. However the geographical scale of the lighting study is much smaller than the BCU scale. However the geographical scale of the lighting study is much smaller than the BCU scale.

22 Another possible distribution The bivariate lognormal gives a rather neat solution and seems to be in accord with data. The bivariate lognormal gives a rather neat solution and seems to be in accord with data. However a less neat result comes from a bivariate Gamma distribution. (Assuming constant coefficient of variation). However a less neat result comes from a bivariate Gamma distribution. (Assuming constant coefficient of variation).

23 Gamma probability plot to the marginal distributions

24 Disadvantage of the Gamma The expressions for the Expectation and Variance of ln CPR contain the initial values, x 1 and x 2, as a ratio with the mean of the distribution. For example: Expected ln CPR = ln( x 1 /x 2 ) - ln((1+ρ( x 1 /μ x -1)/( 1+ρ( x 2 /μ x -1)) Here ρ is the correlation and μ x the mean (on the original scale) Here ρ is the correlation and μ x the mean (on the original scale) Note: Forthcoming presentation by Paul Baxter to RSS2007 ‘Statistics and public policy-making: hope versus reality’

25 Some conclusions A ‘Methods Scale’ seems to suggest that designs weaker than RCTs might suffice, but there is a loss. RTM is one problem in non-RCT studies. A ‘Methods Scale’ seems to suggest that designs weaker than RCTs might suffice, but there is a loss. RTM is one problem in non-RCT studies. RCTs can be problematic enough. (We need registered trials, published protocols, pre-defined analysis plans, concealment of allocation, blinding etc…..) RCTs can be problematic enough. (We need registered trials, published protocols, pre-defined analysis plans, concealment of allocation, blinding etc…..) Difficult to calculate the effect of RTM because of model choice, outcome choice, appropriate parameter values. (Problem of unobserved confounders in non-RCTs) Difficult to calculate the effect of RTM because of model choice, outcome choice, appropriate parameter values. (Problem of unobserved confounders in non-RCTs) Evaluations of policies once implemented need to be done to a high scientific standard. Evaluations of policies once implemented need to be done to a high scientific standard.

26 References Farrington D.P. and Welsh B.C. (2006) How Important is Regression to the Mean in Area- Based Crime Prevention Research?, Crime Prevention and Community Safety 8 50 Marchant P.R. (2005) What Works? A Critical Note on the Evaluation of Crime Reduction Initiatives, Crime Prevention and Community Safety Tilley N., Pease K., Hough M. and Brown R. (1999) Burglary Prevention: Early Lessons from the Crime Reduction Programme, Crime Reduction Research series Paper1 London Home Office