Could statistical science have caught Harold Shipman earlier? David Spiegelhalter MRC Biostatistics Unit Cambridge BA Festival of Science, September 6th 2005

Shipman background 1974: General Practitioner in Todmorden 1975: Left when convicted of drug offences 1977: Joins group practice in Hyde 1992: Sets up in single-handed practice Jan 1998: Positive report from Tameside audit group: ‘Keep up the good work!’

March 1998: local GP contacts coroner with suspicions July 1998: police contacted concerning a possibly forged will Sept 1998: Shipman arrested. Enquiries made concerning 192 deaths. Jan 2000: convicted of murdering 15 patients 2001: Report from Richard Baker for DoH 2001: Public Inquiry examines 401 deaths 2004: Shipman commits suicide in prison

Shipman Inquiry July 2002: 215 definite victims 45 probable

First comparative analyses by Baker compares - ‘Observed’ deaths: those at home or in his practice and certified by Shipman, with ‘Expected’ deaths: corresponding rates for (small sample of) local GPs.

(NB: Shipman Inquiry total of definite or probable victims: 189 female > 65, 55 male over 65)

Sequential monitoring Accumulates evidence of ‘excess risk’ Simplest method is a cumulative plot of Observed - Expected (O – E) Good intuitive plot But when should we ‘blow the whistle’?

The Bristol Inquiry into excess mortality

But when to sound an alarm?

How soon can we tell if we are ‘off-target’?

Meanwhile, back in 1943 A group of statisticians called SR17 were working in the Ministry of Supply Included Barnard, Lindley, Plackett, Armitage etc Barnard developed the `Sequential probability ratio test’ Simultaneously discovered by Abraham Wald in the US George Barnard later wrote British Standard 3704 for condoms

Sequential probability ratio test (SPRT) Most powerful sequential test between two hypotheses H0 and H1 Based on log(likelihood ratio) LLR = log [ p(data| H1) / p(data| H0) ] Straightforward to adapt to ‘risk-adjustment’ model Horizontal thresholds set by required error rates

Risk-adjusted SPRT for count data Suppose observe O adverse events E are expected under H0 Alternative hypothesis H1: risk ratio r Contribution to log(likelihood ratio) is LLR = O log r - ( r -1) E Suppose we want to detect a doubling of risk (r = 2), then LLR = O log 2 - E i.e. plot 0.69 O – E , rather than O - E

For Bristol data

Setting thresholds = Probability of eventually rejecting H0 when it is true (Type I error) = Probability of eventually rejecting H1 when it is true (Type II error). Upper threshold: a = log [  / ( 1-  ) ] Lower threshold: b = log [ (1-) /  ]

Using Cardiac Surgical Register data (CSR), crossed 0.001 threshold in 1991

What threshold to choose? For a single series, perhaps set =  = 0.1 for ‘alert’, =  = 0.01 for ‘alarm’ ? But what if we are monitoring many GPs? Greatly increases risk of one GP crossing threshold by chance alone – problem of ‘multiple testing’ There are 27,000 GPs in England Perhaps use =  = 0.000001 ?

Shipman: older females would have set off ‘alarm’ in 1985, after only 40 deaths

But is this an appropriate comparison? Excess of home death certificates could be due to a good terminal care programme by a caring GP

Second comparative analyses by Baker O = All deaths in Shipman’s practice : ‘Expecteds’ from ONS data. Special linkage required as deaths not routinely linked to general practice. But overall death rates will only be changed by substantial reductions in length of life

Problems with this analysis? SPRT ‘built up credit’ while Shipman’s performance seemed reasonable Overall death rates will only be changed by substantial reductions in length of life, not by killing terminally ill patients

How might monitoring work in practice, using all death certificates? Group at Imperial College commissioned by Shipman Inquiry Mortality data from 1993 on 1000 GPs Very limited age-sex adjustment Huge variation in mortality rates found among GPs ( multiplicative over-dispersion model ) Risk-adjusted CUSUMs used (like SPRT but avoids problem of ‘building up credit’)

‘Signalling’ GPs were subsequently investigated

So what is happening now? “The Department of Health must take the lead in developing a national system for monitoring GP patient mortality rates” “It is disappointing that the arrangements for the monitoring and analysis of mortality data that were envisaged at the time of the Inquiry seminar have not progressed” Dame Janet Smith – Fifth report, Shipman Inquiry

References P Aylin, NB Best, A Bottle, EC Marshall EC (2003) Following Shipman: a pilot system for monitoring mortality rates in primary care. Lancet 362, 485-491 R Baker. Harold Shipman’s clinical practice 1974-1998: a Review Commissioned by the Chief Medical Officer. London: The Stationery Office, 2001. The Shipman Inquiry. www.the-shipman-inquiry.org.uk/ DJ Spiegelhalter, R Kinsman, O Grigg and T Treasure. (2003) Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman, and adult cardiac surgery. International Journal for Quality in Health Care 15:7–13 DJ Spiegelhalter and NG Best (2004) Harold Shipman’s statistical legacy. Significance, 1, 10-12 SH Steiner, RJ Cook, VT Farewell, T Treasure (2000) Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 1: 441-552