1. Could statistical science have caught Harold Shipman earlier?
David Spiegelhalter MRC Biostatistics Unit Cambridge
BA Festival of Science, September 6th 2005

2. Shipman background 1974: General Practitioner in Todmorden
1975: Left when convicted of drug offences
1977: Joins group practice in Hyde
1992: Sets up in single-handed practice
Jan 1998: Positive report from Tameside audit group: ‘Keep up the good work!’

3. March 1998: local GP contacts coroner with suspicions
July 1998: police contacted concerning a possibly forged will
Sept 1998: Shipman arrested. Enquiries made concerning 192 deaths.
Jan 2000: convicted of murdering 15 patients
2001: Report from Richard Baker for DoH
2001: Public Inquiry examines 401 deaths
2004: Shipman commits suicide in prison

4. Shipman Inquiry July 2002:
215 definite victims
45 probable

6. First comparative analyses by Baker compares -
‘Observed’ deaths: those at home or in his practice and certified by Shipman,
with
‘Expected’ deaths: corresponding rates for (small sample of) local GPs.

7. (NB: Shipman Inquiry total of definite or probable victims:
189 female > 65, 55 male over 65)

8. Sequential monitoring
Accumulates evidence of ‘excess risk’
Simplest method is a cumulative plot of Observed - Expected (O – E)
Good intuitive plot
But when should we ‘blow the whistle’?

9. The Bristol Inquiry into excess mortality

10. But when to sound an alarm?

11. How soon can we tell if we are ‘off-target’?

12. Meanwhile, back in 1943
A group of statisticians called SR17 were working in the Ministry of Supply
Included Barnard, Lindley, Plackett, Armitage etc
Barnard developed the `Sequential probability ratio test’
Simultaneously discovered by Abraham Wald in the US
George Barnard later wrote British Standard 3704 for condoms

13. Sequential probability ratio test (SPRT)
Most powerful sequential test between two hypotheses H0 and H1
Based on log(likelihood ratio)
LLR = log [ p(data| H1) / p(data| H0) ]
Straightforward to adapt to ‘risk-adjustment’ model
Horizontal thresholds set by required error rates

14. Risk-adjusted SPRT for count data
Suppose observe O adverse events
E are expected under H0
Alternative hypothesis H1: risk ratio r
Contribution to log(likelihood ratio) is
LLR = O log r - ( r -1) E
Suppose we want to detect a doubling of risk (r = 2), then
LLR = O log 2 - E
i.e. plot O – E , rather than O - E

15. For Bristol data

16. Setting thresholds
= Probability of eventually rejecting H0 when it is true (Type I error)
= Probability of eventually rejecting H1 when it is true (Type II error).
Upper threshold: a = log [  / ( 1-  ) ]
Lower threshold: b = log [ (1-) /  ]

17. Using Cardiac Surgical Register data (CSR),
crossed threshold in 1991

18. What threshold to choose?
For a single series, perhaps set =  = 0.1 for ‘alert’, =  = 0.01 for ‘alarm’ ?
But what if we are monitoring many GPs?
Greatly increases risk of one GP crossing threshold by chance alone – problem of ‘multiple testing’
There are 27,000 GPs in England
Perhaps use =  = ?

19. Shipman: older females would have set off ‘alarm’ in 1985, after only 40 deaths

20. But is this an appropriate comparison?
Excess of home death certificates could be due to a good terminal care programme by a caring GP

21. Second comparative analyses by Baker
O = All deaths in Shipman’s practice :
‘Expecteds’ from ONS data.
Special linkage required as deaths not routinely linked to general practice.
But overall death rates will only be changed by substantial reductions in length of life

24. Problems with this analysis?
SPRT ‘built up credit’ while Shipman’s performance seemed reasonable
Overall death rates will only be changed by substantial reductions in length of life, not by killing terminally ill patients

25. How might monitoring work in practice, using all death certificates?
Group at Imperial College commissioned by Shipman Inquiry
Mortality data from 1993 on 1000 GPs
Very limited age-sex adjustment
Huge variation in mortality rates found among GPs ( multiplicative over-dispersion model )
Risk-adjusted CUSUMs used (like SPRT but avoids problem of ‘building up credit’)

26. ‘Signalling’ GPs were subsequently investigated

27. So what is happening now?
“The Department of Health must take the lead in developing a national system for monitoring GP patient mortality rates”
“It is disappointing that the arrangements for the monitoring and analysis of mortality data that were envisaged at the time of the Inquiry seminar have not progressed”
Dame Janet Smith – Fifth report, Shipman Inquiry

28. References
P Aylin, NB Best, A Bottle, EC Marshall EC (2003) Following Shipman: a pilot system for monitoring mortality rates in primary care. Lancet 362,
R Baker. Harold Shipman’s clinical practice : a Review Commissioned by the Chief Medical Officer. London: The Stationery Office, 2001.
The Shipman Inquiry.
DJ Spiegelhalter, R Kinsman, O Grigg and T Treasure. (2003) Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman, and adult cardiac surgery. International Journal for Quality in Health Care 15:7–13
DJ Spiegelhalter and NG Best (2004) Harold Shipman’s statistical legacy. Significance, 1,
SH Steiner, RJ Cook, VT Farewell, T Treasure (2000) Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 1: