2. 100 years of living science Date Location of Event Monitoring clinical performance Dr Paul Aylin Dr Foster Unit Imperial College p.aylin@imperial.ac.uk 8 th November 2007

3. Contents Background Data sources Clinical information systems Routinely collected hospital data Methods Casemix adjustment Analysis and presentation Interpretation of performance data

4. Florence Nightingale

5. Uniform hospital statistics would: “ Enable us to ascertain the relative mortality of different hospitals as well as of different diseases and injuries at the same and at different ages, the relative frequency of different diseases and injuries among the classes which enter hospitals in different countries, and in different districts of the same country” Nightingale 1863

6. Heart operations at the BRI “Inadequate care for one third of children” Harold Shipman Murdered more than 200 patients

7. “Bristol was awash with data. There was enough information from the late 1980s onwards to cause questions about mortality rates to be raised both in Bristol and elsewhere had the mindset to do so existed.” Final report of the Bristol Inquiry

8. Clinical databases Need to collect extensive clinical information to facilitate adequate adjustment for case-mix has contributed to the creation and maintenance of clinical databases A survey of multicentre clinical databases found the existence of 105 such clinical databases in many areas of UK healthcare Black et.al. BMJ 2004

9. Bristol (Kennedy) Inquiry Report Data were available all the time “From the start of the 1990s a national database existed at the Department of Health (the Hospital Episode Statistics database) which among other things held information about deaths in hospital. It was not recognised as a valuable tool for analysing the performance of hospitals. It is now, belatedly.”

10. Hospital Episode Statistics UK administrative data Electronic record of every inpatient or day case episode of patient care in every NHS (public) hospital 14 million records a year 300 fields of information including Patient details such as age, sex, address Diagnosis using ICD10 Procedures using OPCS4 Admission method Discharge method HES regarded as unreliable by many clinicians

11. Comparison of administrative data vs clinical databases Isolated CABG HES around 10% fewer cases compared to National Cardiac Surgical Database Fifth National Adult Cardiac Surgical Database Report 2003. The Society of Cardiothoracic Surgeons of Great Britain and Ireland. Dendrite Clinical Systems Ltd. Henley-Upon-Thames. 2004. Vascular surgery HES = 32,242 National Vascular Database = 8,462 Aylin P; Lees T; Baker S; Prytherch D; Ashley S. (2007) Descriptive study comparing routine hospital administrative data with the Vascular Society of Great Britain and Ireland's National Vascular Database. Eur J Vasc Endovasc Surg 2007;33:461-465 Bowel resection for colorectal cancer HES 2001/2 = 16,346 ACPGBI 2001/2 = 7,635 ACPGBI database, 39% of patients had missing data for the risk factors Garout M, Tilney H, Aylin, P. Comparison of administrative data with the Association of Coloproctology of Great Britain and Ireland (ACPGBI) colorectal cancer database. International Journal of Colorectal Disease 2007.

12. Cost Administrative data £1 per record Clinical databases range from £10 (UK Cardiac Surgical Register) to £60 (Scottish Hip Fracture Audit) Raftery J, Roderick P, Stevens A. Potential use of routine databases in health technology assessment. Health Technol Assess 2005;9(20)

13. Whatever source of information Timely feedback Accessible to clinicians Case mix adjustment

14. Limited within administrative data? Age Sex Emergency/Elective

15. Risk adjustment models using HES on 3 index procedures CABG AAA Bowel resection for colorectal cancer

16. Risk factors AgeRecent MI admission SexCharlson comorbidity score (capped at 6) Method of admissionNumber of arteries replaced Revision of CABGPart of aorta repaired YearPart of colon/rectum removed Deprivation quintilePrevious heart operation Previous emergency admissionsPrevious abdominal surgery Previous IHD admissions

17. ROC curve areas comparing ‘simple’, ‘intermediate’ and ‘complex’ models derived from HES with models derived from clinical databases for four index procedures Aylin P; Bottle A; Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334: 1044

18. Calibration plots for ‘complex’ HES-based risk prediction models for four index procedures showing observed number of deaths against predicted based on validation set Aylin P; Bottle A; Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334: 1044

19. Current casemix adjustment model for each procedure and diagnosis Adjusts for age sex emergency status socio-economic deprivation diagnosis subgroup (3 digit ICD10) or procedure subgroup co-morbidity – Charlson index number of prior emergency admissions palliative care year Month of admission (for some respiratory diseases)

20. Current ROC (based on 1996/7-2006/7 HES data) for 30 day in- hospital mortality Repair of AAA = 0.792 Infra-inguinal bypass = 0.800 AP resection of rectum = 0.808 Anterior resection of rectum = 0.813 Hip replacement = 0.851 Transplantation of heart and lung = 0.569 Excision of head of pancreas = 0.681 Graft of bone marrow = 0.666

21. Issues Important to only adjust for parameters outside the control of the unit in question

22. Comparison of percentage of AVSD operations including outcome (death, alive or unknown) by age at admission (in months) between UBHT and elsewhere in England during supra-regional funding period (HES 1 April 1991 to 31 March 1995) aged under 18 months

23. Comparison of percentage of open operations including outcome (death, alive or unknown) by age at admission (in months) between UBHT and individual centres during supra-regional funding period (HES 1 April 1991 to 31 March 1995) aged under 18 months

24. Presentation of clinical outcomes “Even if all surgeons are equally good, about half will have below average results, one will have the worst results, and the worst results will be a long way below average” Poloniecki J. BMJ 1998;316:1734-1736

25. RR of death following CABG HES data 1999/00 to 2001/02

26. Criticisms of ‘league tables’ Spurious ranking – ‘someone’s got to be bottom’ Encourages comparison when perhaps not justified 95% intervals arbitrary and no consideration of multiple comparisons Single-year cross-section – what about change?

27. Account has to be taken of chance variation Bayesian approach using Monte Carlo simulations can provide confidence intervals around ranks Can also provide probability that a unit is in top 10%, 5% or even is at the top of the table See Marshall et al. (1998). League tables of in vitro fertilisation clinics: how confident can we be about the rankings? British Medical Journal, 316, 1701-4.

28. Rankings for CABG mortality 1999/00 to 2001/02

29. Statistical Process Control (SPC) charts Shipman: Aylin et al, Lancet (2003) Mohammed et al, Lancet (2001) Spiegelhalter et al, J Qual Health Care (2003) Surgical mortality: Poloniecki et al, BMJ (1998) Lovegrove et al, CHI report into St George’s Steiner et al, Biostatistics (2000) Public health: Terje et al, Stats in Med (1993) Vanbrackle & Williamson, Stats in Med (1999) Rossi et al, Stats in Med (1999) Williamson & Weatherby-Hudson, Stats in Med (1999)

30. Common features of SPC charts Need to define: in-control process (acceptable/benchmark performance) out-of-control process (that is cause for concern) Test statistic difference between observed and benchmark performance calculated for each unit at each time point Pre-defined alarm threshold minimise false alarms but remain sensitive to true signals

31. Types of SPC chart Shewhart test statistic based on current observation only no formal adjustment for multiple testing Funnel plots Can incorporate adjustment for between centre variation Easy to interpret

32. Mortality for paediatric cardiac surgery, 1991-Mar 95 for open operations for children aged under 1 year using SCTS data with 95% and 99.8% control limits based on the national average

33. Funnel plots No ranking Visual relationship with volume Takes account of increased variability of smaller centres

34. Prospective surveillance and multiple testing No prior hypothesis Prospective surveillance involves monitoring at multiple time points Sensitivity and specificity of surveillance methods depend on number of tests (time points) carried out Statistical process control charts (SPC) among the most widely used methods for sequential analysis Care required when applying SPC charts in health care setting

35. Prospective SPC charts Cumulative sums of outcomes accumulate information on performance over time formal assessment of sensitivity and specificity different ways of deriving test statistic Log-likelihood CUSUM (our preferred method) Sequential Probability Ratio Test (SPRT) Exponentially Weighted Moving Average (EWMA)

36. Risk-adjusted Log-likelihood CUSUM charts STEP 1: estimate pre-op risk for each patient, given their age, sex etc. This may be national average or other benchmark STEP 2: Order patients chronologically by date of operation STEP 3: Choose chart threshold(s) of acceptable “sensitivity” and “specificity” (via simulation) STEP 4: Plot function of patient’s actual outcome v pre-op risk for every patient, and see if – and why – threshold(s) is crossed

37. More details Based on log-likelihood CUSUM to detect a predetermined increase in risk of interest Taken from Steiner et al (2000); pre-op risks derived from logistic regression of national data The CUSUM statistic is the log-likelihood test statistic for binomial data based on the predicted risk of outcome and the actual outcome Models can adjusts for age, sex, emergency status, socio-economic deprivation etc.

39. AAA mortality monitoring

43. Urology Score Card My Practice Score Cards

44. Page 2 – Urology – Scrotal procedures

45. How do you interpret performance data?

46. Pyramid model of investigation to find credible cause explanation Lilford et al. Lancet 2004; 363: 1147-54

47. How do you interpret performance data? Check the data Difference in casemix Examine organisational or procedural differences Only then consider quality of care

48. Challenges Data quality Consensus Primary care Does information change practice?

49. Food for thought an estimated one in ten patients admitted to hospital suffers an adverse event an estimated 850,000 adverse events might occur each year in NHS hospitals some adverse events will be inevitable complications of treatment, but around half may be avoidable - that is, over 400,000 potentially avoidable adverse events every year Vincent, C.A. Presentation at BMJ conference ‘Reducing Error in Medicine;London. March 2000 Adverse events in British hospitals: preliminary retrospective record review Charles Vincent, Graham Neale, and Maria Woloshynowych BMJ 2001; 322: 517-519.

50. Food for thought eight per cent of adverse events result in death and six per cent in permanent disability - that is, over 34,000 preventable deaths and over 25,000 preventable permanent disabilities every year compensation for clinical negligence costs the NHS more than £400 million a year and altogether outstanding claims for clinical negligence add up to over £2.4 billion. Vincent, C.A. Presentation at BMJ conference ‘Reducing Error in Medicine;London. March 2000 Adverse events in British hospitals: preliminary retrospective record review Charles Vincent, Graham Neale, and Maria Woloshynowych BMJ 2001; 322: 517-519.