1. Does pay for performance save lives? Martin Roland University of Cambridge UK

2. Does pay for performance save lives? University of Manchester Matt Sutton, Ruth Boaden, Søren Rud Kristensen, Rachel Meacock, Alex Turner Warwick Business School Ruth MacDonald University of Cambridge Martin Roland All authors declare that they have no conflict of interest. The study was funded by the National Institute of Health Research

3. What’s the evidence on pay for performance? Current literature suggests: Clear evidence that pay for performance can improve the process of care Not a ‘magic bullet’ – effects likely to be limited in the absence of other quality improvement measures Very limited evidence of impact on outcome

4. P4P scheme introduced in 2008 in all 24 NHS hospitals in the North West of England (population 6.8m) Process indicators for acute myocardial infarction, heart failure, pneumonia, hip and knee surgery and coronary artery bypass surgery Evaluation of ‘Advancing Quality’

5. Months 1-12: Tournament system in which only the top performers received a bonus (top quartile received extra 4% second quartile extra 2%) Months 13-18: Bonuses for both achieving targets and improving score Months 19-42: Penalties if hospitals failed to achieve targets ‘Advancing Quality’ payment rules

6. Impact on mortality for acute myocardial infarction, heart failure and pneumonia in: First 18 months (Sutton et al NEJM 2012;367:1821–8) Next 24 months (NEJM ‘revise and resubmit’) We did not include hip and knee surgery (low mortality) or coronary bypass surgery (not carried out in most hospitals) Evaluation of ‘Advancing Quality’ over 42 months

7. Study design We studied 30 day in-hospital mortality comparing mortality for acute myocardial infarction, heart failure and pneumonia using a difference in difference analysis with three sets of controls: Mortality for these conditions in all other 132 hospitals in England Mortality for non incentivised conditions in: a) hospitals in the North West of England and b) all other hospitals in England

8. Selection of non-incentivised conditions not clinically linked to any incentivized condition sufficient volume (over 9,000 admissions in England per year) 30-day mortality over 6% more than 80% of deaths within 30 days occurring in a hospital Six conditions met these criteria: acute renal failure, alcoholic liver disease, intracranial injury, and intestinal obstruction, pulmonary embolism and duodenal ulcer

9. Data available for analysis Used routine hospital activity data collected from NHS hospitals All admissions for 18 months before and 42 months after the introduction of the scheme Patients treated at the 24 NHS hospitals in the North West and the 132 NHS hospitals in all other regions of England

10. Number of admissions in each analysis 18 months48 months Pneumonia410,384761,954 Heart failure201,003338,921 Acute myocardial infarction 245,187390,652 Non-incentivised conditions 241,009333,991 Includes admissions for 18 months prior to P4P scheme

11. Difference in differences analyses controlled for: Gender Age Type of admission (planned/unplanned, transfer) Location from which the patient was admitted 31 co-morbidities (Elixhauser) derived from secondary diagnostic codes

12. Incentivised conditions Non - incentivised conditions Comparison 2 Incentivised conditions Non - incentivised conditions Comparison 3 Incentivised conditions Comparison 1 Incentivised conditions 24 hospitals in North West England 132 hospitals in the rest of England

13. Difference-in-differences analyses Changes in mortality summarised as triple difference in differences analysis

14. Sensitivity analyses Controlling for baseline mortality Controlling for changes in patient volumes Controlling for hospital quality and financial rating (data from national regulator) Controlling for hospital type (teaching) Different method of classifying co-morbidities (Charlson ) Checking pre-intervention trends in mortality no different between incentivised and non-incentivised conditions or between areas. Comparing North West with a subset of similar English districts instead of the whole of England

15. Summary of analysis of first 18 months Significant reduction in absolute mortality of 1.3% (relative reduction of 6%) Driven by significant reduction for pneumonia and non-significant reductions for MI and heart failure Equivalent to 890 fewer deaths in the North West in the 18 month period Sutton M, Nikilova S, Boaden R, Lester H, McDonald R, Roland M. Reduced mortality with hospital pay for performance in England. New England Journal of Medicine 2012;367:1821–8

16. Analysis of the next 24 months Between-region difference-in-differences analysis Time trends for each of the 20 quarter year periods Separate estimates of impact of short and long term periods

17. What happened in the second time period? Mortality continued to fall in the North West, but there was no longer a significant difference compared to controls Mortality for incentivized conditions fell more in control hospitals than in Advancing Quality hospitals (0.7 percentage points more, 95% CI 0.3 to 1.2) Mortality for non-incentivized conditions fell more in Advancing Quality hospitals than in control hospitals (1.2 percentage points more, 95% CI 0.4 to 2.0).

18. What happened in the second time period? Incentivised conditions Non- incentivised conditions Advancing Quality hospitals (North West)  First 18 months-1.6%-0.5%  Next 24 months-1.8%-2.9% Rest of England (controls)  First 18 months-0.8%-1.2%  Next 24 months-2.3%-1.7% Estimates are from weighted least-squares regression models Results are robust to a range of specifications of standard errors and weights

19. Possible explanations Mortality for incentivized conditions fell more in control hospitals than in Advancing Quality hospitals Did the other hospitals start to use the AQ quality indicators?

20. Possible explanations Mortality for incentivized conditions fell more in control hospitals than in Advancing Quality hospitals Did the other hospitals start to use the AQ quality indicators? Mortality for non-incentivized conditions fell more in Advancing Quality hospitals than in control hospitals Were there ‘spillover effects’ on other aspects of care in the intervention hospitals?

21. Possible explanations Did the other hospitals start to use the AQ quality indicators? The results of the AQ pilot were widely disseminated and given national publicity Two other regions of the country developed P4P schemes using the AQ indicators The reduction in mortality in these regions was more than in other regions of England

22. Possible explanations Were there ‘spillover effects’ on other aspects of care in the intervention hospitals? We looked at which doctors cared for patients with the non-incentivised conditions There was substantial overlap between doctors caring for the incentivised conditions and two of the non- incentivised ones – acute renal failure and alcoholic liver disease The fall in mortality for these two non-incentivised conditions was greater than for the other non-incentivised conditions

23. Lessons for evaluation Impact on outcomes may not be long-lasting Spillover effects may hinder evaluation by masking true effects, e.g. into un-incentivised hospitals onto un-incentivised conditions (spillover effects are deliberate)

24. How did we explain the reduction in mortality? Not solely on the basis of changes in process measures Wide range of quality improvement strategies New data collection systems and feedback on performance Employment of new specialist nurses Regular face to face learning events between hospitals

25. Why was it different from the US (HQID)? Self-selected hospitals in US Larger bonuses Greater probability of earning a bonus More face to face learning between hospitals

26. Pay for performance can have a beneficial impact on outcomes The causal pathway is likely to be multi-factorial and not just related to improvements in incentivised aspects of care The impact may not be long-lasting, but spillover effects (which are welcome) may mask longer term improvement

27. Quality indicators in the Advancing Quality P4P scheme Pneumonia Blood cultures performed in the ER prior to initial antibiotics Adult smoking cessation advice/counselling Initial antibiotic received within six hours of hospital arrival Initial antibiotic selection in immuno-competent patients Oxygenation assessment Heart failure Discharge instructions Adult smoking cessation advice/counselling ACEI or angiotensin receptor blocker for left ventricular systolic dysfunction Left ventricular systolic function assessment Acute myocardial infarction Adult smoking cessation advice/counselling Beta blocker prescribed at discharge Aspirin at arrival Aspirin prescribed at discharge Fibrinolytic therapy received within 30 minutes of hospital arrival ACEI or angiotensin receptor blocker for left ventricular systolic dysfunction