1. National Institute of Economic and Social Research Metrics, Targets and Performance: Hospital Star Ratings Mary O’Mahony, Philip Stevens and Lucy Stokes

2. Introduction The setting and use of targets is generating growing interest Aim of targets to improve the quality, performance and accountability of public services Targets need to be measurable Need to assess unintended consequences Triangulation of methods

3. Types of index Output index How much of each service is being produced Welfare index What the value is to final users Performance management index How the services are being produced Composite index Includes elements of the above three

4. Properties of indices Fit for purpose Answers the question(s) being asked Appropriate domain Includes the appropriate services Aggregation framework Index should use appropriate weights Technical properties of index Transitivity, IIR, path independence

5. Public Sector Productivity Many problems in assessing public sector productivity Difficulty in identifying the outputs Lack of market prices Attribution Outputs previously measured by inputs Makes productivity measurement impossible, by definition

6. Measuring NHS outputs and productivity Important to distinguish between Activities (e.g. operative procedures, diagnostic tests, outpatient visits) Outputs (e.g. courses of treatment - may require a bundle of activities) Outcomes (e.g. the characteristics of output of value to individuals) In private sector focus is on outputs Prices value characteristics Activities embedded in outputs No markets and hence prices for NHS outputs

7. Private sector – Hi-fi Activities Soldering, assembling, QA Outputs Hi-fi system Characteristics Sound quality, reliability, CD, mp3 Outcomes Listening pleasure Bundled together as Which produce Example – Hi-fi Embody certain

8. Public sector – Healthcare Activities Surgery, drug prescription Outputs Course of treatment Characteristics Mortality, waiting times, pain Outcomes Health Bundled together as Embody certain Which produce Example – health

9. Example: Star Rating System Introduced in 2000/1 – Final year 2004/5 Will be replaced by ‘annual health check’ in 2005/6 Small set of ‘Key Targets’ A larger set of ‘balanced scorecard’ indicators Does it correlate with a comprehensive productivity index?

10. Star rating system Three starsTrusts with the highest levels of performance in the measured areas Two starsTrusts with mostly high levels of performance, but which are not consistent across all measured areas One starTrusts where there is some cause for concern regarding particular areas of measured performance No starsTrusts with the poorest level of measured performance

11. Key targets used in star ratings for acute trusts 2000/012001/022002/032003/042004/05 Shorter inpatient waiting lists * Inpatients waiting longer than the standard ***** Reduction in outpatient waiting * Outpatients waiting longer than the standard ***** Outpatient and elective (inpatient and daycase) booking ** Cancer: % seen within 2 weeks(BC only) **** Financial management ***** 12 hours waits for emergency admission via A&E following decision to admit ***** Total time in A&E: 4 hours or less *** Cancelled operations *** Improving Working Lives **** Hospital cleanliness *****

12. Did the stars move? (2002/3-2003/4) 2003/4 2002/30123 0243211 16612428 2015241958 312133753 9275262150 Trusts increasing rating = 44 (29%) Trusts decreasing rating = 37 (25%)

13. Did the stars move? (2000/1-2003/4) 2003/4 2000/10123 0116210 1474217 2211313478 30451827 7234656132 Trusts increasing rating = 49 (37%) Trusts decreasing rating = 26 (20%)

14. Implementation Cost-weighted activity index (CWAI) Implies costs reflect valuations at the margin Does this hold for the NHS? Explicit and implicit decisions Labour productivity index where C i = Reference cost of treatment Q i = Number of episodes of treatment W j = Expenditure share of labour type X j = Quantity of labour type

15. Implementation Output Hospital Episode Statistics All elective an non-elective day patients, except regular day and night attenders (10m+ episodes) Reference costs by healthcare resource group and elective/non-elective (1000+ combinations) Input NHS Census for staff numbers (whole time equivalent) Staff weighted using expenditure shares Earnings come from NHS Earnings Survey

16. Labour productivity index 2003/4 (median)

17. Productivity index: Quartile range 2003/4

18. Labour productivity index 2003/4 (mean)

19. Star-ratings and labour productivity Some relationship between star-ratings & labour productivity, but highly variable Quality adjustment Waiting times Mortality Capital and intermediate inputs Missing outputs, characteristics or outcomes Exogenous factors

20. The next steps Quality adjustment Waiting times Mortality Alternative methodologies Stochastic frontier analysis Data envelopment analysis Exogenous factors Accounting for influence of demographics