Patient experience metrics: using the data to good effect Veena Raleigh The King’s Fund HSRUK Patient feedback: Potential or problem in a changing NHS? Exeter University, 7 April

Introduction Patient experience data used widely – and wisely? Some issues to consider: - clarity about aims - familiarity with the data - realistic expectations for change - case-mix - wider system impacts Examples from analysis of trends over 9 years in inpatient survey data for 156 NHS acute trusts (KF/Picker collaboration)

Clarity about aims Patient experience data used by multiple audiences for multiple purposes: Common misconception that one tool can serve multiple aims eg FFT Clarity about aim of measurement vital Aim should drive choice of data, ensuring metrics are fit for purpose AimPurposeAudience Quality improvementInternal useProviders Performance assessment, P4P etc External judgmentDH, NHSE, CQC, commissioners Transparency, patient choice Public usePublic, patients

Matching choice of data to aims AIM OF MEASUREMENT DATA COLLECTION MODES (examples) SurveysFFT, other real-time data collections Social media Quality improvement INTERNAL USE  Performance management, CQC ratings, P4P etc EXTERNAL JUDGMENT  X X Information for patients, public PUBLIC USE   

Features DATA COLLECTION MODE (examples) SurveysFFT, other real- time data collections Social media Large, representative sample  Standard data collection methods  Case-mix adjustment  Statistical reliability  Comparative data across organisations  Timely data  Locality specific data  Free text data  Features of data on PE

1. Trusts consistently show higher performance in some areas of patient experience than others. 2. Inter-trust differences are consistently wider in some areas than others. Q 21: How would you rate the hospital food?Q 37: Were you given enough privacy when being examined or treated? Understanding the data

3. Much year-on-year variation is random, regression to the mean. Q 59: Did staff tell you about danger signals to watch for after you went home? Q 67: Overall, did you feel you were treated with respect & dignity?

Not significant Taking the long view can be useful Q 55: Did staff explain the purpose of the medicines in a way you could understand?

Q17 How clean was the hospital room or ward that you were in? sig p<0.01

Q 27: When you had important questions to ask a nurse, did you get answers that you could understand?

National data show relatively little change over a decade At trust level, performance is mixed – some improvement, some decline Most trusts show statistically significant change on few questions, and the magnitude of change is generally small Should be taken into account by eg commissioners when setting contracts, assessing performance, in P4P Having realistic expectations about change

Change in national scores for selected questions

4. Evidence of more improvement where performance is lower and a ceiling effect. Q 17: How clean was the hospital room or ward that you were in?

Case-mix is a confounder Several factors influence how patients respond, irrespective of quality: - age, gender, social class, self-reported health status, deprivation, ethnicity, LTCs, specialist vs general acute services Case-mix varies between trusts and changes over time Should be taken into account when assessing performance, comparing organisational performance, in P4P Other factors: does lack of change reflect changed expectations over time?

Many trusts showed improvements in policy priority areas with targets: - cleanliness - waiting times to admission In contrast, many trusts showed deterioration resulting from wider system pressures: - waiting time to get to a bed after admission - noise levels at night - delayed discharge Consider wider system effects

5. Some aspects of patient experience showed widespread evidence of deterioration. Q9 From arrival at hospital, length of wait to get to a bed on a ward Q52 On the day you left hospital, was your discharge delayed/?

NHS patient survey programme one of the largest internationally Data under-used nationally and locally for QI Barriers to use cited by trusts but also examples of changes in practice Policy-makers, regulators, commissioners should be cognisant of data-related issues and set realistic expectations for performance improvement Risks in inappropriate use of data eg misuse of resources More guidance needed on using the data appropriately and to good effect KF/Picker report makes recommendations for policymakers, commissioners and providers Final thoughts

Patients’ experience of using hospital services: an analysis of trends in inpatient surveys in NHS acute trusts in England, V RaleighChris Graham James ThompsonSteve Sizmur Joni JabbalAlice Coulter December