1. Journal Club
29/11/13
Jessica Griffith

2. Chosen Paper
‘Measuring the quality of anaesthesia from a patient’s perspective: development, validation, and implementation of a short questionnaire’
Hocking et al
Br J Anaesth 2013 Dec; 111(6)

3. Background
An important aspect of the quality of anaesthetic care is the satisfaction of the patient with their care
Need a suitable instrument to measure this
Simple and easily administered
Validated

4. Quality Of health care Through the patient’s Eyes (QuOTE)
Series of studies performed in Netherlands Institute of Health Services Research
Items generated and ranked for importance by patients
No input from experts

5. Validity
Definition: degree to which a research study measures what it intends to measure
Compare with existing psychometric tool
Use statistics e.g. Cronbach’s α

6. Novel Test of Validity
“a valid satisfaction instrument should contain information about the concept that is being measured – the patient’s perception of the quality of anaesthesia. If this is true, effective feedback of this information to anaesthetists might be expected to change the patient’s perception of quality of anaesthesia”

7. Aims of Study
Develop and validate a short instrument the Perception of Quality in Anaesthesia (PQA)
Identify the most important aspects of anaesthesia quality from the patient’s perspective

8. Methods Ethical approval
Single centre - Sir Charles Gairdner Hospital, Perth – university affiliated tertiary institution
October 2008 – Oct 2009

9. Part 1: Derivation and Validation

10. Generating of Items
Asked patients and members of the public to describe any attributes they consider important for ‘high quality anaesthesia’
Face-to-face and by
Recruitment continued until no new attributes were generated

11. Ranking of Items Attributes were printed onto a pack of cards
A different set of participants ranked them in order of importance
12 top ranked attributes used in pilot study
Pilot study assessed wording, understanding and practicality

12. Perception of Quality in Anaesthesia Questionnaire
If you had a general anaesthetic, do you remember anything during surgery?
If you did not have a general anaesthetic (had a nerve block or epidural, etc.) did you feel any pain during the operation?
How would you rate the management of your pain after the operation?
Did you have any nausea (feeling sick) or vomiting after the operation?

13. Did your anaesthetist address any concerns you had regarding your anaesthetic?
Did your anaesthetist talk to you in a way you could understand?
Did you feel confident in the ability of your anaesthetist?
How would you rate the amount of information your anaesthetist gave you?
How gentle was your anaesthetist?
How would you rate your anaesthetist’s technical skills? (needles, drips etc.)
How friendly was your anaesthetist?
How would you rate the amount of time you had with your anaesthetist?

14. Comparative Questions
Overall satisfaction:
How would you rate your anaesthetic experience overall?
Would you recommend your anaesthetist to a good friend?
Overall anxiety and understanding of risk:
Were you nervous about having this anaesthetic?
What level of risk did you think you were taking in undergoing anaesthesia?

15. Validating Included 714 patients over 4 week period
Participants invited to suggest additional questions more important than those on questionnaire
Test-retest reliability assessed on 100 patients
Content validity study performed with separate sample of post-op patients and medical experts

16. Execution of Questionnaire
Interview conducted on day after surgery by one of 4 trained interviewers
Inclusion criteria:
adult patients undergoing elective and emergency surgery requiring anaesthetic care
Excluded:
patients ventilated post-op
intellectual impairment requiring permanent care

17. Scoring
Each question has a five point scale e.g. very poor to very good
Unsatisfactory patient response defined as score ≤ 3
Performance Score = proportion of patients with unsatisfactory response
Importance Score = mean rank for each attribute
Quality Index = importance score x performance score

18. Additional Data Data forms filled in by anaesthetist included:
Patient characteristics
Anaesthetic details
Perioperative details
Data forms cross-checked against anaesthetic record and operating theatre management systems

19. Part 2: Evaluation of the PQA in a Feedback Study

20. Feedback 48 week before and after study
To determine if providing anaesthetists with information from the PQA was associated with any change in patient perception of quality of anaesthesia
6 month period of individualised feedback
After this PQA performance scores were re-measured

22. Statistical Analysis Principle component analysis
Used in exploratory data analysis to determine the structure of correlations within data
Content validity study – Aiken V coefficients were calculated
Poisson regression – to see if responses to PQA questions were associated with various aspects of medical care and patient characteristics

23. Results: Part 1 Generation, Ranking and Pilot Testing

24. Generation and Ranking
100 participants generated 52 items
Ranked in order of importance by further 20 participants
Top 12 items included

25. Patient Validation of the PQA
714 patients completed questionnaire out of 799 eligible (89% response rate)
Questionnaire completed within 3 mins
Pearson correlation coefficient of test-retest was 0.88
Highest quality index questions:
PONV question (2.3)
Pain management question (1.04)
Communication with anaesthetist (0.3)

26. Content Validity Study
30 post-op patients
Mean Aiken V coefficient of 0.84 (P<0.0001)
54 medical experts
Mean Aiken V coefficient of 0.72 (P<0.0001)
Indicate satisfactory content validity
Cronbach’s α for all questions was 0.7

27. Results: Part 2 Evaluation of the PQA in a Feedback Study

28. Pre-feedback group Studied weeks 1 – 14
2359 patients eligible and 2046 consented (87% response rate)
Unsatisfactory responses
≥ 1 = 45%
≥ 2 = 12%
≥ 3 = 3%

29. Feedback group Studied in weeks 15-39
4251 patients returned questionnaire

30. Post-feedback group Weeks 40-49
1721 eligible with 1421 returning questionnaires (83% response rate)
Unsatisfactory responses:
≥ 1 = 35% (45%)
≥ 2 = 6% (12%)
≥ 3 = 1% (3%)

31. Classification of Components of PQA

34. PQA Responses Indicating Unsatisfactory Performance

35. Results Summary
Similar pre- and post-feedback differences were observed if the PQA completed by interview, or left at bedside
Patients in pre-feedback group had 45% higher rate of unsatisfactory responses
Differences in anaesthetic management in post-feedback group:
Increased use of 5HT3 anti-emetics
More inductions of anaesthesia where consultant assisted trainee
Reduced use of morphine as intra-op opioid

36. Conclusions
“We have developed a short instrument to measure the patient’s perception of the quality of anaesthesia and demonstrated that it is valid and reliable”
“Individualised, repeated feedback of patient satisfaction-related performance to health practitioners can improve the patient’s experience”

37. Anaesthesia Quality Communication with anaesthetist Avoidance of PONV
Gentleness and attention
Information and confidence
Addressing the patient’s concerns
Avoidance of PONV
Post-op pain management

38. Discussion
Validity?
Difficult to determine if any psychometric instrument truly measures what is intended
No gold standard as reference
Conventional methods used
In addition tested PQA in before and after study

39. Limitations
No power calculation used before starting the investigation
No control group for before and after study (parallel group of patients treated by anaesthetists not exposed to feedback)
Confounded by patient characteristics and surgical differences between pre-and post-feedback groups

40. Additional Factors Affecting PQA
Patients less likely to give an adverse rating of their experience if they:
underwent shorter surgery
male
older age
less pre-operative anxiety

41. How is this tool different?
Several instruments already developed to measure patient satisfaction
Aim to measure all aspects of satisfaction – opinions of patients and experts
Lengthy to perform
Factors often included:
Waiting/delays
Recovery nursing care
Continuity of care
Emotional support

42. Is this relevant to Arrowe Park Hospital?

43. Any questions or comments?