Journal Club 29/11/13 Jessica Griffith

Chosen Paper ‘Measuring the quality of anaesthesia from a patient’s perspective: development, validation, and implementation of a short questionnaire’ Hocking et al Br J Anaesth 2013 Dec; 111(6) 979-89

Background An important aspect of the quality of anaesthetic care is the satisfaction of the patient with their care Need a suitable instrument to measure this Simple and easily administered Validated

Quality Of health care Through the patient’s Eyes (QuOTE) Series of studies performed in Netherlands Institute of Health Services Research Items generated and ranked for importance by patients No input from experts

Validity Definition: degree to which a research study measures what it intends to measure Compare with existing psychometric tool Use statistics e.g. Cronbach’s α

Novel Test of Validity “a valid satisfaction instrument should contain information about the concept that is being measured – the patient’s perception of the quality of anaesthesia. If this is true, effective feedback of this information to anaesthetists might be expected to change the patient’s perception of quality of anaesthesia”

Aims of Study Develop and validate a short instrument the Perception of Quality in Anaesthesia (PQA) Identify the most important aspects of anaesthesia quality from the patient’s perspective

Methods Ethical approval Single centre - Sir Charles Gairdner Hospital, Perth – university affiliated tertiary institution October 2008 – Oct 2009

Part 1: Derivation and Validation

Generating of Items Asked patients and members of the public to describe any attributes they consider important for ‘high quality anaesthesia’ Face-to-face and by e-mail Recruitment continued until no new attributes were generated

Ranking of Items Attributes were printed onto a pack of cards A different set of participants ranked them in order of importance 12 top ranked attributes used in pilot study Pilot study assessed wording, understanding and practicality

Perception of Quality in Anaesthesia Questionnaire If you had a general anaesthetic, do you remember anything during surgery? If you did not have a general anaesthetic (had a nerve block or epidural, etc.) did you feel any pain during the operation? How would you rate the management of your pain after the operation? Did you have any nausea (feeling sick) or vomiting after the operation?

Did your anaesthetist address any concerns you had regarding your anaesthetic? Did your anaesthetist talk to you in a way you could understand? Did you feel confident in the ability of your anaesthetist? How would you rate the amount of information your anaesthetist gave you? How gentle was your anaesthetist? How would you rate your anaesthetist’s technical skills? (needles, drips etc.) How friendly was your anaesthetist? How would you rate the amount of time you had with your anaesthetist?

Comparative Questions Overall satisfaction: How would you rate your anaesthetic experience overall? Would you recommend your anaesthetist to a good friend? Overall anxiety and understanding of risk: Were you nervous about having this anaesthetic? What level of risk did you think you were taking in undergoing anaesthesia?

Validating Included 714 patients over 4 week period Participants invited to suggest additional questions more important than those on questionnaire Test-retest reliability assessed on 100 patients Content validity study performed with separate sample of post-op patients and medical experts

Execution of Questionnaire Interview conducted on day after surgery by one of 4 trained interviewers Inclusion criteria: adult patients undergoing elective and emergency surgery requiring anaesthetic care Excluded: patients ventilated post-op intellectual impairment requiring permanent care

Scoring Each question has a five point scale e.g. very poor to very good Unsatisfactory patient response defined as score ≤ 3 Performance Score = proportion of patients with unsatisfactory response Importance Score = mean rank for each attribute Quality Index = importance score x performance score

Additional Data Data forms filled in by anaesthetist included: Patient characteristics Anaesthetic details Perioperative details Data forms cross-checked against anaesthetic record and operating theatre management systems

Part 2: Evaluation of the PQA in a Feedback Study

Feedback 48 week before and after study To determine if providing anaesthetists with information from the PQA was associated with any change in patient perception of quality of anaesthesia 6 month period of individualised feedback After this PQA performance scores were re-measured

Statistical Analysis Principle component analysis Used in exploratory data analysis to determine the structure of correlations within data Content validity study – Aiken V coefficients were calculated Poisson regression – to see if responses to PQA questions were associated with various aspects of medical care and patient characteristics

Results: Part 1 Generation, Ranking and Pilot Testing

Generation and Ranking 100 participants generated 52 items Ranked in order of importance by further 20 participants Top 12 items included

Patient Validation of the PQA 714 patients completed questionnaire out of 799 eligible (89% response rate) Questionnaire completed within 3 mins Pearson correlation coefficient of test-retest was 0.88 Highest quality index questions: PONV question (2.3) Pain management question (1.04) Communication with anaesthetist (0.3)

Content Validity Study 30 post-op patients Mean Aiken V coefficient of 0.84 (P<0.0001) 54 medical experts Mean Aiken V coefficient of 0.72 (P<0.0001) Indicate satisfactory content validity Cronbach’s α for all questions was 0.7

Results: Part 2 Evaluation of the PQA in a Feedback Study

Pre-feedback group Studied weeks 1 – 14 2359 patients eligible and 2046 consented (87% response rate) Unsatisfactory responses ≥ 1 = 45% ≥ 2 = 12% ≥ 3 = 3%

Feedback group Studied in weeks 15-39 4251 patients returned questionnaire

Post-feedback group Weeks 40-49 1721 eligible with 1421 returning questionnaires (83% response rate) Unsatisfactory responses: ≥ 1 = 35% (45%) ≥ 2 = 6% (12%) ≥ 3 = 1% (3%)

Classification of Components of PQA

PQA Responses Indicating Unsatisfactory Performance

Results Summary Similar pre- and post-feedback differences were observed if the PQA completed by interview, or left at bedside Patients in pre-feedback group had 45% higher rate of unsatisfactory responses Differences in anaesthetic management in post-feedback group: Increased use of 5HT3 anti-emetics More inductions of anaesthesia where consultant assisted trainee Reduced use of morphine as intra-op opioid

Conclusions “We have developed a short instrument to measure the patient’s perception of the quality of anaesthesia and demonstrated that it is valid and reliable” “Individualised, repeated feedback of patient satisfaction-related performance to health practitioners can improve the patient’s experience”

Anaesthesia Quality Communication with anaesthetist Avoidance of PONV Gentleness and attention Information and confidence Addressing the patient’s concerns Avoidance of PONV Post-op pain management

Discussion Validity? Difficult to determine if any psychometric instrument truly measures what is intended No gold standard as reference Conventional methods used In addition tested PQA in before and after study

Limitations No power calculation used before starting the investigation No control group for before and after study (parallel group of patients treated by anaesthetists not exposed to feedback) Confounded by patient characteristics and surgical differences between pre-and post-feedback groups

Additional Factors Affecting PQA Patients less likely to give an adverse rating of their experience if they: underwent shorter surgery male older age less pre-operative anxiety

How is this tool different? Several instruments already developed to measure patient satisfaction Aim to measure all aspects of satisfaction – opinions of patients and experts Lengthy to perform Factors often included: Waiting/delays Recovery nursing care Continuity of care Emotional support

Is this relevant to Arrowe Park Hospital?

Any questions or comments?