1. Peer review process Trottie Kirwan Chair APA Peer Review Committee SPAN Meeting 24 April 2009

2. Original process

3. Original Peer Review Process APA has run interdepartmental peer review since 1999 Started in the big specialist children’s hospitals Aim –raise standards of clinical practice within a paediatric anaesthetic department. Allow for experiences to be shared, with the dissemination of good or innovative practice (Review document) –to assess departments in relation to standards which should characterise anaesthetic departments throughout the country (Peter Crean in Anaesthesia 2003) Based on Good Practice – a guide for departments of anaesthesia. RCA and AAGBI. London, 1998 2 day visit by a team of four including a lay member All departments, visitors and visited, found it valuable

4. BUT The process is very demanding of both the visited department and the visitors –2 day visit including interviews with 16 different staff members as well as parents –Detailed report averaging 30 pages APA decided to extend the process to be available to all anaesthetic departments with paediatric practice Original process too great a commitment for smaller departments with fewer paediatric anaesthetists, particularly as visitors to write the reports

5. New Process

6. Simplified Peer Review Scheme New scheme developed by Peer Review Committee –Alistair Cranston, Peter Crean, Trottie Kirwan, Madeleine Wang Intended as a tool for any department treating children to –Check compliance with existing criteria of RCA, NSF, HCC/CQC –Demonstrate that standards of practice are good –Help in discussions with management and colleagues Simplify the process for departments with a smaller paediatric workload and fewer paediatric anaesthetists Reduce the demands on the reviewing team

7. Simplified Peer Review Scheme Provide a structure for local self-appraisal to help departments in raising standards of clinical practice. –Self assessment –Peer verification Based on –Good Practice Guide 2006 (JCGP RCA & AA), –RCA guidelines for the provision of anaesthetic services 2004 –NSF for Children Standard for Hospital Services 2003 –Criteria in the proposal come from RCA guidelines

8. Self-assessment

9. Departments self check against criteria in a freely available template –Department’s perceived strengths and weaknesses compiled by the paediatric anaesthetists –Check-list of criteria for good practice (from RCA guidelines) –360° appraisal by colleagues and patients/families Tool to monitor their own practice, without any external input

10. Self-assessment Checklist (from RCA guidelines) Organisation Theatres and day surgery facilities Recovery Equipment Emergency department Support services Critical care Transfer Staff Training Clinical governance Child and family experience

11. Self-assessment Checklist RCoA guidelines Equipment 2.1 A full range of monitoring devices, paediatric anaesthetic equipment and disposable items for general and regional anaesthesia should be available in theatres and all other areas where children are anaesthetised. This should include a full range of disposable equipment including the following which should be appropriate for use in children of all sizes and ages: blood pressure cuffs intravenous cannulae temperature probes pulse oximetry. 2.2 Resuscitation drugs and equipment, including an appropriate defibrillator, should be routinely available at all sites where children are to be anaesthetised. 2.3 Anaesthetic machines should incorporate ventilators, which have controls and bellows permitting their use over the entire age range together with the facility to provide pressure controlled ventilation. 2.4 There should be appropriate thermostatic control of the operating room; temperature monitoring and patient warming devices should be available in both the operating room and recovery area. 2.5 Intravenous fluids should normally be administered only by volumetric infusion pumps. Checklist Wherever children are anaesthetised: i. There is appropriate paediatric anaesthetic equipment ii. Monitoring equipment meets the current standards of the Association of Anaesthetists iii. Ventilators are suitable for children and can deliver pressure controlled mode iv. There are resuscitation drugs and equipment v. There are suitable temperature monitoring, and warming devices There are volumetric infusion pumps for IV fluid administration

12. Strengths and Weaknesses The paediatric anaesthetists, separately and collectively, list of those aspects of their work which they perceive as good and those they think need improving –The aspects which need improving are considered, any which can be fixed from within the department are addressed –The process is repeated until eventually the list consists only of strengths, and of weaknesses over which the group has no control Found to be a useful tool for discussing problems people have got used to but could actually deal with

13. 360° Appraisal by Colleagues Colleagues are invited to help in the department’s self-review –Chief Executive –Medical Director –Head of Department of Anaesthesia, anaesthetic trainees –Consultant paediatricians, paediatric surgeons, ED, PICU & transfer –Theatre Manager, Senior ODP / anaesthetic nurse –Directorate Business Manager, Anaesthetic Secretary –Senior paediatric ward nurses, play specialist, pain team Explanatory letter to say that this is a voluntary process Process focussed on the department’s organisation and practice – concerns about individuals should be reported through the normal channels

14. 360° Appraisal by Colleagues How the paediatric anaesthetic department is perceived –Concerns with the management of the department or the service Anaesthetists working as a team –Formal working relationships with colleagues –Informal collaboration for lists and individual patients Effective management of the service –Theatre time, emergency support –Cover for absent colleagues Staffing –Staff levels, recruitment and retention problems –System to deal sensitively with colleagues below an acceptable level Effective planning for equipment, staffing and budget

15. Patient/Parent Appraisal Areas covered –Pre-admission –Pre-operative preparation –Anaesthetic room –Recovery –Analgesia Design of questionnaire –Open questions elicit more information –Closed questions give more consistency Timing –At discharge easy and cheap –After return home more informative

16. Peer review

17. Peer Review Department completes self-assessment and volunteers for peer verification Visiting team –2 doctors and a lay visitor, one from outside the region –Lead reviewer with peer review experience –Could include an observer Self-assessment paperwork sent to the visiting team One day visit –Review the self-assessment evidence –meet paediatric anaesthetists –visit to clinical areas –debriefing session Review team send a report within a month

18. Peer Review External APA report –confirmation of good work –suggestions for consideration to improve organisation & practice –recommendations for use in negotiating plans with colleagues and management Problems arising at visit –visiting team cannot substantiate the self-assessment: this will be stated at the debriefing meeting and no report sent –a potentially serious problem is unearthed during a review: the review team leader will contact the regional co-ordinator and an APA peer review committee member immediately

19. Peer review process

20. Organisation of the Process Peer review is organised on a regional basis –within local paediatric anaesthetic networks where they exist –regional co-ordinator arranges teams of reviewers –each team has an out-of-region member –national co-ordinator for hospitals not in a local network Extending the process nationally includes –smaller departments of fewer paediatric anaesthetists –less experience of review visiting and report writing –more support is needed Standard job description documents –task lists for regional co-ordinators, review teams, and departments being reviewed –criteria for eligibility for Peer Review Team membership, anaesthetic and lay

22. Guidance for departments being reviewed 1. The department needs to agree that peer review will be useful, and carry out the self-evaluation, using it as a tool for self-improvement (unless you are perfect already) 2. Carry out the SWOT process, getting all the paediatric anaesthetists to agree to a consensus document 3. Once the self-evaluation is completed and you are ready to be reviewed, contact the local paediatric anaesthesia peer review co-ordinator, where there is one, or peer review committee chair. 4. Make contact with the review team leader 5. Inform relevant individuals, wards and departments in the Trust, emphasising that this is a voluntary process undertaken by the anaesthetic department to recognise and improve the quality of the service, and not because of a complaint. 6. Collate the short portfolio, self evaluation documents and supporting evidence paperwork and send a copy to each member of the reviewing team, as much as possible in electronic format, 4 weeks before the visit to send to the visiting team: Paperwork to send to the review team members The short portfolio SWOT consensus document The results of the colleague appraisals The self-assessment check-list Check-list criteria evidence which should be sent: Reports of incident reviews Department policies and guidelines Summaries of completed audits from the last 2 years Minutes of meetings which contain relevant information Copies of patient information Evidence of any research – facilities, support (ethics), trainee input Critical incident reports and presentations to M&M meetings List of any complaints and the management of them

23. Organisation of the Process Confidentiality –Reports, and the information they are based on, are confidential and will not be passed to anyone by the visiting team members –The visited department may use the report as they choose Quality control –individual performance monitoring by formal feedback after each review by both the visitors and visited –consistency monitoring by an external visitor Support for the process –peer review committee –e-network on APA website –meetings at APA ASM and linkmen meetings

24. Feedback Self-assessmentSurprisingly useful. We all thought that we knew the problems in our hospital but when we discussed our thoughts we found some important concerns that I had not considered before. Additionally it encouraged us to have a detailed meeting at which I could disseminate useful information that others paed anaesthetists had been unaware. Arranging the reviewTime consuming. Very difficult to arrange reviewers. People need a lot of notice. Clear requirements for evidence I may not have read the documentation closely enough but I missed the requirement for evidence. Would have been helpful to give a long list of possible sources. Visit discussions useful and professional Visit discussion was very useful, there were no great surprises Report received promptly, and contents as expected from visit discussions Not received yet ( only 2 weeks ago) Process helpful and fairProcess very helpful. Stimulated discussion and action within the hospital before the review and added political muscle to help implement long overdue change in some areas. Other commentsTime consuming but completely worthwhile process. A continuous cycle of review of the process will help to make it less time consuming and more rewarding

25. Process development Peer review intended to be a collaboration with users –Proposals on APA website – colleagues invited to feed back –SWACA pilot Changes to self-assessment –drop case studies –flexibility in training and experience criteria Changes to process –better guidance on how to do it –standard templates for questionnaires, reports, feedback –addition of out-of-region review team member Good ideas –children included in reviewing team

26. Benefits Visited department –Incentive to look critically at local practice and make changes before writing self-assessment –Team building within the department and with other colleagues –Reports have been useful in getting organisational change and improvements in facilities and equipment Visitors –Good ideas to manage common problems: meeting life support criteria, managing scheduling of lists Region –Clinical links reinforced –Recognition that our colleagues in other hospitals are good doctors and friendly humans

27. Misgivings Conspiracy –HCC approach – choose inappropriate criteria and stop good departments doing good work –Predator approach – local big hospital stealing work –Forced marriage approach – departments merge and one gets the cream Cock-up –Huge amount of work for no outcome –Report which will be ignored –Review will have no power to compel change May be overcome if some hospitals make a start while others wait to see how it turns out

28. SPAN Some interest Some cynicism Risk/benefit balance in favour of giving peer review a try Perhaps start with the psychopathically keen

29. References Draft proposal for revised peer review scheme http://www.apagbi.org.uk/docs/APA_PeerReview(Draft).pdf http://www.apagbi.org.uk/docs/APA_PeerReview(Draft).pdf Good Practice: A Guide for Departments of Anaesthesia. The Royal College of Anaesthetists and Association of Anaesthetists of Great Britain and Ireland, 3 rd edition 2006 http://www.rcoa.ac.uk/docs/goodpractice(oct2006).pdfhttp://www.rcoa.ac.uk/docs/goodpractice(oct2006).pdf Department of Health (2003) Getting the right start: National Service Framework for Children - A Standard for Hospital Services http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndG uidance/DH_4006182 http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndG uidance/DH_4006182 Guidelines for the Provision of Anaesthetic Services. Royal College of Anaesthetists. 2004 http://www.rcoa.ac.uk/docs/GPAS.pdfhttp://www.rcoa.ac.uk/docs/GPAS.pdf Quality in paediatric anaesthesia: a pilot study of interdepartmental peer review. Crean PM, Stokes MA, Williamson C, Hatch DJ Anaesthesia. 2003 Jun;58(6):543-8.