Prof Dame Lesley Southgate: St Georges Medical School, University of London Developing revalidation for UK physicians Maintenance of competence in the age of technology

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Purpose of this talk To show that evidence for revalidation of physicians can be collected and judged within a systematic, transferable framework which could apply to all physicians at any stage of their career To ask you, as I discuss some of the slides, to think over ways that storage and exchange of this evidence could be achieved without distorting or losing its meaning or creating false synergies

Prof Dame Lesley Southgate: St Georges Medical School, University of London Revalidation has a primary purpose to demonstrate that doctors on the register continue to meet the standards that apply to their own discipline Implies a basic standard that nearly all doctors should reach

Prof Dame Lesley Southgate: St Georges Medical School, University of London It can also be seen to have a number of secondary purposes To promote continuous professional development among practitioners To encourage improvement in the quality of care, team working, communications and appropriate behaviour of practitioners  Implies standards which allow progress towards excellence to be documented

Prof Dame Lesley Southgate: St Georges Medical School, University of London It can also be seen to have a number of secondary purposes To identify doctors, as far as is practicable, for whom there are significant concerns about their fitness to practise To reassure, as far as is possible, the public and individual patients that their doctors are up to date and fit to practise  Implies accuracy around the ‘cut point’ for the basic standard

Prof Dame Lesley Southgate: St Georges Medical School, University of London Competence versus Performance Competence is necessary, but not sufficient, for acceptable clinical performance and this concept is especially important when assessing the performance of doctors in difficulty where it is essential to determine if poor performance stems from a compromised knowledge and skill base or not- the difference between can’t perform, or won’t which may be related to a failure of professionalism or poor health or social circumstances

Prof Dame Lesley Southgate: St Georges Medical School, University of London GMC Good Medical Practice provides the broad domains underpinning a potential assessment framework All Royal Medical Royal Colleges have elaborated the guidance to define the characteristics of the excellent (and unacceptable) physician. GMP underpins every aspect of UK medical practice from undergraduate through residency to continuing professional development It underpins licensure and certification

Prof Dame Lesley Southgate: St Georges Medical School, University of London Good Medical Practice Can be mapped to Canmeds and the ACGME competencies We have a framework/classification for how we record information about a physician through the continuum of practice

Prof Dame Lesley Southgate: St Georges Medical School, University of London Domains of performance from Good Medical Practice Knowledge, skills and performance  Maintain your professional performance  Apply knowledge and experience to practice  Keep clear, accurate and legible records Safety and Quality  Put into effect systems to protect patients and improve care  Respond to risks to safety  Protect patients from any risk posed by your health

Prof Dame Lesley Southgate: St Georges Medical School, University of London Domains of performance from Good Medical Practice Communication, partnership and team work  Communicate effectively  Work constructively with colleagues and delegate effectively  Establish and maintain partnerships with patients Maintaining trust  Show respect for patients and colleagues  Treat patients and colleagues fairly and without discrimination  Act with honesty and integrity

Prof Dame Lesley Southgate: St Georges Medical School, University of London The domains of performance together determine the territory within which  high level statements of values and principles (criteria/standards) are set out which then need elaboration in order to map to actual clinical practice  to set levels of achievement (levels/standards in the other sense)  identify the best methods by which evidence of that achievement can be gathered and judged for a defined purpose

Prof Dame Lesley Southgate: St Georges Medical School, University of London Headings for a blueprint for recertification Knowledge, skills performance Safety, quality Communication Partnership teamwork Maintaining trust Curriculum Headings evidence Core practice evidence Actual practice evidence

Prof Dame Lesley Southgate: St Georges Medical School, University of London Criteria and standards From these, meaningful sub-divisions or criteria can be defined (high level statements that encompass values and principles), leaving the term “standard” as the word to be used to define the level of quality or excellence in relation to each criterion

Prof Dame Lesley Southgate: St Georges Medical School, University of London And Evidence The selection of tools to gather evidence will, however, also depend on the characteristics of the different methods, published research in the field of assessment, the likelihood of legal challenge to recommendations perceived by individuals as perverse, and the contribution each tool can make to the provision of evidence that samples the territory described within the domains, in the context of the doctor’s actual practice.

Prof Dame Lesley Southgate: St Georges Medical School, University of London The utility index The design of any assessment system will also need to take into account its utility. The utility, or usefulness, of an assessment has been defined as a product of its reliability, validity, feasibility, acceptability and educational impact.

Prof Dame Lesley Southgate: St Georges Medical School, University of London A programme of workplace assessment should be Specified (domains, criteria, standards) Developmental Based on the collection of evidence using an appropriate variety of methods Triangulated Quality assured  These principles should be applied to any emerging framework for recertification.

Prof Dame Lesley Southgate: St Georges Medical School, University of London The importance of context and actual practice in performance assessment Setting defensible standards for performance assessment is difficult  Assessment methods are required that provide evidence about how doctors respond to real clinical encounters and situations, rather than simulations.  As each encounter occurs in unique circumstances, assessors may have less data on which to base a decision about the level of performance, whether it be recorded numerically or as a qualitative judgment.  Most performance assessment methods are relatively new, particularly when used for making decisions that might restrict clinical practice.

Prof Dame Lesley Southgate: St Georges Medical School, University of London Sources of evidence for recertification of (for example) family physicians will be considered in three sections. Clinical governance data routinely available (in theory!) from an employer and collected as a result of the doctor’s clinical practice. Workplace assessment of the individual practitioner Tests of knowledge and skills

Prof Dame Lesley Southgate: St Georges Medical School, University of London Attribution The concept of using performance data from sources other than the assessment of individual doctors is not new.  controversial  debates about confidentiality and consent, individual freedoms and clinical autonomy,  no single organization or employer is likely to possess sufficient data to set standards for performance-based measures that reflect the scope of an individual’s clinical practice.  need a framework for the development of standard-setting procedures that use patient-outcomes-orientated measures of clinical performance, based on a collaborative approach between several health care stakeholders

Prof Dame Lesley Southgate: St Georges Medical School, University of London Clinical governance data Early indications have suggested that routinely gathered data across nine areas related to practice might generate some useful insights into practice level performance Areas 1-4. May be useful for individual assessment: but there are problems with attribution.  All to be accompanied by development of standardised documentation, a commentary and discussion within the appraisal meeting.

Prof Dame Lesley Southgate: St Georges Medical School, University of London PACT Prescribing Analysis and Cost (PACT) Data. Data must be attributable to individuals and prescribing in the acute situation, and within defined therapeutic areas is a promising approach. Recommended for piloting

Prof Dame Lesley Southgate: St Georges Medical School, University of London Referral Information ….. there is conflicting evidence about the usefulness of these data and referral decisions may be better judged by methods proposed later. Not recommended

Prof Dame Lesley Southgate: St Georges Medical School, University of London Complaints with a reflective piece to accompany quantitative data in the portfolio of evidence. The definition of a complaint, and it’s separation from informal concerns raised in the course of practice, will be essential. Recommended for piloting

Prof Dame Lesley Southgate: St Georges Medical School, University of London Significant Events with specification for types of cases. Standard documentation and a commentary including own contribution to be part of the portfolio of evidence available for appraisal. Recommended for piloting

Prof Dame Lesley Southgate: St Georges Medical School, University of London Quality and Outcomes Framework (QOF) Data. Area 5 Principally about the practice/team, but may be useful as a basis for reflection/discussion with appraiser (documentation to be developed) These data will be difficult to use in their present form. Not recommended

Prof Dame Lesley Southgate: St Georges Medical School, University of London Practice achievement with the exception reporting tallies The best candidate for recertification to be used as the basis for a written commentary in the portfolio of evidence available for discussion at appraisal This is a major area for piloting, and must not be ignored because of political and academic difficulties

Prof Dame Lesley Southgate: St Georges Medical School, University of London Clinical governance data we recommend use of the methods proposed above for piloting with volunteer PCTs. To include  development of standardised reporting documentation,  guidelines for how to present a commentary  training for appraisers  external review of judgements and recommendations in order to begin the definition of standards.

Prof Dame Lesley Southgate: St Georges Medical School, University of London Workplace assessment of the individual practitioner The range of methods proposed below is included because together they will provide evidence in the domains for recertification/revalidation, to be considered, triangulated and judged with data generated from clinical governance activities.

Prof Dame Lesley Southgate: St Georges Medical School, University of London Medical record review We recommend piloting in conjunction with volunteer PCTs and practices. The work to include  survey of present approaches to examining records during PCT visits to practices  development of standardised documentation  training of new, and identification of experienced reviewers  and standard setting and judgments

Prof Dame Lesley Southgate: St Georges Medical School, University of London Case based discussion (CbD) We recommend this tool for recertification as part of accredited CPD and with one formal event as part of the 5 year appraisal cycle. Piloting as part of a prototype will be necessary.

Prof Dame Lesley Southgate: St Georges Medical School, University of London Multi-source feedback The choice of tools will be based on their evaluation against the utility index. MSF cannot stand alone to provide evidence for recertification. It should be triangulated with other evidence in order to stand legal challenge. Data protection, freedom of information and attention to the provision of feedback are crucial, as is central data management.

Prof Dame Lesley Southgate: St Georges Medical School, University of London Patient satisfaction questionnaires (PSQ The review by the Picker Institute Europe 17 has identified five patient feedback questionnaires that have been used within the UK and might be considered (after suitable modification) for use within a framework of recertification.

Prof Dame Lesley Southgate: St Georges Medical School, University of London Formal Competence testing methods The possibility of using the certifying examination/methods of assessment on a voluntary basis to satisfy some of the evidential requirements as part of CPD or as evidence to include in their recertification portfolio. This will not be a requirement and will be available only on request. Further discussion will be necessary to determine the weight that might be placed on these tests for doctors who pass them, and what other evidence they may substitute.

Prof Dame Lesley Southgate: St Georges Medical School, University of London Managed CPD:1 Will be based on GP centred reflection and learning. Linked to “Essential General Practice,” this will be a systematic approach to CPD requiring GPs to demonstrate assimilation of new and evolving knowledge through an e-learning format.

Prof Dame Lesley Southgate: St Georges Medical School, University of London Managed CPD:2 Some of the tools for WBA will form part of CPD within a programme where the assessor and the doctor are peers, both of whom are participating in an accredited programme to be recorded in the portfolio. As confidence grows in the process formal occasions may be requested to provide evidence of the doctor’s knowledge base. We recommend that this approach be piloted soon, as we are certain that a minimum data set that depends heavily on MSF and untried clinical governance data will be a high risk strategy

Prof Dame Lesley Southgate: St Georges Medical School, University of London Synthesis

Prof Dame Lesley Southgate: St Georges Medical School, University of London Post script: who will judge the evidence? The training needs of appraisers are not addressed in this presentation. They are considerable, and in particular they will be acute in the arena of feedback. The potential to do harm is greatest in this area, either to individuals or teams. But the appraiser is not the judge and jury for recertification, or indeed revalidation. While they will make judgements, they will be one of many, made both within the evidence assembled for the portfolio, and by other appraisers within the 5 year cycle

Prof Dame Lesley Southgate: St Georges Medical School, University of London