1. Measuring the Important Regular Practice Review
IPAC / CPE Conference
5 October 2012
Dr John Adams - Chairperson
David Dunbar – Registrar
Medical Council of New Zealand

2. Presentation Outline The New Zealand environment
legal framework for recertification in New Zealand
Regular practice review
philosophy and background
targeted approach
tools
Challenges encountered with implementation of RPR
Next steps

4. New Zealand – the numbers
Population million (NZ)
Ireland million
Sydney million
Washington million
Toronto million
Land area (sq kilometres) 270,000 (NZ)
Alberta ,000
Victoria (Australia) 227,000
United Kingdom 244,000
Colorado 270,000

5. Doctors in New Zealand – a profile
Nearly 14,500 registered doctors
1,550 new doctors registered each year
Around 40% of doctors have overseas primary medical qualifications, coming from over 100 countries
Principal categories of registration (scopes of practice)
9,432 Vocational (35 scopes of practice)
7,889 General
3,510 Provisional General
330 New Zealand-trained
1,220 International medical graduates (over 75%)

6. Legal framework in New Zealand
The Health Practitioners Competence Assurance Act 2003 regulates 22 professions and over 70,000 health practitioners, through 16 regulatory authorities
The Act’s purpose: to protect the health and safety of members of the public by providing for mechanisms to ensure that health practitioners are competent and fit to practise their professions
MCNZ ‘s legal functions include to review and promote the competence of doctors, and to recognise, accredit, and set programmes to ensure ongoing competence
MCNZ sets general recertification programmes and (largely individualised) competence programmes - with significant powers to enforce compliance
Issuing annual practising certificate to doctor equates to annual certification by MCNZ that a doctor is competent and fit to practise
Compliance with recertification programme obligations key to that annual process

7. Trust and confidence
NZ Doctors are respected for the high standard of care they provide
Most doctors meet high standards routinely and are dedicated to lifelong learning
Medical regulation needs to support this professionalism
Regulation should be as much about improving and assuring the standards of doctors, as it is about identifying and addressing poor performance
MCNZ’s overriding interest in the safety of patients and the quality of care they receive from their doctor
Medical regulation must provide a framework to maintain trust and confidence

8. Recertification as one mechanism to assure competence
CPD has been mandatory for doctors since 1996
MCNZ historically relied on CPD as a principal mechanism for assuring doctors’ competence for recertification
Large majority of doctors responsibly meet CPD requirement
Now we recognise more focus needs to be placed on the effectiveness of CPD activities on performance

9. General scope recertification requirements
A professional development plan
20 hours of continuing medical education
10 hours of peer review
Annual participation in audit of medical practice
Minimum number of meetings with nominated collegial relationship provider (6 the first year, 4 yearly thereafter)
Essentials test in the first year of enrolment and then every 3 years
Multisource or patient feedback questionnaires (every 3 years)
Regular practice review (also every 3 years)

10. Regular Practice Review
Workplace based assessment can vary from longer term observation and the use of assessment tools in the doctor’s everyday work, through extended courses or assessment processes, to practice visits of a day or two’s duration.
In our conceptualisation, RPR is a workplace based practice visit using a variety of tools and observations over one to two days.
A formative process as a part of CPD, similar in format to our competence assessments, but with the primary purpose of improving doctors’ practice.

11. Purpose of regular practice review
To help maintain and improve standards of the profession – a quality improvement process
May also assist in the identification of poor performance
GOAL
To help individual doctors identify areas where aspects of their performance could be improved, benefiting not only their own professional development but also the quality of care that their patients receive.

12. MCNZ initial targeted approach
General scope - RPR is now mandatory for doctors registered in a general scope of practice (as one component of a structured and strengthened recertification programme)
Vocational scope – doctors may choose to participate in RPR on a voluntary basis. MCNZ is working closely with medical colleges encouraging them to develop RPR processes and make them available to doctors as a component of CPD activities.
General scope doctors are those who have not completed postgraduate training
General scope doctors are more highly represented in competence and conduct referrals.
Gathering momentum with the medical colleges. Those who have implemented some type of RPR processes in their CPD activities are:
Obstetricians and gynaecologists
Orthopaedic surgeons (they have made RPR a mandatory component for all of their members)
Dermatologists
General practitioners
Those working in internal medicine

13. RPR Context
Our view is that RPR should be primarily based in the profession and that our role as a regulator is to support and encourage this. The programme for General Registrants is a more active first step with the regulator taking a very clear lead.
“Professional self-regulation is dependent on effective and credible assessment. From this perspective, assessment becomes an important act of professionalism by the physician. (6,10)”
“assessment should not be just about identifying the incompetent physicians.(13)
Holmboe E, Journal of Continuing Education In The Health Professions, 28(s1):s4–s10, 2008

14. Workplace-Based Assessment – why bother?
“Although many forms of assessment can be used to show a doctor’s knowledge or competence, there is evidence that competence does not reliably predict performance in clinical practice (6); one major advantage of workplace based assessment is its ability to evaluate performance in context. (7)”
Miller A, Archer J. BMJ 2010;341:c5064
“…doctors’ abilities when assessed in a controlled (deconstructed) environment do not dependably predict their actual day-to-day performance.(7,8)”
Crossley J, Jolly B. Medical Education 2012: 46: 28–37
“Doctors have limited ability to self-assess their learning needs…… The worst accuracy is in the least skilled and the most confident.”
JAMA 2006;296:

15. What activities do not show improved performance or outcomes?
Formal CME conferences without enabling or practice-reinforcing strategies have little impact
Didactic sessions
Self assessment and self directed learning
Large group teaching

16. What activities do improve performance?
Interactive programmes
Providing doctors with their own performance data
Activities based on identified learning needs.
Activities anchored in a doctor’s own work environment and individual practice
assessed by an external body or person, rather than based on self evaluation

17. Tools commonly used Observation of clinical activities Records review
Discussion of clinical cases
Chart stimulated recall
Case based discussion
Multisource feedback
Miller A, Archer J. BMJ 2010;341:c5064

18. RPR Process
In a 2012 review, Crossley and Jolly make a convincing case for subjective global assessments by experienced observers in assessing performance.
“In essence, scraping up the myriad evidential minutiae of the subcomponents of the task does not give as good a picture as standing back and considering the whole.”
Crossley J & Jolly B, Medical Education 2012: 46: 28–37

19. RPR Process
“Four general principles emerge: the response scale should be aligned to the reality map of the judges; judgements rather than objective observations should be sought; the assessment should focus on competencies that are central to the activity observed, and the assessors who are best placed to judge performance should be asked to participate.”
Crossley J & Jolly B, Medical Education 2012: 46: 28–37

20. Is there evidence that ‘RPR’ works?
“Considering the emphasis placed on workplace based assessment as a method of formative performance assessment, there are few published articles exploring its impact on doctors’ education and performance.”
Miller A, Archer J. BMJ 2010;341:c5064

21. Overeem K et al, Medical Education 2007: 41: 1039–1049

22. Workplace-Based Assessment Impact
“Eight studies examined multisource feedback,(17-24) four concentrated on the mini-clinical evaluation exercise,(25-28) one investigated direct observation of procedural skills,(29) and three looked at multiple assessment methods.(30-32)”
“There is now convincing evidence that systematic feedback delivered by a credible source can change clinical performance,”
“The strongest evidence for workplace based assessment improving performance comes from studies examining multisource feedback.”
“We were unable to unearth any clear evidence to show that the mini-clinical evaluation exercise, direct observation of procedural skills, or case based discussion can lead to improvements in performance.”
Miller A, Archer J. BMJ 2010;341:c5064

23. Challenges in Implementation
Doctor resistance
“Is there any evidence that it works?”
Stakeholder concerns:
Royal Colleges and Specialist Societies:
Bi-national differences
Evidence
Placing requirements on their members
Capacity
Cost
District Health Boards and other employers:
Service efficiency
Unions
Will this be used by employers in disputes with employees?
Implications for an already stretched workforce.

24. Challenges in Implementation
The MCNZ has consulted on the use of RPR widely.
In discussion with Colleges and Societies the MCNZ has been clear that it sees RPR as a very useful addition to CPD requirements.
We have been clear that we are not at this stage requiring RPR in CPD regimes for Vocationally Registered doctors, but that we support its development.
Some Colleges and Societies (O&G, Orthopaedics) have taken this up.
We have required RPR as a part of our new CPD Programme for General Registrants.

25. NZ Orthopaedic Association pilot report

26. NZ Orthopaedic Association pilot report

27. NZ Orthopaedic Association Pilot

28. Next steps
Consideration of a risk based screening approach, that is identifying risk factors, and targeting mandatory RPR to those who demonstrate particular risk
Continue to work closely with medical colleges encouraging implementation of RPR
Evaluation of effectiveness (for example longitudinal study) of RPR

29. Regular Practice Review
Discussion

30. Queenstown, New Zealand
IPAC 2013
Queenstown, New Zealand
Monday 30 September – Wednesday 2 October 2013