Assessment of Clinical Competencies Zubair Amin MD MHPE Associate Professor; Dept of Pediatrics Yong Loo Lin School of Medicine Senior Consultant; Dept of Neonatology National University Hospital

Overview of clinical assessment Basic psychometric principles on clinical assessment Bias Sampling Context specificity Basic educational principles about clinical assessment Educational impact Feedback Quality assurance General properties of innovative instruments

“ Assessment Drives Student Learning.” George E Miller 1919-1998

“Assessment drives learning in at least four ways: its content, its format, its timing and any subsequent feedback given to the examinee.” van der Vleuten, C. (1996) The Assessment of Professional Competence: Developments, Research and Practical Implications, Advances in Health Sciences Education, 1, pp. 41–67.

Knowledge and Performance Professional authenticity Does Shows how Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Examiner Question Ex 1 Ex 2 Ex 3 Ex 4 Ex 5 Q 1 X Q 2 Q 3 Q 4 Q 5

Examiner Question Ex 1 Ex 2 Ex 3 Ex 4 Ex 5 Q 1 X

Examiner Question Ex 1 Ex 2 Ex 3 Ex 4 Ex 5 Q 1 X Q 2 Q 3 Q 4 Q 5

One case: a patient with diabetes mellitus Scenario A One student One case: a patient with diabetes mellitus Examiner 1 Examiner 2 30 minutes for long case Decision: Pass Total Examiners’ Time: 60 minutes (30X2)

15 minutes for each long case Decision: Pass Scenario B Diabetes Mellitus Rheumatoid Arthritis Asthma Stroke Examiner A Examiner B Examiner C Examiner D 15 minutes for each long case Decision: Pass Total Examiners’ Time: 60 minutes (15X4)

You are the Chairperson of the examination committee You are the Chairperson of the examination committee. The particular candidate is known to be weak. In which situation (Scenario A or Scenario B) you are more comfortable in examiners’ decision to pass the candidate? What are the comparative advantages and disadvantages between Scenario A and Scenario B?

Context Specificity There is no generic problem solving skills Geoff Norman Management of Diabetes Ketoacidosis = Outpatient management of diabetes Management of COPD = Management of rheumatoid arthritis Diagnosis of asthma = Counseling of asthma patient Typical correlation of “skills” across problems is 0.1 – 0.3

“Assessment drives learning in the direction you wish.” “The ‘law’ of educational cause and effect states that: for every evaluative action, there is an equal (or greater) (and sometimes opposite) educational reaction.” Schuwirth, L.W.T. (2001) General Concerns About Assessment. Web address: www.fdg.unimaas.nl/educ/lambert/ubc “Assessment drives learning in the direction you wish.”

Unintended Consequences of Assessment Evolution of Medical Students Website by NUS students: http://medicus.tk Unintended Consequences of Assessment

Educational Impact and Feedback Any test is anxiety provoking for the students and (staff) Test has potential positive and negative steering effects on learning and professional development We tend to assess more, but provide considerably less feedback Provision for feedback should be strongly considered in performance-based assessment

Linking Learning and Assessment

‘All or none state’ – not really Concept of Mastery ‘All or none state’ – not really

Continuum of Clinical Competencies Student Trainee Doctor in Practice

A examination that attempts to test students’ competency at a given point of time is less preferable than one that tests the competency over a span of time

Some Common Features in Assessment of Clinical Competency Multiple examiners Multiple cases (patients, problems) Context free Over a span of time Feedback and improvement of learning

Traditional Clinical Examination One/two examiners evaluate a candidate Candidate takes one patient Tasks Take history Complete physical examination Interpret data Generate differential diagnosis Discuss the patient problem with examiners

Issues Single patient: context specificity Single examiner: bias Patient: mostly in-patients “Exam” patients Artificial Unlike most physician-patient encounters Limited or no opportunity for observation of communication Limited or no opportunity for feedback

Instruments for Clinical Competency Assessment OSCE Mini-CEX DOPS 360 degree evaluation Portfolio Many others

Mini-CEX Assessor observes a trainee in any setting Takes into account of contexts Complexity Focus of visit (diagnosis, management, follow-up) Uses standard form (not customized to particular patient/problem) Completes 6-10 encounters

Implementation Issues Generally reliable and reproducible At least 4-6 encounters Different assessors, different patient, different context Feedback

The mini-clinical evaluation exercise ( mini -CEX) form Norcini, J. J. et. al. Ann Intern Med 2003;138:476-481

Direct Observation of Procedural Skills Mini-CEX equivalent for procedural skills assessment Similar psychometric properties Observation of procedures with real patients Minimum of 4-6 procedures, different assessors Provision of feedback

3600 Assessment / Multi-Source Feedback Involves evaluation of the candidate by multiple individuals within the sphere of influence Self, superior, peers, co-workers (nurse, ward clerk) Assessment of professionalism

3600 Assessment Multiple observations by multiple observers Over a period of time Provide evidence, as opposed to impression Highly valued as a developmental tool

3600 Assessment Focus on behavior: action taken by the individual Not used for knowledge or skills Nurses often are better discriminators Difficult to standardized grades Unsatisfactory Satisfactory Superior Customize to your needs

Fundamental Principles Assessment should be driven by purpose Consider both psychometric and non-psychometric parameters of assessment tool Create a backbone of assessment with few high quality, practical tools Strive towards holistic profiling of a candidate