1. Assessment of Clinical Competencies
Zubair Amin MD MHPE
Associate Professor; Dept of Pediatrics
Yong Loo Lin School of Medicine
Senior Consultant; Dept of Neonatology
National University Hospital

2. Overview of clinical assessment
Basic psychometric principles on clinical assessment
Bias
Sampling
Context specificity
Basic educational principles about clinical assessment
Educational impact
Feedback
Quality assurance
General properties of innovative instruments

3. “ Assessment Drives Student Learning.”
George E Miller

4. “Assessment drives learning in at least four ways: its content, its format, its timing and any subsequent feedback given to the examinee.”
van der Vleuten, C. (1996) The Assessment of Professional Competence: Developments, Research and Practical Implications, Advances in Health Sciences Education, 1, pp. 41–67.

5. Knowledge and Performance
Professional authenticity
Does
Shows how
Knows how
Knows
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

6. Examiner
Question
Ex 1
Ex 2
Ex 3
Ex 4
Ex 5
Q 1 X
Q 2
Q 3
Q 4
Q 5

7. Examiner
Question
Ex 1
Ex 2
Ex 3
Ex 4
Ex 5
Q 1 X

8. Examiner
Question
Ex 1
Ex 2
Ex 3
Ex 4
Ex 5
Q 1 X
Q 2
Q 3
Q 4
Q 5

9. One case: a patient with diabetes mellitus
Scenario A
One student
One case: a patient with diabetes mellitus
Examiner 1
Examiner 2
30 minutes for long case
Decision: Pass
Total Examiners’ Time: 60 minutes (30X2)

10. 15 minutes for each long case Decision: Pass
Scenario B
Diabetes Mellitus
Rheumatoid Arthritis
Asthma
Stroke
Examiner A
Examiner B
Examiner C
Examiner D
15 minutes for each long case
Decision: Pass
Total Examiners’ Time: 60 minutes (15X4)

11. You are the Chairperson of the examination committee
You are the Chairperson of the examination committee. The particular candidate is known to be weak.
In which situation (Scenario A or Scenario B) you are more comfortable in examiners’ decision to pass the candidate?
What are the comparative advantages and disadvantages between Scenario A and Scenario B?

12. Context Specificity There is no generic problem solving skills
Geoff Norman
Management of Diabetes Ketoacidosis = Outpatient management of diabetes
Management of COPD = Management of rheumatoid arthritis
Diagnosis of asthma = Counseling of asthma patient
Typical correlation of “skills” across problems is 0.1 – 0.3

13. “Assessment drives learning in the direction you wish.”
“The ‘law’ of educational cause and effect states that: for every evaluative action, there is an equal (or greater) (and sometimes opposite) educational reaction.”
Schuwirth, L.W.T. (2001) General Concerns About Assessment. Web address:
“Assessment drives learning in the direction you wish.”

14. Unintended Consequences of Assessment
Evolution of Medical Students
Website by NUS students:
Unintended Consequences of Assessment

15. Educational Impact and Feedback
Any test is anxiety provoking for the students and (staff)
Test has potential positive and negative steering effects on learning and professional development
We tend to assess more, but provide considerably less feedback
Provision for feedback should be strongly considered in performance-based assessment

16. Linking Learning and Assessment

17. ‘All or none state’ – not really
Concept of Mastery
‘All or none state’ – not really

18. Continuum of Clinical Competencies
Student
Trainee
Doctor in Practice

19. A examination that attempts to test students’ competency at a given point of time is less preferable than one that tests the competency over a span of time

20. Some Common Features in Assessment of Clinical Competency
Multiple examiners
Multiple cases (patients, problems)
Context free
Over a span of time
Feedback and improvement of learning

21. Traditional Clinical Examination
One/two examiners evaluate a candidate
Candidate takes one patient
Tasks
Take history
Complete physical examination
Interpret data
Generate differential diagnosis
Discuss the patient problem with examiners

23. Issues Single patient: context specificity Single examiner: bias
Patient: mostly in-patients
“Exam” patients
Artificial
Unlike most physician-patient encounters
Limited or no opportunity for observation of communication
Limited or no opportunity for feedback

24. Instruments for Clinical Competency Assessment
OSCE
Mini-CEX
DOPS
360 degree evaluation
Portfolio
Many others

25. Mini-CEX Assessor observes a trainee in any setting
Takes into account of contexts
Complexity
Focus of visit (diagnosis, management, follow-up)
Uses standard form (not customized to particular patient/problem)
Completes 6-10 encounters

26. Implementation Issues
Generally reliable and reproducible
At least 4-6 encounters
Different assessors, different patient, different context
Feedback

27. The mini-clinical evaluation exercise ( mini -CEX) form
Norcini, J. J. et. al. Ann Intern Med 2003;138:

28. Direct Observation of Procedural Skills
Mini-CEX equivalent for procedural skills assessment
Similar psychometric properties
Observation of procedures with real patients
Minimum of 4-6 procedures, different assessors
Provision of feedback

29. 3600 Assessment / Multi-Source Feedback
Involves evaluation of the candidate by multiple individuals within the sphere of influence
Self, superior, peers, co-workers (nurse, ward clerk)
Assessment of professionalism

30. 3600 Assessment Multiple observations by multiple observers
Over a period of time
Provide evidence, as opposed to impression
Highly valued as a developmental tool

31. 3600 Assessment Focus on behavior: action taken by the individual
Not used for knowledge or skills
Nurses often are better discriminators
Difficult to standardized grades
Unsatisfactory Satisfactory Superior
Customize to your needs

32. Fundamental Principles
Assessment should be driven by purpose
Consider both psychometric and non-psychometric parameters of assessment tool
Create a backbone of assessment with few high quality, practical tools
Strive towards holistic profiling of a candidate