Work based assessment Challenges and opportunities
Choosing assessments? What do you want to assess - e.g. Professionalism Clinical reasoning Technical skills Clinical evaluation Who do you want to assess Med student Resident Practising physician
Choosing assessments Patient outcome gold standard but attribution a problem Judgements largely based on process measures Case mix and reliability an issue
Assessors If sufficient subjective judgements are combined the collated judgement about performance can be reliable i.e objectivity and reliability are not the same thing Assumes that assessor has both observed competence in question and can make a judgement about its quality How many assessors is enough?
MSF in healthcare settings Feedback about observable behaviors is provided by some or all of Physician colleagues (peers, referring MDs, referral MDs) Co-workers (e.g., nurses, pharmacists, dieticians) Patients Self If trainee borderline or in difficulty sample more
Insight
MSF One to one feedback Problems may need further diagnostic work Need support mechanisms in place
Bias Consider sources of bias such as Gender Working relationship Working environment Ethnic group University of graduation
ConsSASG F1/F2 SHO SpR Nurse F=524.1 p= <0.001 GP AHP
Mini-CEX case complexity
Validity Extent to which a test assesses what it purports to Content validity Criterion validity Construct validity Face validity Consequential validity Recognised need for further validity data in relation to MSF and other work based assessments 1 Evans, R., G. Elwyn, and A. Edwards, Review of instruments for peer assessment of physicians. BMJ, (7450): p
Validity Content validity Blueprint assessments to the curriculum Evidence from a range of sources supporting the hypothesis that the score really measures what it is meant to
Relationship with patient ratings SHEFFPAT vs SPRAT r=0.12 NS SHEFFPAT vs PATSCORE r=0.45 p<0.01
Predictive and consequential validity
Centralisation Facilitates standardisation and robust QA Facilitates movement between locations Economies of scale Strategies to enhance local ownership important
Practicalities Is it feasible? Electronic vs paper Centralised vs localised Who assesses? How is training organised? Who will do the QA?
Acknowledge legitimate concerns Healthy scepticism Resources - especially time Conflicting demands espy CLINICAL Need for sampling Lack of standardisation Training needs Loss of local ownership
Support doctors in difficulty
Be patient Op-ti-mist n Somebody who tends to feel hopeful and positive about future outcomes Recognise extent of cultural change Ensure work based assessment is done as well as possible “ Every problem is just an opportunity waiting to be made use of ”
Who assesses? 57.6% 14.8% 19.4% 7.1%
The programme should be quality assured RVECA importance 100 % 0 % aspectF van der Vleuten C. The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education. 1996;1:41-67.
Sampling Content specificity Being good at one thing doesn’t mean you are good at everything Must sample clinical content widely Map to curriculum - blueprint Sources of variance Assessors significant source of variance Use lots of assessors
Classification Scheme for Work-Based Assessment Norcini BMJ 2003
Quality assurance Quality assure assessment system Modify assessment system in response to QA
Effective (work based) assessment RESOURCES and CO-ORDINATION TIME FUNDING
Key messages Sampling is crucial Consider sources of bias Have as many different clinical problems and as many assessors as possible Subjectivity ≠ unreliability Train the raters Holmboe, E. S., R. E. Hawkins, et al. (2004). "Effecs of training in direct observation of medical residents' clinical competence; a randomized trial." Ann Intern Med 140 :
PERFORMANCE OF PHYSICIAN Personal pressures: Home Health Attributes of training: Post Programme Supervisor Context of training: Workload Colleagues Patients Attributes of physician: Personality Background Aptitude for specialty MSF