1. Work based assessment h.davies@shef.ac.uk Challenges and opportunities

2. Choosing assessments? What do you want to assess - e.g. Professionalism Clinical reasoning Technical skills Clinical evaluation Who do you want to assess Med student Resident Practising physician

3. Choosing assessments Patient outcome gold standard but attribution a problem Judgements largely based on process measures Case mix and reliability an issue

4. Assessors If sufficient subjective judgements are combined the collated judgement about performance can be reliable i.e objectivity and reliability are not the same thing Assumes that assessor has both observed competence in question and can make a judgement about its quality How many assessors is enough?

5. MSF in healthcare settings Feedback about observable behaviors is provided by some or all of Physician colleagues (peers, referring MDs, referral MDs) Co-workers (e.g., nurses, pharmacists, dieticians) Patients Self If trainee borderline or in difficulty sample more

6. Insight

7. MSF One to one feedback Problems may need further diagnostic work Need support mechanisms in place

8. Bias Consider sources of bias such as Gender Working relationship Working environment Ethnic group University of graduation

9. ConsSASG F1/F2 SHO SpR Nurse F=524.1 p= <0.001 GP AHP

10. Mini-CEX case complexity

11. Validity Extent to which a test assesses what it purports to Content validity Criterion validity Construct validity Face validity Consequential validity Recognised need for further validity data in relation to MSF and other work based assessments 1 Evans, R., G. Elwyn, and A. Edwards, Review of instruments for peer assessment of physicians. BMJ, 2004. 328(7450): p. 1240.

12. Validity Content validity Blueprint assessments to the curriculum Evidence from a range of sources supporting the hypothesis that the score really measures what it is meant to

13. Relationship with patient ratings SHEFFPAT vs SPRAT r=0.12 NS SHEFFPAT vs PATSCORE r=0.45 p<0.01

14. Predictive and consequential validity

15. Centralisation Facilitates standardisation and robust QA Facilitates movement between locations Economies of scale Strategies to enhance local ownership important

16. Practicalities Is it feasible? Electronic vs paper Centralised vs localised Who assesses? How is training organised? Who will do the QA?

17. Acknowledge legitimate concerns Healthy scepticism Resources - especially time Conflicting demands espy CLINICAL Need for sampling Lack of standardisation Training needs Loss of local ownership

18. Support doctors in difficulty

19. Be patient Op-ti-mist n Somebody who tends to feel hopeful and positive about future outcomes Recognise extent of cultural change Ensure work based assessment is done as well as possible “ Every problem is just an opportunity waiting to be made use of ”

20. Who assesses? 57.6% 14.8% 19.4% 7.1%

21. The programme should be quality assured RVECA importance 100 % 0 % aspectF van der Vleuten C. The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education. 1996;1:41-67.

22. Sampling Content specificity Being good at one thing doesn’t mean you are good at everything Must sample clinical content widely Map to curriculum - blueprint Sources of variance Assessors significant source of variance Use lots of assessors

23. Classification Scheme for Work-Based Assessment Norcini BMJ 2003

24. Quality assurance Quality assure assessment system Modify assessment system in response to QA

25. Effective (work based) assessment RESOURCES and CO-ORDINATION TIME FUNDING

27. Key messages Sampling is crucial Consider sources of bias Have as many different clinical problems and as many assessors as possible Subjectivity ≠ unreliability Train the raters Holmboe, E. S., R. E. Hawkins, et al. (2004). "Effecs of training in direct observation of medical residents' clinical competence; a randomized trial." Ann Intern Med 140 : 874-881.

28. PERFORMANCE OF PHYSICIAN Personal pressures: Home Health Attributes of training: Post Programme Supervisor Context of training: Workload Colleagues Patients Attributes of physician: Personality Background Aptitude for specialty MSF