1. Keele University School of Medicine Reliability of the L eicester C linical procedure A ssessment T ool (LCAT), a tool to support holistic generic assessment of clinical procedure skills RK McKinley, J Strand, L Schuwirth T Gray, T Alun-Jones, H Miller

2. Keele University School of Medicine Background Assessment –Repertoire of competence Micro-certification –Changing roles Skills extension Skills cascade –Volume –Implications for assessment

3. Keele University School of Medicine Instrument specification Generic Holistic Multi-professional Multi-level Multi-modal Enable high quality assessment: –reliable, valid, acceptable, feasible, educational impact Nationally accredited

4. Keele University School of Medicine Methods Multi stage: –Systematic review –Qualitative development –National validation –Psychometrics –Implementation

5. Keele University School of Medicine Stage 1: Systematic review LCAT v1.7

6. Keele University School of Medicine Stage 2: Qualitative Focus groups Observed assessments, debrief interviews Trial ‘hi-fi’ OSCE LCAT v2.5

7. Keele University School of Medicine Stage 3: Delphi National sampling frame –Recruitment –Questionnaire Definition of consensus Agreement –Categories: All >95% –Components: All >89% LCAT v3.0

8. Keele University School of Medicine LCAT v3.0 Category –Communication and working with the patient –Safety –Infection Prevention –Procedural Competence –Team working Components 9 7 6 12 4

9. Keele University School of Medicine Stage 4: Psychometrics Hi-fi OSCE –8 stations (2X3 tracks) Prosthetics + simulator Venepuncture, venous cannulation, IV drugs, ABG, ♀&♂ urinary catheterisation, skin suture, ECG ‘Prep station’ 15 minutes + 5 feedback

10. Keele University School of Medicine Stage 4: Psychometrics Analysis –Used data from 1 assessor per station –Assessors not systematically assigned to stations –Variable group of assessors –Unbalanced design

11. Keele University School of Medicine Stage 4: Psychometrics Analysis –P (candidate) –S (stations) –A:(PS) assessors within candidates x stations –PS,e interaction candidates and stations + error

12. Keele University School of Medicine Stage 4: Psychometrics 46 candidates 50 assessors –19 medical –27 nurses –1 midwife –3 HCA

13. Keele University School of Medicine Stage 4: Psychometrics Results: variance EffectDfVariance component % of total variance P (candidates)412.2824.1 S (stations)70.9810.3 PS,e (candidate x stations plus general error) 2873.6138.1 A:PS (assessors within candidates x stations) 1492.5927.3

14. Keele University School of Medicine Stage 4: Psychometrics Results: D analysis N stations N Assessors 123 60.660.700.71 70.690.730.75 80.720.760.77 90.740.780.79 100.760.790.81 120.790.820.83 150.830.850.86 0.76 0.79

15. Keele University School of Medicine Summary Generic Holistic Multi-professional Multi-level Multi-modal Enable high quality assessment: –reliable, valid, acceptable, feasible, educational impact Nationally accredited

16. Keele University School of Medicine Summary Generic Holistic Multi-professional Multi-level Multi-modal Quality: –reliable, valid, acceptable, feasible, educational impact Nationally accredited

17. Keele University School of Medicine Acknowledgements Analysis: –Ron Hoogenboom, Cees van der Vleuten and Arno Muijtjens Colleagues: –Linda Ward, Clinical Librarian UHL –University Hospitals of Leicester –Eastern Leicester PCT –Students and staff University of Leicester De Montfort University Funding: –Nation Health Service University –Leicestershire, Northamptonshire and Rutland WDC

18. Keele University School of Medicine