1. What are we to do? Jim Crossley for Maastricht 2012

2. Outline A crisis in WPBA A brief history in lenses Construct-aligned scales Some implications

3. Outline A crisis in WPBA A brief history in lenses Construct-aligned scales Some implications

4. Crisis UK Academy of Medical Royal Colleges 2010 “The profession has rightly lost all confidence in Workplace Based Assessment”

5. Outline A crisis in WPBA A brief history in lenses Construct-aligned scales Some implications

6. Getting a clear picture

8. First lens - constructs ?

9. Getting a clear picture

11. Domain of Interest Test Sample  Sampling

12. Second lens - sampling ?

13. Getting a clear picture

15. Outline A crisis in WPBA A brief history in lenses Construct-aligned scales Some implications

16. Surgical skills study 3 instruments 1635 assessments 437 cases 15 index procedures 5 specialities 85 surgical trainees 155 judges Modelled reliability with 3 cases & 2 observers PBA, G=0.81 NOTTS, G=0.59 OSATS, G=0.65

17. PBA response scale Level 1 Unable to perform the procedure, or part observed, under supervision Level 2 Able to perform the procedure, or part observed, under supervision Level 3 Able to perform the procedure with minimum supervision (needed occasional help) Level 4 Competent to perform the procedure unsupervised (could deal with complications that arose) Very Poor Poor Good Very good

18. It generalises

19. Getting a clear picture

20. Third lens – judge alignment ?

21. Outline A crisis in WPBA A brief history in lenses Construct-aligned scales Some implications

22. Understanding development Please join in!

23. judgement and obscurity Now we see as through a glass darkly; then we shall see face to face. Now I know in part; then I shall know fully, even as I am fully known. 1 Corinthians 13:12