1. Standard setting Determining the pass mark

2. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna I can see dividing the bottom from the rest

3. Assessment Blueprinting Assessments Pass or Fail? Methods of Standard Setting - Angoff - Borderline - Hofstee …………………………………………………………. Assessment of Professional Behaviours

4. Vision Statement The University of Sheffield strives to produce excellent medical graduates. The medical curriculum will be outcome focussed where the aim is to produce graduates who are able to fulfil their role as junior doctors in the NHS and who also possess the generic skills expected of students attending a research-led university. The course will feature increased opportunities to see patients in the community; a high degree of integration; an emphasis on facilitating student learning; and an increase in student choice. The course will be organised on a body system basis with a progressive emphasis on learning around undifferentiated patient problems. The instructional approach will consist of a spine of problem, case and patient- based integrated learning activities complemented by a range of other teaching and learning activities. There will be an increase in systematic teaching of some components to ensure competence in key areas. Students will be expected to become progressively more self-directed, aided by increasing reliance on IT-based and distance learning materials and activities. Assessment, both formative and summative, will be closely matched to defined outcomes. The curriculum will be managed centrally by a multidisciplinary team, including those with a stake in the outcome of medical education. The Department of Medical Education, the Administration and an IT-based curriculum management system will provide support. A monitoring system will be established to evaluate the implementation of the curriculum and to support a process of continuous improvement. Excellent graduates Outcome focussed Integrated programme Student choice Undifferentiated problems Integrated Learning Activities Able learners Assessment matches outcomes Curriculum management Quality enhancement

5. Align Objectives Teaching & learning Assessment

6. Summary Assessments should match local/national outcomes for Validity Multiple observations required for Reliability Electronic data collection for optimal Feasibility Good information to students and staff to enhance Acceptance

7. Pass Competent Fail Incompetent

8. measurement error Pass Fail

9. Standard setting KnowledgeSkills

10. Angoff Hofstee Borderline method KnowledgeSkills Standard setting

11. Angoff (modified) What is the mark that a student who just passes would score for each item? D I S C U S +/- M O D I F Y Mean mark for all items is the pass mark Written and clinical examinations

12. The pass mark for this exam using a modified Angoff is 63.7% Angoff

13. Angoff standard setting Determine the pass mark for the ‘just-competent’ student and write it in the box Do not assume that the pass mark is 50% or 60% or constant

14. Borderline procedure Used for clinical (OSCE) examinations

16. Borderline procedure Used for clinical (OSCE) examinations Examiners score the students’s performance at the station e.g 17/20 Examiners judge the overall performance clear pass / borderline / clear fail Mark sheets rated borderline identified and the scores of borderline students averaged Process repeated for each station Calculate the median borderline score for all stations

17. Borderline Method

18. A 1 A 2 A 3 Pass/Fail Station 15 (static) Oral Lesions 0.001.002.003.004.005.006.007.008.009.0010.00 Score 0 10 20 30 40 50 Count AAAA A A A AAAAAAAAAA A A A A A

19. Borderline group pass fail Borderline score 58.41

20. Borderline group pass fail Borderline score 58.41 How wide should this band be?

21. Borderline group pass fail Borderline score 58.41 How wide should this band be? +/- 1 - standard deviation - standard error - or what?

22. The Standard Error of Measurement depends on the reliability of the test (R) depends on the standard deviation of the test (SD) SEM = SD  1 - R acts as a confidence interval in high stakes situations

23. The Standard Error of Measurement depends on the reliability of the test (R) depends on the standard deviation of the test (SD) SEM = SD  1 - R acts as a confidence interval in high stakes situations R = Variance Variance x Error

24. Borderline group Pass Fail Borderline score 58.41 Pass Score 60.93 } - 1 SEM Fail score 55.88 } + 1 SEM

25. Station/MEQ IDCompetenceSystemProblemMax marks available Median Borderline score Station1 Phys ExamCNSAbnormal Gait40 23.50 Station2 (static) Data (x-ray)RSWheeze10 6.00 Station3 Phys ExamGIAbdominal Mass20 11.75 Station4 HistoryRSHaemoptysis20 12.00 Station6 Pat ed/commsSkin/MiscRash20 13.00 Station7 HistoryEndoWeight loss20 13.50 Station8 (static) Pract skillsOncPain10 5.00 Station9 HistoryEyes/ENTVisual Disturbance20 12.00 Station10 Phys ExamMSS/CNSLaceration20 11.00 Station11 Pract SkillsCVSCollapse20 10.50 Station12 Phys examOncBreast Lump20 13.00 Station14b Pat ed/CommsOncDying Patient20 12.50 Station15 (static) DataSkin/MiscItch10 6.00 Written1 Prob solving/Dx/ManageCVSClaudication10 6.00 Written2 Prob solving/Dx/ManageCVS/EndoPalpitations10 6.00 Written3 Prob solving/Dx/ManageCVSHeart failure10 6.00 Written4 Prob solving/Dx/ManageRespHaemoptysis10 7.00 Written5 Prob solving/Dx/ManageMentalMania10 5.00 Written6 Prob solving/Dx/ManageENTHearing Difficulty10 4.00 Written7 Prob solving/Dx/ManageCNSComa10 7.00 Written8 Prob solving/Dx/ManageMSSJoint Pain10 5.00 Written9 Prob solving/Dx/ManageMSSFracture10 5.00 Written10 Prob solving/Dx/ManageGIDiarrhoea10 7.00 Written11 Prob solving/Dx/ManageGIHaematemesis10 6.00 Written12 Prob solving/Dx/ManageMentalOverdose10 5.00 Written13 Prob solving/Dx/ManageEndoWeight loss10 7.00 Written14 Prob solving/Dx/ManageOncBreast Lump10 6.00 Written15 Prob solving/Dx/ManageHaemBleeding10 5.00 Written16 Prob solving/Dx/ManageOncTesticular Mass10 4.00 Written17 Prob solving/Dx/ManageEyesRed Eye10 5.00 Written18 Prob solving/Dx/ManageEndoHairy10 5.00 Written19 Prob solving/Dx/ManageCNSWeakness10 5.50 Written20 Prob solving/Dx/ManageGIVomiting10 7.00 Static1 Investig /InterpretationCNSHeadache10 6.00 Static2 Investig /InterpretationGIJuandice10 5.00 Static3 Investig /InterpretationRespSOB10 6.00 Static4 Investig /InterpretationEndoDehydration10 5.00 Static5 Investig /InterpretationCNSConsciuoness10 6.00 Static6 Investig /InterpretationENTAbnormal Hearing10 6.00 Overall borderline score 510 297.25 Overall borderline score % 100 58.28

26. Written14 Prob solving/Dx/ManageOncBreast Lump10 6.00 Written15 Prob solving/Dx/ManageHaemBleeding10 5.00 Written16 Prob solving/Dx/ManageOncTesticular Mass10 4.00 Written17 Prob solving/Dx/ManageEyesRed Eye10 5.00 Written18 Prob solving/Dx/ManageEndoHairy10 5.00 Written19 Prob solving/Dx/ManageCNSWeakness10 5.50 Written20 Prob solving/Dx/ManageGIVomiting10 7.00 Static1 Investig /InterpretationCNSHeadache10 6.00 Static2 Investig /InterpretationGIJuandice10 5.00 Static3 Investig /InterpretationRespSOB10 6.00 Static4 Investig /InterpretationEndoDehydration10 5.00 Static5 Investig /InterpretationCNSConsciuoness10 6.00 Static6 Investig /InterpretationENTAbnormal Hearing10 6.00 Overall borderline score 510 297.25 Overall borderline score % 100 58.28

27. Number of students Mean Score (%) Overall Borderline Score (%) Standard deviation Standard Error of Measurement Pass mark (%) Reliability 0419573.2558.285.502.4060.680.81 0321574.458.415.772.5260.930.81 0221473. 059. 814.952.9862.790.64 0119571.8558.905.863.2162.110.70

28. Internal Reliability of the Exam Cronbach’s Alpha Cronbach’s alpha Shows whether randomly split halves of the exam by item vary together Split half Correlations (SH) Item / Stations  Population (students) 

29. Borderline Grading System Borderline Fail Pass Good Pass Top Mark Borderline Mark Distinction Pass Mark Pass Fail

30. Reducing measurement error 1. Increase reliability of exam 2. Compromise with feasibility/cost * Content specificity * More items in the exam * Blueprinting * Quality assurance of item writing * Training examiners/standardised patients * Feedback from exam performance

31. Hofstee procedure Used for written examinations Judges review a copy of the whole exam Judges indicate: minimum % failure rate maximum % failure rate minimum acceptable cut percentage maximum acceptable cut percentage Administer test, plot curve and read off standard

32. 0 10 20 30 40 50 60 70 80 90 100 50607080 Student score % % candidates

33. 0 10 20 30 40 50 60 70 80 90 100 50607080 Student score % % candidates

34. 0 10 20 30 40 50 60 70 80 90 100 50607080 Student score % % candidates

35. 0 10 20 30 40 50 60 70 80 90 100 50607080 Student score % % candidates

36. 0 10 20 30 40 50 60 70 80 90 100 50607080 Student score % % candidates

37. 0 10 20 30 40 50 60 70 80 90 100 50607080 Student score % % candidates