Standard setting Determining the pass mark

The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna I can see dividing the bottom from the rest

Assessment Blueprinting Assessments Pass or Fail? Methods of Standard Setting - Angoff - Borderline - Hofstee …………………………………………………………. Assessment of Professional Behaviours

Vision Statement The University of Sheffield strives to produce excellent medical graduates. The medical curriculum will be outcome focussed where the aim is to produce graduates who are able to fulfil their role as junior doctors in the NHS and who also possess the generic skills expected of students attending a research-led university. The course will feature increased opportunities to see patients in the community; a high degree of integration; an emphasis on facilitating student learning; and an increase in student choice. The course will be organised on a body system basis with a progressive emphasis on learning around undifferentiated patient problems. The instructional approach will consist of a spine of problem, case and patient- based integrated learning activities complemented by a range of other teaching and learning activities. There will be an increase in systematic teaching of some components to ensure competence in key areas. Students will be expected to become progressively more self-directed, aided by increasing reliance on IT-based and distance learning materials and activities. Assessment, both formative and summative, will be closely matched to defined outcomes. The curriculum will be managed centrally by a multidisciplinary team, including those with a stake in the outcome of medical education. The Department of Medical Education, the Administration and an IT-based curriculum management system will provide support. A monitoring system will be established to evaluate the implementation of the curriculum and to support a process of continuous improvement. Excellent graduates Outcome focussed Integrated programme Student choice Undifferentiated problems Integrated Learning Activities Able learners Assessment matches outcomes Curriculum management Quality enhancement

Align Objectives Teaching & learning Assessment

Summary Assessments should match local/national outcomes for Validity Multiple observations required for Reliability Electronic data collection for optimal Feasibility Good information to students and staff to enhance Acceptance

Pass Competent Fail Incompetent

measurement error Pass Fail

Standard setting KnowledgeSkills

Angoff Hofstee Borderline method KnowledgeSkills Standard setting

Angoff (modified) What is the mark that a student who just passes would score for each item? D I S C U S +/- M O D I F Y Mean mark for all items is the pass mark Written and clinical examinations

The pass mark for this exam using a modified Angoff is 63.7% Angoff

Angoff standard setting Determine the pass mark for the ‘just-competent’ student and write it in the box Do not assume that the pass mark is 50% or 60% or constant

Borderline procedure Used for clinical (OSCE) examinations

Borderline procedure Used for clinical (OSCE) examinations Examiners score the students’s performance at the station e.g 17/20 Examiners judge the overall performance clear pass / borderline / clear fail Mark sheets rated borderline identified and the scores of borderline students averaged Process repeated for each station Calculate the median borderline score for all stations

Borderline Method

A 1 A 2 A 3 Pass/Fail Station 15 (static) Oral Lesions Score Count AAAA A A A AAAAAAAAAA A A A A A

Borderline group pass fail Borderline score 58.41

Borderline group pass fail Borderline score How wide should this band be?

Borderline group pass fail Borderline score How wide should this band be? +/- 1 - standard deviation - standard error - or what?

The Standard Error of Measurement depends on the reliability of the test (R) depends on the standard deviation of the test (SD) SEM = SD  1 - R acts as a confidence interval in high stakes situations

The Standard Error of Measurement depends on the reliability of the test (R) depends on the standard deviation of the test (SD) SEM = SD  1 - R acts as a confidence interval in high stakes situations R = Variance Variance x Error

Borderline group Pass Fail Borderline score Pass Score } - 1 SEM Fail score } + 1 SEM

Station/MEQ IDCompetenceSystemProblemMax marks available Median Borderline score Station1 Phys ExamCNSAbnormal Gait Station2 (static) Data (x-ray)RSWheeze Station3 Phys ExamGIAbdominal Mass Station4 HistoryRSHaemoptysis Station6 Pat ed/commsSkin/MiscRash Station7 HistoryEndoWeight loss Station8 (static) Pract skillsOncPain Station9 HistoryEyes/ENTVisual Disturbance Station10 Phys ExamMSS/CNSLaceration Station11 Pract SkillsCVSCollapse Station12 Phys examOncBreast Lump Station14b Pat ed/CommsOncDying Patient Station15 (static) DataSkin/MiscItch Written1 Prob solving/Dx/ManageCVSClaudication Written2 Prob solving/Dx/ManageCVS/EndoPalpitations Written3 Prob solving/Dx/ManageCVSHeart failure Written4 Prob solving/Dx/ManageRespHaemoptysis Written5 Prob solving/Dx/ManageMentalMania Written6 Prob solving/Dx/ManageENTHearing Difficulty Written7 Prob solving/Dx/ManageCNSComa Written8 Prob solving/Dx/ManageMSSJoint Pain Written9 Prob solving/Dx/ManageMSSFracture Written10 Prob solving/Dx/ManageGIDiarrhoea Written11 Prob solving/Dx/ManageGIHaematemesis Written12 Prob solving/Dx/ManageMentalOverdose Written13 Prob solving/Dx/ManageEndoWeight loss Written14 Prob solving/Dx/ManageOncBreast Lump Written15 Prob solving/Dx/ManageHaemBleeding Written16 Prob solving/Dx/ManageOncTesticular Mass Written17 Prob solving/Dx/ManageEyesRed Eye Written18 Prob solving/Dx/ManageEndoHairy Written19 Prob solving/Dx/ManageCNSWeakness Written20 Prob solving/Dx/ManageGIVomiting Static1 Investig /InterpretationCNSHeadache Static2 Investig /InterpretationGIJuandice Static3 Investig /InterpretationRespSOB Static4 Investig /InterpretationEndoDehydration Static5 Investig /InterpretationCNSConsciuoness Static6 Investig /InterpretationENTAbnormal Hearing Overall borderline score Overall borderline score %

Written14 Prob solving/Dx/ManageOncBreast Lump Written15 Prob solving/Dx/ManageHaemBleeding Written16 Prob solving/Dx/ManageOncTesticular Mass Written17 Prob solving/Dx/ManageEyesRed Eye Written18 Prob solving/Dx/ManageEndoHairy Written19 Prob solving/Dx/ManageCNSWeakness Written20 Prob solving/Dx/ManageGIVomiting Static1 Investig /InterpretationCNSHeadache Static2 Investig /InterpretationGIJuandice Static3 Investig /InterpretationRespSOB Static4 Investig /InterpretationEndoDehydration Static5 Investig /InterpretationCNSConsciuoness Static6 Investig /InterpretationENTAbnormal Hearing Overall borderline score Overall borderline score %

Number of students Mean Score (%) Overall Borderline Score (%) Standard deviation Standard Error of Measurement Pass mark (%) Reliability

Internal Reliability of the Exam Cronbach’s Alpha Cronbach’s alpha Shows whether randomly split halves of the exam by item vary together Split half Correlations (SH) Item / Stations  Population (students) 

Borderline Grading System Borderline Fail Pass Good Pass Top Mark Borderline Mark Distinction Pass Mark Pass Fail

Reducing measurement error 1. Increase reliability of exam 2. Compromise with feasibility/cost * Content specificity * More items in the exam * Blueprinting * Quality assurance of item writing * Training examiners/standardised patients * Feedback from exam performance

Hofstee procedure Used for written examinations Judges review a copy of the whole exam Judges indicate: minimum % failure rate maximum % failure rate minimum acceptable cut percentage maximum acceptable cut percentage Administer test, plot curve and read off standard

Student score % % candidates

Student score % % candidates

Student score % % candidates

Student score % % candidates

Student score % % candidates

Student score % % candidates