Medical school attendedPassing grade Dr JohnNorthsouth COM (NSCOM)80% Dr SmithEastwest COM (EWCOM)50% Which of these doctors would you like to treat you? Why?

Purpose of Workshop To demonstrate that the cut-off (passing) grade is meaningless if it is not underpinned with substance (Substance > > Figures) Substance = Validity, Reliability, Standard

Objectives  Discuss the purpose of assessment  Define standard and standard setting  Set and justify the passing grade (cut-off point) of an assessment instrument  Discuss challenges in standard setting

What is the Purpose of Assessment?

Purpose of Assessment  Learning improvement (Feedback and Feedforward)  Motivate learners  Information for decision-making  Certification of competence  Faculty evaluation  Curriculum evaluation / improvement  Public trust / accountability

What is a Standard?

What is a standard? “A special score that serves as a boundary between those who perform well enough and those who do not.” - J.J. Norcini

Why should we set standards?

Why Standard Setting? The passing score is to be based on the instructor’s judgment of what constitutes an adequate achievement on the part of the student and not on the performances by the student relative to his or her class or to any other particular group of students - Nedelsky 1954

Two types of standards  Norm- Referenced (Relative)  Ranks students based on test scores  Used for selection/admissions  Criterion-Referenced (Absolute)  Determines students who meet minimum standard  Tests for competence

Examples of Standards  Norm- Referenced (Relative)  Fixed percentage method  Criterion-Referenced (Absolute)  Angoff’s method (judgment based on Q evaluation; Industry standard)  Contrasting group method (judgment based on examinee evaluation)  Compromise method  Hofstee

Criteria for selecting standard type  Consistent with purpose of test  Based on expert judgment  Informed by data  Supported by research  Transparent  Due diligence

Standard Setting Process  Decide on type of standard  Decide on method of standard setting  Select judges  Meeting of judges  Judges set the cut-point  Decide on its application

Angoff’s Method  Judges define characteristics of a borderline candidate (50% chance of passing)  Individually, judges review each question and estimate the percentage of borderline candidates answering it correctly  Judges discuss and are free to individually alter their estimates  The sum of the means of the judges’ estimates is the cut-score for the instrument.

Angoff Method ItemJudge 1: % of BLE answering correctlyAngoff Score 148% % % % %0.54 Angoff cut-off score= 2. 7/ 5 = 0.54 = 54% Judge 1 = 54% Judge 2= 56% Judge 1 = 52% Cut-off score= 54% Total = 2.7/5

EXERCISE

Medical school attendedPassing grade Dr JohnNorthsouth COM (NSCOM)80% Dr SmithEastwest COM (EWCOM)50% Which of these doctors would you like to treat you? Why?

Angoff’s Method  Relatively easy to use  Focused on quality of questions  Sizably evidence based  Absolute standards good for competence and licensing exams  Judges’ decision may be arbitrary  Time consuming Advantages Disadvantages

Summary  Set cut-off scores in context with learning outcomes.  Select standard type according to use/purpose.  Follow a structured process when setting standards.

References Angoff, W.H. (1971). Scales, norms and equivalent scores. Educational Measurement. Washington, DC: American Council on Education. Norcini, J.J. (2003). Setting standards on educational tests. Medical Education, (37), Retrieved from new1.heacademy.ac.uk/assets/Documents/subjects/medev/61_Setting_standards_on_ educational_tests_Norcini.pdf. new1.heacademy.ac.uk/assets/Documents/subjects/medev/61_Setting_standards_on_ educational_tests_Norcini.pdf Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, (14), 3-19.

THANK YOU!