2. Medical school attendedPassing grade Dr JohnNorthsouth COM (NSCOM)80% Dr SmithEastwest COM (EWCOM)50% Which of these doctors would you like to treat you? Why?

3. Purpose of Workshop To demonstrate that the cut-off (passing) grade is meaningless if it is not underpinned with substance (Substance > > Figures) Substance = Validity, Reliability, Standard

4. Objectives  Discuss the purpose of assessment  Define standard and standard setting  Set and justify the passing grade (cut-off point) of an assessment instrument  Discuss challenges in standard setting

5. What is the Purpose of Assessment?

6. Purpose of Assessment  Learning improvement (Feedback and Feedforward)  Motivate learners  Information for decision-making  Certification of competence  Faculty evaluation  Curriculum evaluation / improvement  Public trust / accountability

7. What is a Standard?

8. What is a standard? “A special score that serves as a boundary between those who perform well enough and those who do not.” - J.J. Norcini

9. Why should we set standards?

10. Why Standard Setting? The passing score is to be based on the instructor’s judgment of what constitutes an adequate achievement on the part of the student and not on the performances by the student relative to his or her class or to any other particular group of students - Nedelsky 1954

11. Two types of standards  Norm- Referenced (Relative)  Ranks students based on test scores  Used for selection/admissions  Criterion-Referenced (Absolute)  Determines students who meet minimum standard  Tests for competence

12. Examples of Standards  Norm- Referenced (Relative)  Fixed percentage method  Criterion-Referenced (Absolute)  Angoff’s method (judgment based on Q evaluation; Industry standard)  Contrasting group method (judgment based on examinee evaluation)  Compromise method  Hofstee

13. Criteria for selecting standard type  Consistent with purpose of test  Based on expert judgment  Informed by data  Supported by research  Transparent  Due diligence

14. Standard Setting Process  Decide on type of standard  Decide on method of standard setting  Select judges  Meeting of judges  Judges set the cut-point  Decide on its application

15. Angoff’s Method  Judges define characteristics of a borderline candidate (50% chance of passing)  Individually, judges review each question and estimate the percentage of borderline candidates answering it correctly  Judges discuss and are free to individually alter their estimates  The sum of the means of the judges’ estimates is the cut-score for the instrument.

16. Angoff Method ItemJudge 1: % of BLE answering correctlyAngoff Score 148%0.5 255%0.55 365%0.65 446%0.46 554%0.54 Angoff cut-off score= 2. 7/ 5 = 0.54 = 54% Judge 1 = 54% Judge 2= 56% Judge 1 = 52% Cut-off score= 54% Total = 2.7/5

17. EXERCISE

18. Medical school attendedPassing grade Dr JohnNorthsouth COM (NSCOM)80% Dr SmithEastwest COM (EWCOM)50% Which of these doctors would you like to treat you? Why?

19. Angoff’s Method  Relatively easy to use  Focused on quality of questions  Sizably evidence based  Absolute standards good for competence and licensing exams  Judges’ decision may be arbitrary  Time consuming Advantages Disadvantages

20. Summary  Set cut-off scores in context with learning outcomes.  Select standard type according to use/purpose.  Follow a structured process when setting standards.

21. References Angoff, W.H. (1971). Scales, norms and equivalent scores. Educational Measurement. Washington, DC: American Council on Education. Norcini, J.J. (2003). Setting standards on educational tests. Medical Education, (37), 464-469. Retrieved from http://www- new1.heacademy.ac.uk/assets/Documents/subjects/medev/61_Setting_standards_on_ educational_tests_Norcini.pdf.http://www- new1.heacademy.ac.uk/assets/Documents/subjects/medev/61_Setting_standards_on_ educational_tests_Norcini.pdf Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, (14), 3-19.

22. THANK YOU!