1. Assessment in Education Patricia O’Sullivan Office of Educational Development UAMS

2. Last Month Dr. Allen described a situation about an individual asking her how to get to Freeway Medical Center. How would you determine how successful Dr. Allen was?

3. Objectives l Provide terminology and principles fundamental to assessment l Introduce various assessment approaches l Review tips for a common assessment: Multiple Choice Questions l Review elements for a less common assessment: Portfolio

4. What is Assessment? l Appraising performance through measurement to make an evaluation l For what purpose? n Formative n Summative

5. What Assessments are Used at UAMS?

6. What Makes a Good Summative Assessment? l Fair and practical l Based on appropriate criteria that are shared with the learners l Valid l Reliable

7. Validity l Key to the interpretability and relevance of scores n Do the scores have a plausible meaning? n What are the implications of interpreting the scores? n What is the utility of the scores? n What are the actual social consequences?

8. Validity Evidence for Proposed Interpretation l Content/Construct Evidence n Content matches the objectives & teaching n Score is shown to be meaningful l Criterion/Predictor Evidence n Score relates to future outcomes

9. Reliability l Assessment data must be reproducible n Written tests Internal consistency Cronbach’s alpha or KR-20 (values range 0-1) Test-retest Correlation ranging from 0-1

10. Reliability (cont.) l Rater-based assessments n Interrater consistency or agreement (generally correlation coefficient or intraclass correlation) l OSCE & performance assessments n Generalizability Theory

11. Improve Validity l Match assessment with objectives l Increase the sample of objectives and content on the assessment l Review blueprint (or produce one) l Use test methods appropriate to objectives l Ensure test security

12. Improve Reliability l Clear questions l Appropriate time allotted l Simple, clear and unambiguous instructions l High quality scoring l Increase number of observations or questions

13. Norm or Criterion Referenced l Norm-referenced (relative standard) n Use the results of the assessment to set the standards (e.g. proportion to receive each grade); performance judged by comparison with others l Criterion-referenced (absolute standard) n Learners achieve some minimal standard of competency

14. Standard Setting Procedures l Judgments based on n Holistic impressions of the exam or item pool n Content of individual test items n Examinee’s test performance

15. Anghoff Standard Setting Procedure l Judges are instructed to think of a group of “minimally acceptable” persons l For each item estimate the proportion of this group that will answer item correctly l Sum up proportions to get the minimum passing score for a single judge l Average across judges

16. Assessment Methods l ACGME Toolbox l Additional methods n Other written assessments Essay Short answer and computation Patient management problems Modified essay questions n Create a game n Self-assessment

17. Matching Assessment Method and Objectives l Examine ACGME matrix

18. Alternative Testing Modes l Take-home tests l Open-book tests l Group exams l Paired testing

19. Reducing Test Anxiety l Make the first exam relatively easier l Give more than one exam l Give advice on how to study l Encourage study groups l Give a diagnostic test early

20. Reducing Test Anxiety l Before a test explain format l Provide sample items l Provide a pool of final items with syllabus

21. Technology and Assessment l Computer based testing n Use of images and video clips l Computer Adaptive Testing (CAT)

22. Take a Test

23. Multiple Choice Questions: Writing Tips l http://www.nbme.org/about/itemwriting.asp l Simple suggestions: n Design for one best answer n Place hand over stem. Can you guess the answer? l Use a template l Attend to Graveyard of Test Items

24. Template Examples l A (patient description) is unable to (functional disability). Which of the following is most likely to have been injured? l Following (procedure), a (patient description) develops (symptoms and signs). Laboratory findings show (findings). Which of the following is the most likely cause?

25. Graveyard l B-type: matching heading with words or phrases (can use heading more than once) l D-type: complex matching—two step process l K-type: stem with four options and choices are n A=1,2,3 only B=1,3 only C=2,4 only n D=4 only, E=all are correct

26. Portfolios

27. Case Records Logs Conferences Evaluations

28. Portfolios l Definition n A purposeful collection of student work that exhibits to the student (and/or other) the student’s efforts, progress, or achievement in (a) given area(s).

29. l This collection must include student participation in n Selection of portfolio content n The criteria for selection n The criteria for judging merit n Evidence of student reflection

30. Portfolios l Learning n Showing progress n Showing effort n Showing remediation l Assessment n Showing competence

31. Case Records Presentations Evaluations Logs

32. Criteria ______ Criteria ______

33. Criteria

34. R e f l e c t i o n Portfolio W h y s e l e c t e d H o w r e f l e c t s s k i l l s a n d a b i l i t i e s

35. Using Portfolios the Learner Engages in Assessment l Making choices l Critically self-examining l Explaining what they are doing

36. Well-structured Portfolio l Menu l Selection criteria l Scoring rubric l Benchmarks l Trained raters

37. Determine Reliability and Validity l Reliability: Generalizability theory n Found needed 6 entries with 2 raters l Validity: number of studies n Resident performance across years n Resident performance in areas n Resident perception n Resident performance when program made an intervention n Correlation of portfolio performance and other areas

38. 10-Step Approach 1. Define the purpose 2. Determine competencies to be assessed 3. Select portfolio materials 4. Develop an evaluation system 5. Select and train examiners

39. 10-Step Approach 6. Plan the examination process 7. Orient learners 8. Develop guidelines for decisions 9. Establish reliability and validity evidence 10. Design evaluation procedures