1. The Irkutsk State Medical Academy for Postgraduate Education (Tempus IV 159328) STEERING COMMITTEE MEETING & INTERNATIONAL FINAL CONFERENCE Evaluation of Medical Student: Knowledge, know-how and life skills Pr Sylvain Meuris Faculté de Médecine Université Libre de Bruxelles

2. Objective : To Reach Professional Competence ! Is the habitual and judicious use of:  Communication  Knowledge  Evidence-based decision-making  Technical Skills  Emotions  Values and reflection to improve the health of the individual patient and the community

3. Who Assess and What Should Be Assessed ?  Transmission of knowledge, know-how & life skills between teacher and student  Who : Teacher - During the training process  Academic steeplechase (Education Minister)! Peers - End of the training process  Accreditation Council for Graduate Medical Education (Health Minister)  What : Numerous and Cumulative Competencies : Knowledge + Practice-based learning and improvement + Interpersonal and communication skills + Ethics & Professionalism + System-based practice + Patient care,…

4. Is Assessment Necessary ?  No formal assessment before the Renaissance in Europe Chinese invention (VI century) to  Select Imperial officers Imported by Portuguese Jesuits to  Select Representatives for the Portuguese Colonial Empire  Widely adopted in European administrations and Universities !!!  Probably useful

5. Is Assessment Necessary ?  A useful tool to : Train students and Assume graduation responsibility  Two questions:  WHY do we need to assess ?  WHAT do we want to measure ?

6. Is Assessment Necessary ?  Answers depends on the position  Teacher verify if the minimum level of “knowledge” is reached  Learning strategy  Student prepare examination to reach the minimum level to be authorized to continue…  Strategy to pass

7. WHY do we assess ?  To ensure safety of patients our responsibility to the public  To achieve of a minimum standard responsibility to the candidate and to the University ! In principle…

8. WHY do we assess ?  To ensure competence  As a means of academic competition In practice: the scope…

9. WHY do we assess ?  Formative: to give feedback and advice When the cook tastes the soup, that's formative  Summative: to grade When the guests taste the soup, that's summative  Qualificative: to graduate In practice: the purpose When should we assess ?

10. WHEN should we assess ? Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. Knows Shows how Knows how Does Professional authenticity Written, Oral or Computer based assessment Performance or hands on assessment

11. WHEN should we assess ? C+ Pass/Fail A A- 85% F S Unsatisfactory 67% D C B 93% 4/5 7/10 14/20 Certified Qualified Assessment is continuous (Life-Long-Learning)

12. HOW do we assess ?  Assessment The process of measuring something with the purpose of assigning a numerical value.  Scoring The procedure of assigning a numerical value to assessment task.  Evaluation The process of determining the worth of something in relation to established benchmarks using assessment information.

13. Assessment Concerns  Validity -- Is the test assessing what’s intended? Are test items based on stated objectives? Are test items properly constructed?  Difficulty -- Are questions too easy or too hard? (e.g., 30% to 70% of students should answer a given item correctly)  Discriminability -- Are the performance on individual test items positively correlated with overall student performances? (e.g., only best students do well on most difficult questions)

14. Evaluation Types  Criterion-referenced evaluation -- student performance is assessed against a set of predetermined standards  Norm-referenced evaluation -- student performance is assessed relative to the other students

15. Criterion-Referenced Eval’s  Based on a predetermined (minimum) set of criteria.  For instance, 90% and up = A 80% to 89.99% = B 70% to 79.99% = C 60% to 69.99% = D 59.99% and below = F

16. Criterion-Referenced Eval’s  Pros: - Sets minimum performance expectations. - Demonstrate what students can and cannot do in relation to important content- area standards  Cons: - Some times it’s hard to know just where to set boundary conditions - Lack of comparison data with other students and/or schools.

17. Norm-referenced Evaluation  Based upon the assumption of a standard normal (Gaussian) distribution with n > 30.  Employs the z score: A = top 10% ( z > +1.28) B = next 20% (+0.53 < z < +1.28) C = central 40% (-0.53 < z < +0.53) D = next 20% (-1.28 < z < -0.53) F = bottom 10% ( z < -1.28)

18. ECTS = European Credit Transfer System

19. Norm-referenced Evaluation  Pros: - Ensures a “spread” between top and bottom of the class for clear grade setting - Shows student performance relative to group  Cons: Top and bottom performances can sometimes be very close Dispenses with absolute criteria for performance Being above average does not necessarily imply “A” performance In a group with great performance, some will be ensured an “F.”

20. Criterion and Norm Evaluation  Norm-Referenced Ensures a competitive classroom atmosphere Assumes a standard normal distribution Small-group statistics a problem Assumes “this” class like all others  Criterion-Referenced Allows for a cooperative classroom atmosphere No assumptions about form of distribution Small-group statistics not a problem Difficult to know just where to set criteria

21. Academic Grading ! In Sweden 1 Country = 1 Grading Scale Need to be harmonized

22. WHAT do we measure ? To test not only presence of knowledge …but also the application of knowledge In principle…

23. Model of clinical assessment  Certification of competence - pass / fail a state (and legal) requirement  Grading in rank order for employment / placement purposes  A competition for the award of a prize In principle…a three-fold aim

24. Model of clinical assessment Measurement of:  adequacy of basic clinical skills  ability to interpret clinical findings  facility of communication in practical settings  ability to think analytically about diagnosis  ability to discuss management logically In practice…

25. Practical steps for assessment  At least 6 encounters with different clinical situations (Internal Medicine, Surgery, Pediatrics, Ob-Gyn,…)  At least 2 examiners at every encounter, each examiner giving an individual assessment  Highly structured examination and detailed assessment of skills  Examiners from other Universities for process evaluation and quality control

26. Set-points for Testing  Attitude to patient  Actual examination skills  Presentation of findings  Clinical judgment Clinical examination of patients

27. Methodology-related Issues  Reliability  Validity  Practicality  Transparency

28. Organisational Issues  Resources available  Student numbers  Patient availability  Invasive procedures  Scheduling

29. Role of Grading  Is there need for grading ?  Selection process for employment  Selection for postgraduate training  Quality of assessment method and performance of candidates

30. Final question : What happens to candidates who fail ?  Review of performance – a formative exercise  Counselling at a personal level  Specific attention and individual training  Repeat assessment after a period of time

31. In conclusion I.Assessment of student (a future peer) is a continuous and complex process II.Assessment of clinical skills is Teacher, Medical School, National and Culture-Dependent III.There is place for widespread application of agreed standard methods to assess - clinical competences in core curricula - additional competences essential to individual practice IV.Teachers must give example of what are Knowledge, Know-How and Life-Skills