International Diabetes Federation (IDF) East Mediterranean and Middle East Region (EMME) Workshop on Professional Educational Methodology in Diabetes August 2008 Bibliotheca Alexandrina, Egypt

2 Professional Technology Standards in Educational Development by Prof. Morsi Arab

Evaluation

Written Exams :The Long Essay Questions Disadvantages: 1- Students do not know what is exactly required (vague) 2- Lack reliability ( different marks (scores) by different examiners). 3- Usually not valid (in their future job they are not going to write an article, but expected to do some task). 4- Take long time to mark ( if well done ?). 5- Will not give any feedback to the students. 6- Covers a limited part of the curriculum.

To improve reliability of L. Essay questions: 1- Give a well defined, structured question. 2- Prepare model answer and define in advance: - the main points. - the marks allocated for each point ( scoring ). 3- Explain to students this plan of scoring. 4- More than one examiner. 5- All students subjected to the same examiner for each part of the exam (e.g. Q 4 ).

Essay Questions : Minimal advantages : 1- Very easy to prepare ( for the examiner). 2- Tests ability of student to construct an article. Conclusion: - Better not to use it. - If necessary try to improve.

Multiple Short Essay Questions Advantages: 1- covers a large part of the curriculum 2- less unreliable 3- answered in shorter time ( by students). 4- also marked in shorter time ( by examiner). Disadvantages: Still tests more knowledge and no skills or application of the knowledge in solving problems ( taking decisions).

The MCQ Type 1: Short stem + a number of choices Example: During a hypoglycemic episode : (a) there may be excessive sweating √ × (b) palpitations √ × (c ) there is hypotension √ × (d) very dry mouth √ × (e)……………………………………….. √ × N.B. Here all choices should be taken

Multiple Choice Questions (MCQ) Type 2: one stem +4-6 (usually 5) choices - Example : After describinag some clinica features …………..what is the most probable diagnosis ? (NB To answer select only one. ( a )………………………… Answer: e.g. (c) ( b )………………………… ( c )…………………………. ( d )…………………………. ( e )……………………………

Advantages of MCQ Test - scored in a short time and precisely. - if numerous, covers widely the syllabus. - can be used for self assessment. - high reliability. Disadvantages : - difficult to prepare good MCQs. - takes a long time to prepare, - usually test only knowledge (rarely skills e.g. in decision making ).

Scoring of MCQ - Time : give 2 minutes for each (√ ×) Q with 5 choices. So, for 30 Qs give 30 min. to answer. - Penalty: give +1 for each” correct” answer, 0 for “no answer” & -1 for incorrect answer ( penalty is needed in the type with + 5 Yes/No choices). (no need for penalty in the type of stem + select one choice). - The score to pass: the pass mark should be high e.g. 80% or 90%. - it is better to use easy Qs with high pass % (80-90% than difficult Qs with low pass % (50-60%). - For quick scoring punched paper may be applied.

Problem Solving Questions (To be used best for evaluation of competence in decision making ). * Example : give a clinical situation followed by an MCQ on the best decision to be taken 1-………………………. 2-……………………… 3-………………………. Then add more information followed by more MCQ decision taking questions : 1-………… 2-……………………… 3-……………………… ( cont. )

Problem Solving Q ( cont. ) Advantages : 1- High validity (simulates a clinical situation) 2- high reliability with a good preplan for scoring ** 3- short time to mark (score). **Preplan for scoring write down all expected student responses (both correct and incorrect ) and decide marks to be given for each response.

Oral Examination - By one or more examiners : - Forms : -what do you know about? - what would you do if ……. ( take decision ). - Advantages (few): vivid, can probe increasing depth of knowledge (escalate). - Disadvantages: - induces anxiety to students (stress not existing in practical life). - consumes long time. - unreliable - can not test skills (v. rarely). - does not give good FB to the education process. Conclusion : better avoided

Evaluation by Check Lists - very useful in clinical and practical examinations (Testing skills) - Example: during a particular performance, the student’s actions are observed by the examiner to score for each appropriately performed step. 1-……………………………………………… √ 2-……………………………………………… √ 3-……………………………………………… √ 4-……………………………………………… √ 5-……………………………………………… √ 6-……………………………………………… √ 7-……………………………………………… √ 8-……………………………………………… √ 9-……………………………………………… √ etc. - High reliability - Gives good feed-back to teachers. (e.g., if all the students miss a particular step or do it with bad performance reconsider the training course.

Check List / Rating scale for measuring attitudes (behaviours) of a person during his training or working (job) × × ×××× × 1- shows interest 2- accepts instructions 3- cares for patients 4- shows desire to learn 5- keeps on time 0: strongly lacking…………5: strongly agree/ evident

Evaluation of behaviours during group training (by Check List/Rating Scale) - to report attitudes by observing specific behaviours of different candidates. ( have good relations with patients, accurate, punctual, organized. keeps good records,etc) - it can be used to guide the trainees along the learning process. - it is used for final accreditation. - it guides the educators (FB) the next training course

The Continuous evaluation ( during a training course,e.g. every week, month or term.) Advantages: - lowers the risk from one faulty assessment. - lowers apprehension of a one final test. - encourages students to learn (study) all through the year. - guides the student to improve his study. Disadvantages: - takes a long time and effort in preparation of frequent exams. What form could be different : written test, achievement of skills, observation of attitudes…etc. Evaluation for self assessment.

Record of Satisfactory Performance during a training course for a group * The Record contains a number of tasks enlisted (1,2,3 …). * Each trainee (A,B,C,…) should fulfill the performance of all tasks. * Satisfactory achievement of each task ( separately ) is monitored. * Unsatisfactory performance indicates repeat training for a specific individual or for the whole group

Tasks Trainee DC(B)A X 1 X 2 XX 3 X 4 X 5 XXXX (6)