1 Assessment of Students Competence in Health Professionals Education Professor Hossam Hamdy University of Sharjah

2 Professional Competence The Habitual and Judicious use of: Communication Knowledge Technical Skills Evidence-based decision-making Emotions Values and reflection to improve the health of the individual patient and the community

3 WHAT Should Be Assessed? ACGME Competencies - Patient care - Knowledge - Practice-based learning and improvement - Interpersonal and communication skills - Ethics & Professionalism - System-based practice

4 Blueprinting Test content matches objectives Assessment matches competencies learnt Assessment matches format of learning, “ PBL ”

5 Aim of Blueprinting Reduce: A.Construct under-representation: Biased sampling Few items covering a topic e.g. OSCE B.Construct irrelevant variance: Flawed items format Inadequate sampling of student Behaviour.

6 Domain of Interest  Test Sample

7 Blueprinting Identify clinical problems that examinees should be able to handle. For each problem, define the clinical tasks in which the examinee is expected to be competent. Define level of performance / resolution

8 Blueprinting Prepare a blueprint to guide the selection of problems to be included in the assessment. Let the characteristics of the clinical problems define the tasks to be included in the assessment.

A Simple Blueprint Competence Categories CVSRS … GI …..MSSEndo/ Metab Haem / Oncology ….…. History Taking Physical Exam Tests and Procedures ….. Management Communicatio n and Patient Education

Pre-Clerkship Phase Theme B.Sc. Exam Blueprint Human behavior, ethical & medico legal principle s Interventi on prevention & managem ent principles Investi gative proced ure & interpre tation Immun ological & defense mechan isms Injury, inflammati on & infection; loss, repair and substitutio n; neoplasia Respons e and adaptati on systems & mechani sms Control, regulation & communication within the body, Homeostasis: metabolie, circulatory, acid/base & electrolyte Structur e: function ; position al relation ship Nurtur e, enviro nment & nutriti on Inherita nce, growth & develop ment; degener ation & aging UNITS UNIT VIII: Integrated Multi-System UNIT VII: Nervous System, Special Sense & Human Behavior UNIT VI: Integumentary & Muscular-Skeletal UNIT V: Hematopoietic & Immune Systems UNIT IV: Endocrine, Metabolism & Reproductive System UNIT III: GIT & Renal System UNIT II: Respiratory & CVS UNIT I: Concepts & Principles & Community Health

11

12 A Simple Model of Competence Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. Knows Shows how Knows how Does Professional authenticity Written, Oral or Computer based assessment Performance or hands on assessment

13 Validity Climbing the Pyramid... Knows Shows how Knows how Does Knows Factual tests: MCQ, essay type, oral….. Shows how Performance assessment in vitro: OSCE, SP-based test….. Does Performance assessment in vivo: Masked SPs, Video, Audits….. Knows how (Clinical) Context based tests: MCQ, essay type, oral…..

14 Sample Good Question & ‘Cover The Options’ Rule A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barr é syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis

15 Sample Good Question & ‘Cover The Options’ Rule A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barré syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis

16 Assessment of Doctor’s Performance Fundamental to “ Good Medical Practice ” “ What doctors do in controlled representations ” of practice e.g. “ OSCE ” “ What doctors do in real life ” Rethans et al, Med Ed 2002.

17 Unobserved 30 – 45 minute interview and examination on a selected patient Candidates present only their findings to the examiner The Traditional Long Case Examinations (LCE)

18 It assesses the integrated interaction between the student/doctor and the real patient Authenticity high More valid than the task given in an OSCE Little is known about the construct validity and consequential validity “ Educational Impact ” (LCE) Validity

19 Poor intercase reliability Content specificity is the most crucial issue in the assessment of clinical competence Broad sampling across cases is essential “ Multiple Biopsies ” Logistics will be difficult (LCE) Reliability

20 Clinical Encounter Method Mini – CEX “ Norcini 2003 ” components of the encounter observed Direct Observation Clinical Encounter Examination “ DOCE ” Entire encounter observed “ Hamdy 2000 ”

21 Three hour examination of four 45 minutes observed long cases Real patient selected from a predetermined blue print of common diseases Check List – clinical encounter domains Reliability 0.8 (Cronbach ’ s alpha) (Hamdy et al, Med Educ, 2003) Direct Observation Clinical Encounter Examination The DOCEE

22 Practical Procedural Skills Method Direct Observation Practical Procedure Skills “ DOPS ” Whole procedure observed from start to finish

23 Multi-source Feedback (MSF) 360 Degree Assessment Peers Supervisors Nurses Secretary Lab Technicians Patients

24 Problems of Workplace Based Assessment of Performance Sampling: variability in the complexity of patients problem Judges errors: Time: Requires adequate allocation of time and resources. Management of Data: not easy Validity and Reliability: variable