Download presentation
Presentation is loading. Please wait.
1
Assessment of Students Competence in Health Professionals Education
Assessment of Studnets Competence in Health Professional Education Assessment of Students Competence in Health Professionals Education Professor Hossam Hamdy University of Sharjah
2
Five Basic Questions WHY? WHAT? HOW? WHEN? WHOM?
3
Purpose of Assessment Providing feedback Guiding student learning
Insuring medical competence
4
WHY? Consider assessment as a diagnostic test for the disease “medical incompetence” False negative = failing a competent student False positive = passing a student who is incompetent
5
TRUTH Incompetent Competent F.P T.P Pass T.N F.N Fail TEST
6
Professional Competence
The Habitual and Judicious use of: Communication Knowledge Technical Skills Evidence-based decision-making Emotions Values and reflection to improve the health of the individual patient and the community
7
WHAT Should Be Assessed?
ACGME Competencies Patient care Knowledge Practice-based learning and improvement Interpersonal and communication skills Ethics & Professionalism System-based practice
8
Blueprinting Test content matches objectives
Assessment matches competencies learnt Assessment matches format of learning, “PBL”
9
Aim of Blueprinting Reduce: A. Construct under-representation:
Biased sampling Few items covering a topic e.g. OSCE B. Construct irrelevant variance: Flawed items format Inadequate sampling of student Behaviour.
10
Blueprinting Identify clinical problems that examinees should be able to handle. For each problem, define the clinical tasks in which the examinee is expected to be competent. Define level of performance / resolution
11
Blueprinting Prepare a blueprint to guide the selection of problems to be included in the assessment. Let the characteristics of the clinical problems define the tasks to be included in the assessment.
12
Assessment of Studnets Competence in Health Professional Education
A Simple Blueprint Competence Categories CVS RS … GI ….. MSS Endo/ Metab Haem / Oncology …. History Taking Physical Exam Tests and Procedures Management Communication and Patient Education
13
Pre-Clerkship Phase Theme B.Sc. Exam Blueprint 2003-2004
Human behavior, ethical & medico legal principles Intervention prevention & management principles Investigative procedure & interpretation Immunological & defense mechanisms Injury, inflammation & infection; loss, repair and substitution; neoplasia Response and adaptation systems & mechanisms Control, regulation & communication within the body, Homeostasis: metabolie, circulatory, acid/base & electrolyte Structure: function; positional relationship Nurture, environment & nutrition Inheritance, growth & development; degeneration & aging UNITS 11 10 9 8 7 6 5 4 3 2 1 UNIT VIII: Integrated Multi-System UNIT VII: Nervous System, Special Sense & Human Behavior UNIT VI: Integumentary & Muscular-Skeletal UNIT V: Hematopoietic & Immune Systems UNIT IV: Endocrine, Metabolism & Reproductive System UNIT III: GIT & Renal System UNIT II: Respiratory & CVS UNIT I: Concepts & Principles & Community Health
15
HOW? Choice of Method U = W R x W V x W E x W A x W C 1 2 3 4 5
U = Utility R = Reliability V = Validity E = Educational Impact A = Acceptability C = Cost/Available Resources W = Weight
16
Reliability Reliability = reproducibility Inter-rater:
Multiple examiners across different cases Inter-case: “multi use format (OSCE)”
17
Reliability Reliability is matter of sampling Across content
Across Tasks Across Problems
18
Domain of Interest Test Sample Test Sample
19
“Reliability” Practical Implications
Sample adequately across content Clinical competencies are inconsistent across different tasks Test length is crucial if high-stakes decisions are required Use as many examiners as possible
20
Validity Validity = measures what it intends to measure
Select appropriate test formats for the competencies to be tested. This action invariably results in a composite examination.
21
A Simple Model of Competence
Assessment of Studnets Competence in Health Professional Education A Simple Model of Competence Professional authenticity Performance or hands on assessment Does Shows how Written, Oral or Computer based assessment Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.
22
Validity Climbing the Pyramid . . .
Does Performance assessment in vivo: Masked SPs, Video, Audits….. Does Shows how Performance assessment in vitro: OSCE, SP-based test….. Shows how Knows how (Clinical) Context based tests: MCQ, essay type, oral….. Knows how Knows Factual tests: MCQ, essay type, oral….. Knows
23
Educational impact
24
WHOM Self Peers Supervisors 360º “Different Stakeholders”
25
Written Assessment Methods
Context-free questions Stimulus Format Context-rich questions EMQ, KF, A & R “MCQ” Open ended questions types SAQ, MEQ Response Format MCQ types
26
The MCQ’s Family Dead Sick Alive K Type C Type
One single answer True & False x Type Long PMPs One Best Answer key Features Extended Matching
27
Written Assessment of Clinical Competence
MCQs 1) Best Single Response 2) Extended Matching Key Features Questions Constructed Response Questions
28
Components of an A-type Question
Assessment of Studnets Competence in Health Professional Education Components of an A-type Question A 35-year old man has had a stomach ache all afternoon. He ate the following lunch: two big McDonalds hamburgers, an ice cream shake, large fries. Vignette, Scenario, Or Stem Which is the most likely diagnosis? Lead-in
29
Components of an A-type Question
Abdominal aneurysm Appendicitis Bowel obstruction Cholecystitis Colon cancer Pancreatitis Too much lunch Options A, B, C, D, E & F are distracters Option G is the key
30
Sample Good Question & ‘Cover The Options’ Rule
Assessment of Studnets Competence in Health Professional Education Sample Good Question & ‘Cover The Options’ Rule A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barré syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis
31
Sample Good Question & ‘Cover The Options’ Rule
Assessment of Studnets Competence in Health Professional Education Sample Good Question & ‘Cover The Options’ Rule A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barré syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis
32
Diagram of Rank-Ordered Options for a Good One-Best-Answer Item
Assessment of Studnets Competence in Health Professional Education Diagram of Rank-Ordered Options for a Good One-Best-Answer Item D C A E B Least Most Correct Correct Answer Answer Note: Options must be homogeneous (eg, all diagnoses, all muscles). You must be able to rank-order the options on the same dimension.
33
Sample Flawed Question & ‘Cover The Options’ Rule
Assessment of Studnets Competence in Health Professional Education Sample Flawed Question & ‘Cover The Options’ Rule Which of the following is true about pseudogout? A. It occurs frequently in women. B. It is seldom associated with acute pain in a joint. C. It may be associated with a finding of chondrocalcinosis. D. It is clearly hereditary in most cases. E. It responds well to treatment with allopurinol.
34
Sample Flawed One-Best-Answer Item
Assessment of Studnets Competence in Health Professional Education Sample Flawed One-Best-Answer Item Which of the following is true about pseudogout? A. It occurs frequently in women. B. It is seldom associated with acute pain in a joint. C. It may be associated with chondrocalcinosis. D. It is clearly hereditary in most cases. E. It responds well to treatment with allopurinol. D Inheritance Rx E Gender A Associations False True B C
35
Test application of Knowledge not just recall of isolated facts
Assessment of Studnets Competence in Health Professional Education Test application of Knowledge not just recall of isolated facts Basic Science Recall of Isolated Facts Item: What area is supplied with blood by the posterior inferior cerebellar artery? Basic Science Application of Knowledge Item: A 62-year-old man develops left-sided limb ataxia, Horner's syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. What artery is most likely to be occluded?
36
“Old” Style Biochemistry Question Testing Recall of an Isolated Fact
Assessment of Studnets Competence in Health Professional Education “Old” Style Biochemistry Question Testing Recall of an Isolated Fact Acute intermittent porphyria is the result of a defect in the biosynthetic pathway for A. collagen B. corticosteroid C. fatty acid D. glucose E. heme F thyroxin E
37
Application of Knowledge Item: Biochemistry/Genetics
Assessment of Studnets Competence in Health Professional Education Application of Knowledge Item: Biochemistry/Genetics An otherwise healthy 33-yo man has mild weakness & occasional episodes of steady, severe abdominal pain with some cramping but no diarrhea. His aunt and a cousin have had similar episodes. During an episode, his abdomen is distended, and bowel sounds are decreased. Neurological exam shows mild weakness in the upper arms. A defect in which of the following biosynthetic pathway is the most likely cause of the patient’s problem? A. collagen B. corticosteroid C. fatty acid D. glucose E. heme F. thyroxine E
38
Item Assessing Recall of an Isolated Fact
Assessment of Studnets Competence in Health Professional Education Item Assessing Recall of an Isolated Fact What is arterial blood gas analysis most likely to show in patients with cardiogenic shock? A. Hypoxemia with normal pH B. Metabolic acidosis C. Metabolic alkalosis D. Respiratory acidosis E. Respiratory alkalosis
39
Alternate Item Assessing Application of Knowledge: Diagnosis
Assessment of Studnets Competence in Health Professional Education Alternate Item Assessing Application of Knowledge: Diagnosis A 74-year-old woman is brought to the emergency department because of crushing chest pain. She is restless, confused, and diaphoretic….. Which of the following arterial blood gas analysis results is most likely? A. Hypoxemia with normal pH B. Metabolic acidosis C. Metabolic alkalosis D. Respiratory acidosis E. Respiratory alkalosis
40
Assessment of Studnets Competence in Health Professional Education
Summary Stimulus – Context rich “Patient Vignette” One best answer, not one single correct option Cover the options rule
41
Assessment of Studnets Competence in Health Professional Education
Summary Avoid: All / non of the above Except Negatively worded items. “Which one of the following statements is incorrect.”
42
Extended Matching R-Set
43
Theme: Neuroanatomy: Blood Supply of the Brain
Options: A. Left anterior cerebral artery E. Right anterior cerebral artery B. Left posterior cerebral artery F. Right posterior cerebral artery C. Left middle cerebral artery G. Right middle cerebral artery Left lenticulostriate arteries H. Right lenticulostriate arteries Lead-in: For each patient with neurologic abnormalities, select the artery that is most likely to be involved. Vignette: “Items” 1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response on the right, normal strength of the right arm, and normal facial movements. Ans: A 2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and paralysis of the lower two-thirds of his face on the right. His speech is fluent, and he has normal comprehension of verbal and written commands. Ans: G
44
Assessment of Studnets Competence in Health Professional Education
Options and Lead-in: Diagnosis Theme: Abdominal Pain A. Abdominal aneurysm K. Kidney stone B. Appendicitis L. Mesenteric adenitis C. Bowel obstruction M. Mesenteric artery thrombosis D. Cholecystitis N. Ovarian cyst - ruptured E. Colon cancer O. Pancreatitis F. Constipation P. Pelvic inflammatory disease G. Diverticulitis Q. Peptic ulcer disease H. Ectopic pregnancy – ruptured R. Perforated peptic ulcer I. Endometriosis S. Pyelonephritis J. Hernia T. Torsion For each patient with abdominal pain, select the most likely diagnosis.
45
Sample Diagnosis Items: Abdominal Pain
A 25-year-old woman has sudden onset of persistent right lower abdominal pain that is increasing in severity. She has nausea without vomiting. She had a normal bowel movement just before onset of pain. Examination shows exquisite deep tenderness to palpation in right lower abdomen with guarding but no rebound; bowel sounds are present. Pelvic examination shows a 7-cm, exquisitely tender right sided mass. Hematocrit is 32%. WBC is 18,000/mm3. Serum amylase activity is within normal limits. Test of the stool for occult blood is negative. Ans: B An 84-year-old man in a nursing home has increasing poorly localized lower abdominal pain recurring every 3-4 hours over the past 3 days. He has no nausea or vomiting; the last bowel movement was not recorded. Examination shows a soft abdomen with a palpable, slightly tender, lower left abdominal mass. Hematocrit is 28%. WBC is 10,000/mm3. Serum amylase activity is within normal limits. Test of the stool for occult blood is positive. Ans: E
46
Key – Feature Questions
Any single problem is not equivalent to any other problem in assessing C.P.S. and C.D.M. Problem solving skills are specific to the case or problem encountered. C.P.S. and C.D.M. need to be measured over many problems before general inferences could be made. Clinical problem solving and decision making skills are dependent on the effective manipulation of those few elements of the problem that are critical to its resolution – “key features” Assessment should focus on the problems key features “Bordage and Page, 1991”
47
Question Formats Write-in “WI” - Diagnosis and Treatment
Short-Menu “SM” - History, Physical “2-45 Options” Examination and Laboratory
48
Case 2 Question 1 Question 2
Paul, a 56 year old man, consults you in the outpatient clinic because of pain in his left leg which began two days ago and has been progressively worse. He states his leg is tender below the knee and swollen around the ankle. He has never had similar problems. His other leg is fine. Question 1 What diagnosis would you consider at this time? List up to three? Question 2 With respect to your diagnosis, what elements of his history would you particularly want to elicit? Activity at onset of symptoms Hormonetherapy Previous back problems Alcohol intake Impotence Previous neoplasia Allergies Intermittent claudication Previous urinary tract infection Angina pectoris Low back pain Recent dental procedure Anti-inflammatory therapy Nocturia Recent immobilization Cigarette smoking Palpitations Recent sore throat Colour of stools Parenthesis Recent surgery Cough Paro .. Noctural dyspnea Recent work environment Headache Polydipia Wounds on foot Hemetemesis Previous knee problems Wounds on hand
49
Constructed Response Questions
A written question A statement, journal article or report A problem scenario: clinical, basic science or community Photographic or other investigative data The response generated by the student may be:
50
Constructed Response Questions
A very short answer – one or several words A short answer – notes or lists A long answer – essay, dissertation, referral letter or report A physical task – a project or another complex performance such as cardiopulmonary resuscitation or conducting an investigation
51
A 72-year-old, right-handed man presents with a 6-month history of increasing tremor of his right hand that causes him severe embarrassment, such that he avoids going out. On examination, the tremor is most marked at rest and decreases on maintaining a posture and during movement. There is no intention tremor. Example questions: The name of this type of tremor The anatomical site of the pathological lesion Drugs causing this phenomenon as a side-effect Additional signs to look for on physical examination How to treat the condition
52
The Traditional Long Case Examinations (LCE)
Assessment of Studnets Competence in Health Professional Education The Traditional Long Case Examinations (LCE) Uninterrupted and observed 30 – 45 minute interview and examination on a selected patient Candidates present only their findings to the examiner
53
Assessment of Studnets Competence in Health Professional Education
(LCE) Validity It assesses the integrated interaction between the student/doctor and the real patient Authenticity high More valid than the task given in an OSCE Little is known about the construct validity and consequential validity “Educational Impact”
54
Assessment of Studnets Competence in Health Professional Education
(LCE) Reliability Poor intercase reliability Content specificity is the most crucial issue in the assessment of clinical competence Broad sampling across cases is essential “Multiple Biopsies” Logistics will be difficult
55
Direct Observation Clinical Encounter Examination The DOCEE
Assessment of Studnets Competence in Health Professional Education Direct Observation Clinical Encounter Examination The DOCEE Three hour examination of four 45 minutes observed long cases Real patient selected from a predetermined blue print of common diseases Check List – clinical encounter domains Reliability 0.8 (Cronbach’s alpha) (Hamdy et al, Med Educ, 2003)
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.