1. Gordon Page University of British Columbia Train the Trainer Workshop October 29, 2003 Key Features Hong Kong International Consortium

2. What are key feature problems: what do they assess? Why use key features problems: clinical competence/expertise issues? Why use key features problems: measurement/assessment issues? How do you write key features problems? Overview

3. What are key feature problems : what do they assess? Why use key features problems : clinical competence/expertise issues? Why use key features problems : measurement/assessment issues? How do you write key features problems? Overview

5. 1. What clinical problems would you focus on in your immediate management? List up to three. 1.____________________________________ 2.____________________________________ 3.____________________________________

7. A paper and pencil clinical problem A case scenario typically followed by 2 or 3 questions Questions assess clinical decisions and actions (not underlying knowledge) Questions prepared only in relation to the critical elements ("key features") in the resolution of the problem What is a Key Features Problem?

9. 1. What clinical problems would you focus on in your immediate management? List up to three. 1.____________________________________ 2.____________________________________ 3.____________________________________

11. What are key feature problems : what do they assess? Why use key features problems : clinical competence/expertise issues? Why use key features problems : measurement/assessment issues? How do you write key features problems? Overview

12. Reduced: knowledge absent/disorganized Dispersed: many individual facts -- rote lists, little understanding/integration Elaborated: knowledge organized, integrated & structured – effective reasoning Compiled: knowledge structured around clinical prototypes; problem recognition and resolution through pattern recognition Clinical Decision-Making (Bordage, G. Academic Medicine, 1994, 1999)

13. Reduced “empty mind” Dispersed“cluttered mind” Elaborated “deductive thinking” Compiled “recall/pattern recognition” Clinical Decision-Making (Bordage, G. Academic Medicine, 1994, 1999)

14. Reduced Dispersed Elaborated Compiled Clinical Decision-Making

15. Inter-case correlation =.10 –.30 Each case presents unique challenges (Arthritis  Anemia  Crohns  Diabetes) Case Specificity

16. Key features problems Assess effectiveness, not thoroughness Assess only key steps (usually 2 – 4) in the resolution of a problem Permit wide sampling of problems to address “case specificity”

17. What are key feature problems: what do they assess? Why use key features problems: clinical competence/expertise issues? Why use key features problems: measurement/assessment issues? How do you write key features problems? Overview

18. Reliability – “Focussed" problems – better sampling, more accurate assessment Content Validity – assess the most important clinical decisions within (a representative sample of) problems Key features problems: measurement/assessment issues?

19. Application of Knowledge? Mrs. Stevens has a herniated disc in her cervical spine. What would you expect to find on your examination of Mrs. Stevens? What symptoms is Mrs. Stevens likely to be experiencing?

20. Top-Down Bottom-Up Problem Clinical Features

21. Reliability – “Focussed" problems – better sampling, more accurate assessment Content Validity – assessing the most important clinical decisions within (a representative sample of) problems “Bottom-up” thinking – assessing knowledge application and what clinicians do in real life! Why use key features (Key Features problems: measurement/assessment issues?)

22. Performance & Length of Option Lists PERCENT CORRECT SCORE PERCENT OF STUDENTS

23. Performance on Free Response vs Selected Response Items

24. Non Vignette What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function? (A)acute poststreptococcal glomerulonephritis (B)hemolytic-uremic syndrome *(C)minimal change nephrotic syndrome (D)nephrotic syndrome due to focal and segmental glomerulosclerosis (E)Schönlein-Henoch purpura with nephritis Short Vignette A 2-year-old boy has a 1-week history of edema. Blood pressure is 100/60 mm Hg, and there is generalized edema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis? Long Vignette A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58-9.

25. What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function? AB*C*DEOverall P-value 109900 81 90 1094 A 2-year-old boy has a 1-week history of edema. Blood pressure is 100/60 mm Hg, and there is generalized edema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis? 009820 52 82 8188 A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 019810 109 66 10584 Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58-9.

26. Reliability – “Focussed" problems – better sampling, more accurate assessment Content Validity – assessing the most important clinical decisions within (a representative sample of) problems “Bottom-up” thinking – assessing knowledge application and what clinicians do in real life! “Fidelity” & discriminating power – longer lists of options, free response questions, raw data in case scenarios Why use key features (Key features problems: measurement/assessment issues?)

27. Overview of the Presentation What are key feature problems: what do they assess? Why use key features problems: clinical competence/expertise issues? Why use key features problems: measurement/assessment issues? How do you write key features problems?

28. Key Features Problems Page, G. and Bordage, G. “Developing Key Features Problems and Examinations to Assess Clinical Decision- Making Skills”, Academic Medicine, 70:3, 1995, 194-201.

29. STEPS IN PREPARING KEY FEATURES CASES 1.Select a Clinical Problem (e.g., diarrhea) 2.Select a Clinical Situation

30. Step 2: Clinical situations How does problem present?  Undifferentiated complaint  Single problem, typical/atypical  Multiple problem, multi-system  Urgent, life-threatening problem  Prevention, health promotion

31. STEPS IN PREPARING KEY FEATURES CASES 1.Select a Clinical Problem (e.g., diarrhea) 2.Select a Clinical Situation (e.g., single problem, typical resentation)

32. STEPS IN PREPARING KEY FEATURES CASES 1.Select a Clinical Problem (e.g., diarrhea) 2.Select a Clinical Situation (e.g., single problem, typical resentation) 3.Define the problem's Key Features

33. Define the Problem’s “Key Features” The critical or essential steps in the resolution of the problem *********************************** The elements or steps most likely to lead to errors The difficult aspects of problem identification and management

34. Key Features Relative to the problem diarrhea, the clinician should: 1. Recognize dehydration and its level of severity. 2. Manage dehydration appropriately. 3. Evaluate the possible communicability of the underlying disease (family/hospital spread, possible common source).

35. STEPS IN PREPARING KEY FEATURES CASES 1.Select a Clinical Problem (e.g., diarrhea) 2.Select a Clinical Situation (e.g., single problem, typical resentation) 3.Define the problem's Key Features 4.Select a Case to represent the Problem

36. Key Features Case: Diarrhea A 35-year-old mother of three presents to your office at 1700h with complaints of severe, watery diarrhea. On questioning, she indicates that she has been ill for about 24 hours. She has had fifteen watery bowel movements in the past 24 hours, has been nauseated, but not vomited. She works during the day as a cook in a long-term care facility but left work to come to your office. On her chart, your office nurse notes a resting blood pressure of 105/50 mm Hg supine, (a pulse of 110/minute), 90/40 standing, and an oral temperature of 36.8°C. On physical examination, you find she has dry mucous membranes and active bowel sounds. A urinalysis (urine microscopy) was normal, with a specific gravity of 1.030.

37. STEPS IN PREPARING KEY FEATURES CASES 1.Select a Clinical Problem (e.g., diarrhea) 2.Select a Clinical Situation (e.g., single problem, typical presentation) 3.Define the problem's Key Features 4.Select a Case to represent the Problem 5. Develop Questions to test only the key features 6.Select the Format for the questions

38. 1. What clinical problems would you focus on in your immediate management of this patient? List up to three. 1.____________________________________ 2.____________________________________ 3.____________________________________

39. 2. How should you treat this patient at this time? Select up to three. 1.Antidiarrheal medication 2.Antiemetic medication 3.Intravenous 0.9% NaCl 4.Intravenous 2/3 to 1/3 5.Intravenous gentamicin 6.Intravenous metronidazole 7.Intravenous Ringer lactate 8.Nasogastric tube and suction 9.Nothing by mouth 10.Oral ampicillin 11.Oral chloramphenicol 12.Oral fluids 13.Rectal tube 14.Send home with close follow-up 15.Surgical consultation 16.Transfer to hospital

40. 3. After management of the patient's acute condition, what additional measures, if any, would you take? Select up to four or select #11, None, if none is indicated. 1.Avoid dairy products 2.Colonoscopy 3.Enteric precautions 4.Gastroenterology consultation 5.Give immune serum globulin to patients at long-term care facility 6.Infectious disease consultation 7.Notify Public Health Authority 8.Stool cultures 9.Strict isolation of patient 10.Temporary absence from work 11. None

41. STEPS IN PREPARING KEY FEATURES CASES 1.Select a Clinical Problem (e.g., diarrhea) 2.Select a Clinical Situation (e.g., single problem, typical presentation) 3.Define the problem's Key Features 4.Select a Case to represent the Problem 5. Develop Questions to test only the key features 6.Select the Format for the questions 7.Prepare Scoring Keys

42. Question 1 Scoring Key ScoreKeyed Responses Synonyms 1.0Dehydration Hypovolemia fluid loss fluid depletion 0Listing more than three items

43. Question 3 Scoring Key ScoreKeyed Responses 0.25 each# 3. Enteric precautions # 8. Notify Public Health Authority #11. Stool cultures #13. Temporary absence from work 0# 5. Give immune serum globulin to patients at long-term care facility #12. Strict isolation of patient or Selecting more than four items

44. STEPS IN PREPARING KEY FEATURES CASES 1.Select a Clinical Problem (e.g., diarrhea) 2.Select a Clinical Situation (e.g., single problem, typical presentation) 3.Define the problem's Key Features 4.Select a Case to represent the Problem 5. Develop Questions to test only the key features 6.Select the Format for the questions 7.Prepare Scoring Keys

45. Adequate sampling across problems A focus on key steps in problem resolution (vs. thoroughness) Questions/question formats that present “high fidelity” clinical challenges “bottom-up” application of knowledge free response questions longer option lists presentation of “raw” clinical data The assessment of clinical decision-making is enhanced by: Summary