1. 1 IMPROVEMENT OF RELIABILITY, VALIDITY, AND FEASIBILITY OF THE OBJECTIVE STRUCTURED CLINICAL EXAMINATION (OSCE) WHEN ASSESSING THE PROBLEM SOLVING SKILLS OF FINAL YEAR SURGICAL CLERKSHIP Mohammed Y. Al-Naami*, Omar El Farouq, Gamal Khairy, Safdar Mofti, and Nawaz Anjum Gamal Khairy, Safdar Mofti, and Nawaz Anjum

2. 2 BIO-DATA* Mohammed Y. Al Naami MBBS (1986),FRCSC (1991), M Ed. (1992) Associate Professor Head, General Surgery Division Director, Faculty Development Program College of Medicine, King Saud University Riyadh, Saudi Arabia

3. 3 INTRODUCTION King Saud University founded in 1957. King Saud University founded in 1957. College of Medicine founded in 1967. College of Medicine founded in 1967. Affiliated King Abdulaziz University Hospital # of beds = 104 Affiliated King Abdulaziz University Hospital # of beds = 104 Affiliated King Khalid University Hospital # of beds = 785. Affiliated King Khalid University Hospital # of beds = 785. Number of students admitted annually approximately 300. Number of students admitted annually approximately 300. Traditional 6 years Annual Curriculum. Traditional 6 years Annual Curriculum.

4. 4 INTRODUCTION Department of Surgery has 2 undergraduate courses: 1. Basic Surgery - during 3 rd year - Clinical: 6 weeks Lectures: 4 weeks. 2. Clinical surgery - during 5 th year - Clinics,OR’s, Bedside, Rounds & Tutorials – 10 weeks Number of students averages 100 per course. Clinical examinations (1 long & 2 short cases X many years) were associated with rising dissatisfaction. First OSCE was introduced in May 2005 (r=0.68-0.79), validity at least better than clinical examinations, and feasibility was very low.

5. 5 AIM To improve previous courses’ OSCEs To improve previous courses’ OSCEs by using more problem solving skills by using more problem solving skills stations stations

6. 6 METHODOLOGY 2 mornings OSCE involving 48 and 47 students in days 1 and 2 respectively. Each morning students were split into 2 halves examining them simultaneously through 5 minutes x 24 stations in 2 separate surgical wards (120 minutes / OSCE). 15 active and 9 rest OSCE stations - stability, internal consistency, and concurrent validity were determined using BMPD® statistical software.

7. 7 Table 1: OSCE stations reliability (Day1) Surgical Specialty SMC-Carmine’s Theta (Internal consistency) Cronbach’s Alpha (Stability)DescriptionCompetency Anesthesia0.993360.8740 Laryngoscopy and intubation (PS) 1 Thoracic Surgery 0.985640.8735 Intercostal chest tubing system(PS) 2 General Surgery 0.994480.8712 Incisional hernia (Phx) 3 General Surgery 0.996220.8703 Abdominal examination (Phx) 4 Neurosurgery0.832450.8742 Subarachnoid hemorrhage (PS) 5 Pediatric Surgery 0.573590.8907 Lower intestinal obstruction (PS) 6 General Surgery 0.626880.8948 Small bowel obstruction (PS) 7 Vascular Surgery 0.484650.8894 Peripheral vascular occlusive disease (Hx-CS) 8 Pediatric Urology 0.672350.8901 Nephrostomy drain for pelvi- ureteral obstruction (PS) 9 Adult Urology 0.541260.8961 Hematuria (Hx-CS) 10 General Surgery 0.609790.8818 End-Colostomy (PS) 11 General Surgery 0.450370.8861 Parotid tumor picture (PS) 12 General Surgery 0.576380.8789 Triple lumen catheter (PS) 13 General Surgery 0.999720.8679 Gallstones Jar (PS) 14 Plastic Surgery 0.999710.8678 Burn picture (PS) 15 0.760.87 Total Average SMC,Squared Multiple Correlations; PS, Problem Solving; Phx, Physical Examination; Hx-CS, History-Communication Skills. RESULTS: Table 1: OSCE stations reliability (Day 1)

8. 8 Results: Table 2: OSCE stations reliability (Day 2) Surgical Specialty SMC-Carmine’s Theta (Internal consistency) Cronbach’s Alpha (Stability) DescriptionCompetency Anesthesia0.992870.8818 Spinal anesthesia picture (PS) 1 Thoracic Surgery 0.984920.8818 Pneumothorax chest x-ray (PS) 2 General Surgery 0.994450.8768 Inguinal hernia (Phx) 3 General Surgery 0.995920.8771 Abdominal examination (Phx) 4 Neurosurgery0.722310.8805 MRI-multiple brain lesions (PS) 5 Pediatric Surgery 0.734070.8817 Doudenal atresia X-ray (PS) 6 General Surgery 0.474350.9023 Colon cancer-barium enema (PS) 7 Vascular Surgery 0.567680.8874 Peripheral occlusive vascular disease (Phx) 8 General Surgery 0.710590.8848 Nasogastric tube (PS) 9 Adult Urology 0.643240.8932 Hematuria (Hx-CS) 10 Plastic Surgery 0.580380.8855 Clift lip picture (PS) 11 General Surgery 0.743200.8764 Lateral neck mass picture (PS) 12 Adult Urology 0.700750.8860 Lower urinary tract syndrome (Hx- CS) 13 General Surgery 0.999670.8705 Foley’s catheter (PS) 14 General Surgery 0.999790.8704 Appendix jar (PS) 15 0.790.88 Total average SMC,Squared Multiple Correlations; PS, Problem Solving; Phx, Physical Examination; Hx-CS, History-Communication Skills.

9. 9 VALIDITY Concurrent validity of the OSCE was determined by Pearson’s correlation with the written one best answer MCQs (P=0.65). Content and face validities were considered good by majority of staff and students. Predictive and construct validities were not addressed in this study.

10. 10 FEASIBILITY Initially our OSCEs were labor-intensive Mostly related to inadequate management. However; with repetitions and experience, subsequent OSCEs were more feasible, especially when assessing problem solving skills.

11. 11 DISCUSSION Since its introduction in 1979 by Professors R. Harden & F. Gleeson, OSCEs have developed into different forms depending on the desired competencies to be rested. Our OSCE addressed problem solving skills mainly OSCE preparation including staff commitment, blueprint, stations development, etc. is of paramount importance

12. 12 DISCUSSION The measure of reliability, validity, and feasibility are keys to successful examination. To achieve high reliabilities OSCE has to be longer than is often practical (D Newbel 2004). However, high reliabilities have been achieved by some authors with shorter OSCEs (Cohen R et al.1990; Stillman P et al.1991; Sloan D et al. 1993; and others) Validity is the hallmark of OSCEs despite arguments that encounters are short.( Stillman P et al.; Newbel D et al.) Feasibility is the most critiqued aspect of OSCEs as being costly and labor-intensive. (Barman A 2005)

13. 13 CONCLUSIONS Clinical competence encompasses different aspects that no single examination can cover with high degrees of reliability and validity. Well designed OSCE can assess most of these aspects with variable degrees of reliability and validity. OSCE can be highly reliable and valid when testing problem solving skills. OSCE is more practical when used for testing problem solving skills.

14. 14 THANK YOU