Assalam alaikum and Good Morning

Clinical Examiners’ Reflection about a newly developed eOSCE System: An innovative experience at Alfaisal University Dr. Nasir Ali Afsar MBBS, M.Phil, Ph.D. Department of Pharmacology, College of Medicine, Alfaisal University, Riyadh, Saudi Arabia. Honourable Chair_, ladies & gentlemen, It’s a privilege to address you in beautiful Marrakesh. My topic is ……... Where, after briefly touching upon the basic jargons, I will walk you through a short history of what we were doing and ultimately evolved cost-effectively and indigenously based on our needs.

Introduction What is an Objective Structured Clinical Examinations (OSCEs)? Examiners evaluate students performing clinical tasks, such as on simulated patients or mannequins. Key method to assess learning of clinical skills. 17 Jun 2015 ICTHE 2015, Hivernage

Advantages of an OSCE An evaluation that is, relevant effective in a realistic situation Prevents patients’ risks as well as suffering due to a long examination session. 17 Jun 2015 ICTHE 2015, Hivernage

Limitations of an OSCE A resource-intensive activity, where we need to arrange for, Venue, examiners, standardized patients, mannequins, data entry, evaluation ± feedback, organization, running cost etc. Providing timely and personal feedback to student may be difficult. 17 Jun 2015 ICTHE 2015, Hivernage

Global Problem, Local Perspective College of Medicine at Alfaisal University Riyadh, Saudi Arabia. Regular use of OSCEs in undergraduate and postgraduate clinical examinations. Management of an OSCE: Pre-OSCE preparation Conduction issues: limited space Evaluation issues: validity of evaluation, subjective bias Feedback issues: large number of students Alfaisal University is an IT-friendly institution where both students and faculty extensively use various IT services through intra- and internet. At CoM AU, we conduct clinical skills courses integrated with various organ-system modules, starting from Year-2 Medicine Program. OSCE is regularly used to evaluate students during and/or at the end of the course. However, managing an OSCE for almost 1000 students is challenging. We faced issues at all stages, ie PCEF. With a small team dealing with clinical skills courses there was no choice but to consistently look for enhancing our efficiency in one way or the other. Interestingly, I was a “permanent” member of that team and went through all different stages of evolution and hence here to tell the complete story, because, all other members, both senior and junior to me kept changing ! 17 Jun 2015 ICTHE 2015, Hivernage

Evolution of Process at AU To begin with, we started with the traditional way of paper-pencil OSCE, where examiners would evaluate each student with the help of a single-page-checklist. The students would bring their name and ID stickers (also prepared by ourselves) to the exam venue. Here is shown a portion of a typical OSCE checklist. But the whole exercise or organization would be very tedious, especially compiling data and its analysis. 17 Jun 2015 ICTHE 2015, Hivernage

Evolution of Process at AU What is next ? What is next? You would be familiar with this type of answer sheets used by students in different MCQ exams. However, we used this approach indigenously for examiners instead to evaluate the students. The biggest potential advantage would be quicker data entry. 17 Jun 2015 ICTHE 2015, Hivernage

Evolution of Process at AU Here, a portion of such a Scantron® -transformed checklist is shown. This too was developed indigenously because luckily we could not find anyone dealing with such customization. We used MS excel to design it and then printed them on a sheet of paper. It went well, but still required human interaction to run the sheets and analyze the data. Moreover, at one instance, which happened to be the last time we used it, a very small difference in paper size, so small that we could not appreciate it, led to scanning error and then all sheets which showed error had to be marked manually after the OSCE was over. Just imagine ! 17 Jun 2015 ICTHE 2015, Hivernage

Evolution of Process at AU From paper-pencil OSCE to Scantron® OSCE: Resource-intense, error-prone, cumbersome Finally, an electronic OSCE (eOSCE) was developed utilizing the updated web-based Electronic Learning System at Alfaisal University. 17 Jun 2015 ICTHE 2015, Hivernage

Methods Development of eOSCE application: helped by Alfaisal IT Department. Testing and improvement Examiner Training Post-eOSCE Examiner Survey (POES) conducted anonymously 17 Jun 2015 ICTHE 2015, Hivernage

Methods eOSCE administered in five OSCEs in three different undergraduate courses (Fall 2014, years 1-3). About 550 medical students were evaluated (more than 950 time-limited evaluation instances). Clinical history taking and/or physical examination. Examiners accessed the online OSCE application using laptops provided to them. 17 Jun 2015 ICTHE 2015, Hivernage

Typical eOSCE Interface – browser based function 17 Jun 2015 ICTHE 2015, Hivernage

Typical eOSCE Interface – individual evaluation 17 Jun 2015 ICTHE 2015, Hivernage

Methods – POES All examiners included (n=22) Excluded; incomplete response (n=3) Included examiners (n=19) Seniors: Experience > 1 Year (n=10) Juniors: Experience ≤ 1 Year (n=9) 17 Jun 2015 ICTHE 2015, Hivernage

Results * * Significant difference (p = 0.03) 17 Jun 2015 ICTHE 2015, Hivernage

Results All examiners Preferred eOSCE, mainly due to convenience and automation. Found the system easy-to-learn Wished to use the system again Showed motivation to share their experience, Junior examiners showed higher enthusiasm (p=0.03). 17 Jun 2015 ICTHE 2015, Hivernage

Results The managers of the eOSCE appreciated this program for Ease of use Cost effectiveness Easy compilation of results Feasibility of personalized feedback to students. Dependence on technology was a perceived disadvantage. 17 Jun 2015 ICTHE 2015, Hivernage

Conclusions The eOSCE was a convenient and cost- effective application. The faculty showed a positive response towards its preferential use, especially the younger generation of examiners. 17 Jun 2015 ICTHE 2015, Hivernage

Our Clinical Skills Team Dr. Nasir Ali Afsar, Department of Pharmacology. Dr. Muhammad Zafar, Department of Anatomy and Clinical Skills. Dr. Amna Siddiqui, Department of Anatomy. SPECIAL THANKS TO Dr. M. Faisal Ikram, Assessment Office, helped to develop Scantron® OSCE Mr. Khyam Shahzad, IT department, helped to develop eOSCE application 17 Jun 2015 ICTHE 2015, Hivernage

For queries write to: nafsar@alfaisal.edu THANK YOU 17 Jun 2015 ICTHE 2015, Hivernage