TECHNOLOGY ENHANCED REMOTE ASSESSMENT Deans Award for Innovation 2016 Marie Morris, Ph.D. MSc. BSc. (HONS), DPSN - Teaching, RNT, RGN  

Collaborators Ms Amy Gillis Mr Donal O` Connor Professor Kevin C Conlon Professor Paul F Ridgway

Ethical Considerations The School of Medicine, Faculty of Health Sciences Research and Ethics Committee University of Dublin Trinity College approved this project. Written informed consent was gained from all participants to utilise the data collected for research and publication.

Assessment Governance GMC Domains of Professional Practice (Adapted Irish Medical Council, 2014)

Tuning Project, 2008

Bologna Process – Freedom of Movement Tuning Project, 2008

Tuning Project , 2008

Millers pyramid of assessment (Miller, 1990) Assessment Model Millers pyramid of assessment (Miller, 1990)

Assessment - Term Definitions Validity Construct Validity Reliability Sensitivity Specificity The concept of test validity is to what extent an assessment actually measures what it is intended to measure and permits appropriate generalisations about student’s skills and abilities. (Norman, 1991). The ability to differentiate between groups with known differences in ability - beginners versus intermediates versus experts. Refers to consistency of a test over time over different cases (inter- case) and different examiners (inter-rater). Reliability is measured as a correlation with 1.0 being perfect reliability and below 0.5 as unreliable. Evaluation measurements reliabilities above 0.65 or preferably above 0.85 are recommended. (Norman, 1991) Sensitivity, also called the true positive rate, measures the proportion of positives that are correctly identified as such (e.g. the percentage of sick people who are correctly identified as having the condition). Specificity, also called the true negative rate, measures the proportion of negatives that are correctly identified as such (e.g. the percentage of healthy people who are correctly identified as not having the condition). Utility = reliability x validity x feasibility x acceptability x educational impact.

Assessment Challenges - Curriculum Requirement to provide comprehensive feedback Results should be a quality indicator and guide curriculum reform Results should reflect potential clinical behaviours Identification of students achieving below expected levels Assessments should be consistent, standardised and defendable Increasing class sizes Fottrell Report 2006 Reducing number of available experts Reduced working hours EWTD (2003/88/EC)

Assessment Challenges - Curriculum Limited availability of “Real“ patients Costs – Stimulated Patients, Venues, Staff

Assessment Challenges – Assessors Hawk Dove Complex area of assessor decision making in performance testing (Sadler, 2009; Govaerts, 2011; Kogan et al, 2011; Yorke, 2011). The literature reveals that the factors affecting assessor decision-making can be highly individualised, contextualised influenced by characteristics such as assessor experience and seniority (Pell & Roberts, 2006; Kogan et al, 2011). Halo Effect Horns Effect Central Tendency

Assessment – Students Perspective Accept direct observational is a core course requirement Desire for a fair assessment Passing should not be down to “luck” Transparent process Feedback should be timely Accept dual examiners superior but report that multiple examiners increases anxiety and stress

Novel Approaches to Technical Skills Assessment Brain activity was estimated by means of functional Near Infrared Spectroscopy (fNIRS). fNIRS is a non-invasive technique for optical estimation of cortical activity. fNIRS measurements were taken over the prefrontal cortex since this region is involved in cognitive processing34 Functional near infrared spectroscopy (fNIRS), (Leff et al, 2008; Ohuchida et al, 2009; Cui et al, 2011). Functional Magnetic Imaging (Morris et al, 2012 ; Bahrami et al, 2012).

Why use Technology in Assessment ? Technology savvy millennial generation students Promotes student engagement Facilitates student centred learning Furthers the objectives of the Trinity Education Project. Promotion of non-traditional modes of assessment Embeds 21st Century assessment modalities into the curriculum

R.I.T.A Undergraduate Setting Double RoboticsTM Telepresence Device Remote Interpretation of Technical Ability Double RoboticsTM Telepresence Device Five-foot tall tablet computer on wheels via Wi-Fi https://www.doublerobotics.com/education/

Logistics - Double RoboticsTM Controlled by the absent examiner via their mobile phone Remote assessor appears on the screen Conducts a face-to-face interaction with the student Student and patient can be seen up close Examiner can check for levels of competence in the students’ performance Remote assessor can quickly travel around the exam room

Feasibility Study Mixed methods approach – quantitative and qualitative An investigation into the levels of agreement (correlations) between two examiners assessments – one present, one remote Remote assessment via a tele-presence device Examiner, student and simulated patient evaluations

Alternative Clinical-Skills Examination “ACE”   Common cases (0.5 hour per station); Observed by examiners with GRS raters, standardised patient and materials (referral letters, imaging, investigation results)

Standardisation Content experts within the medical school who have experience with assessment at a local and national level developed the cases ensuring comparable complexity across cases. Each case was reviewed further by content experts and pilot-tested.

Data preparation and analysis Krippendorrf's Alpha - Inter-rater reliability tests - (Greene, 1999) to examine levels of agreement between independent examiners. Low correlation = < 0.59 Medium correlation = 0.6-0.79 High correlation = 0.8-1.0 Free text comments were analysed using the “cut and paste” technique (Richie, 2000)

Methods 48 - Year 3 medical students volunteered to participate in November 2016. The "ACE" format consisted of 4 sequential patient encounters observed by two independent examiners. One present and one remote (n=16). Examiners and students evaluated “RITA” using free text comments.

Standard Setting Exceeds Expectation – IH (5) and 2H (4) Marking Scale 1 – 5 Mark 3 out of 5 on GRS = readiness for Year 4 practice. Examiners must mark the student independent of each other. Professionalism and Patient Safety to be considered in overall grade Expectation regarding performance was Exceeds Expectation – IH (5) and 2H (4) Meets Expectation- Pass (3) Borderline- Safe but needs more practice (2) Below Expectation – Unsafe – Fail (1)

Feedback to students A copy of all examiners marking sheets across 5 encounters were supplied to all students within 3 days Yellow Sticker – issues with professionalism Red sticker – patient safety Large group – PowerPoint on common areas requiring improvement – within one week of the assessment Opportunity to meet for one to one feedback and discussion of gradings and examiners comments.

RESULTS – Quantitative Skill Result History taking 0.487 Physical Examination 0.458 Communication Skills 0.0714 Privacy/Dignity -0.182 Time Management Procedural Skills - Phlebotomy 0.711 Data Interpretation 0.425 Response to Patient queries 0.593 Over all Performance 0.795   (Ordinal Method) More generalisable and robust to imperfect data than Cronbach's Krippendorrf's Alpha

RESULTS – Quantitative 100% agreement between both Examiners regarding Pass/Fail grades. Where marks differed one examiner marked a 4 and the other 5. The remote examiners reduced grades according to the amount of prompting required. This was reflected in the wide variation in grades for time management. The remote assessor reduced grades where the patient was over exposed and exposed for longer periods than required for the physical examination. This was reflected in the wide variation in grades for privacy/dignity .

RESULTS – Qualitative Themes and Issues Identified by Remote Assessor Good Wi-Fi Access required Drainage of battery Picture Good – screen freezing Difficult to hear on occasion- student needs to talk to screen Difficult to move around bed on occasion – practice required Remote assessor observer rather than participant Consider a mounted device

RESULTS – Qualitative Themes and Issues Identified by Present Assessor Picture Good Occasional screen freezing – short lived Had little interaction with remote assessor as observing the student Needed to be aware of where the device was so as not to obstruct view Remote assessor observer rather than participant Consider a mounted device

RESULTS – Qualitative Themes and Issues Identified by Students Non- intrusive Better than 2 Examiners in the room Benefit of feedback from 2 Examiners Unsure re talking to the remote assessor

RESULTS – Qualitative Themes and Issues Identified by Simulated Patients Didn’t take much notice of device or remote assessor Remote assessor more about watching the student rather than interacting with us Need to ensure enough room around bed for device to move quickly Would a mounted device work better ?

Costings Purchase cost of double robotics device = 2,600 Euro per unit Includes Double 2, Charging Doc, Audio and Camera Kit, Driver Apps, AC Adapter 1 Year Warranty.

Conclusion Technology enhanced assessment is feasible. It is a non traditional assessment method - TEP It addresses the issue of limited access to subject experts. It reduces the cost of external examiners. A larger study is needed with more examiner pairings.

Future Larger study week commencing September 2017 Correlation Study – inter-rater reliability between examiner pairings 4 Telepresence devices 8 Examiners – 4 Remote, 4 Present Year 3 Students Checklists and procedural skills

OSCE Format

Checklist PLACEMENT OF ELECTRODES FOR 12 LEAD ECG AND ECG RECOGNITION Accurately identify Angle of Louis then 4th intercostal space 0.5 Place electrodes and state this is the position for V1 on left & V2 on right 0.5 State that they are identifying 5th intercostal space mid clavicular line 0.5 Accurately do this and state that this is the position for V4 0.5 Accurately identify the midpoint between V2 and V4 0.5 Place electrode and state that this is for V3 0.5 Accurately identify 5th intercostal space mid axillary line 0.5 Place electrode and state that this is for V6 0.5 Accurately identify anterior axillary line or midway between V4 and V6 0.5 Place electrode and state that this is for V5 0.5 Place arm leads correctly 0.5 Place leg leads correctly stating which is neutral 0.5 GLOBAL RATING FAIL BORDERLINE SATISFACTORY GOOD EXCELLENT

Future Reach of the Project Remote Ward Round - Professor PF Ridgway Significantly increases the number of patients seen each day. “As modern Consultants work in a number of institutions it is very useful to be able to dial in from wherever you are. “Device has enabled us to be more efficient in delivering diagnosis after assessment, reaching a decision to admit and initiating therapy. “Initial research has shown that patient acceptability has been very much higher than we expected. And this is not just with the younger age group where it might have been expected. Re: communication skills - ability to extend to five feet in height or drop down to eye level if the patient is lying in bed, can interact with patients at eye level at all times.

Future Reach of the Project Colleges & Universities Double is being used in higher education in many ways, including building classrooms with both online and in-person students, to enable Tutors to continue teaching when they move out of the building. House bound Students Double enables the remote student to join the classroom in real-time, interact with other students one-on-one, and participate in group discussion. The telepresence robot is a revolutionary tool for housebound students. https://youtu.be/KOLcTKhPEhE?t=10

Research Output Morris MC, Gallagher TK, Ridgway PF. Tools used to assess medical students competence in procedural skills at the end of a primary medical degree: a systematic review. Med Educ Online. 2012;17. doi: 10.3402/meo.v17i0.18398. Epub 2012 Aug 23. Review Morris MC, Frodl T, D'Souza A, Fagan AJ, Ridgway PF. Assessment of Competence in Surgical Skills Using Functional Magnetic Resonance Imaging: A Feasibility Study. J Surg Educ. 2014 Oct 25. pii: S1931-7204(14)00258-X. doi: 10.1016/j.jsurg.2014.09.007. M Morris, M Bennett, M Hennessy, KC Conlon, PF Ridgway. An Alternative Certification examination "ACE": Can post graduate methods be used to asses clinical skills in medical undergraduates? British Journal of Medicine and Medical Research. 3(4): 1806-1817, 2013 *Morris MC, Gillis AE, Smoothey CO, Hennessy M, Conlon KC, Ridgway PF. An alternative certification examination ("ACE") in surgery. J Surg Educ. 2014 Nov-Dec;71(6):779-89. doi: 10.1016/j.jsurg.2014.05.011. Epub 2014 Jul 11

Marie Morris, Amy Gillis, Nikita R. Bhatt, Paul F Ridgway Marie Morris, Amy Gillis, Nikita R. Bhatt, Paul F Ridgway .A comparison of common Instruments for procedural skills assessment. In Press. IJME. May 2017 Research Output A Gillis & M. Morris, O. Mothersill, AJ Fagan, T. Frodl, G. Donohoe, PF Ridgway. Using fMRI to distinguish levels of expertise among surgeons and surgical trainees in complex surgical tasks. Under Review. JSE . May 2017. Bhatt NR, Morris M, O'Neil A, Gillis A, Ridgway PF. When should surgeons retire? Br J Surg. 2016 Jan; 103 (1):35-42. doi: 10.1002/bjs.9925. Epub 2015 Nov 18. Review. *Marie Morris , Amy Gillis , Donal O` Connor, Kevin C Conlon, Paul F Ridgway Technology Enhanced Remote Assessment . Conference Presentation. NASCE Dublin. Oct 2016.

Acknowledgements Dean of Health Sciences Award 2015 – “ACE” Project Dean of Health Sciences Award 2016 – Remote Assessment Project Collaborators Ms Amy Gillis Mr Donal O` Connor Professor KC Conlon Professor PF Ridgway Clinical Skills Team Ms Clare Whelan Ms Triona Flavin Ms Olive Killoury Dr Erika Keane Facilitators Mr Damien O` Connor Class of 2018 and 2019

THANK YOU