1. TECHNOLOGY ENHANCED REMOTE ASSESSMENT Deans Award for Innovation 2016
Marie Morris, Ph.D. MSc. BSc. (HONS), DPSN - Teaching, RNT, RGN

2. Collaborators
Ms Amy Gillis
Mr Donal O` Connor
Professor Kevin C Conlon
Professor Paul F Ridgway

3. Ethical Considerations
The School of Medicine, Faculty of Health Sciences Research and Ethics Committee University of Dublin Trinity College approved this project. Written informed consent was gained from all participants to utilise the data collected for research and publication.

4. Assessment Governance
GMC
Domains of Professional Practice (Adapted Irish Medical Council, 2014)

5. Tuning Project, 2008

6. Bologna Process – Freedom of Movement
Tuning Project, 2008

7. Tuning Project , 2008

8. Millers pyramid of assessment (Miller, 1990)
Assessment Model
Millers pyramid of assessment (Miller, 1990)

9. Assessment - Term Definitions
Validity
Construct Validity
Reliability
Sensitivity
Specificity
The concept of test validity is to what extent an assessment actually measures what it is intended to measure and permits appropriate generalisations about student’s skills and abilities. (Norman, 1991).
The ability to differentiate between groups with known differences in ability - beginners versus intermediates versus experts.
Refers to consistency of a test over time over different cases (inter- case) and different examiners (inter-rater). Reliability is measured as a correlation with 1.0 being perfect reliability and below 0.5 as unreliable. Evaluation measurements reliabilities above 0.65 or preferably above 0.85 are recommended. (Norman, 1991)
Sensitivity, also called the true positive rate, measures the proportion of positives that are correctly identified as such (e.g. the percentage of sick people who are correctly identified as having the condition).
Specificity, also called the true negative rate, measures the proportion of negatives that are correctly identified as such (e.g. the percentage of healthy people who are correctly identified as not having the condition).
Utility = reliability x validity x feasibility x acceptability x educational impact.

10. Assessment Challenges - Curriculum
Requirement to provide comprehensive feedback
Results should be a quality indicator and guide curriculum reform
Results should reflect potential clinical behaviours
Identification of students achieving below expected levels
Assessments should be consistent, standardised and defendable
Increasing class sizes
Fottrell Report 2006
Reducing number of available experts Reduced working hours EWTD (2003/88/EC)

11. Assessment Challenges - Curriculum
Limited availability of “Real“ patients
Costs – Stimulated Patients, Venues, Staff

12. Assessment Challenges – Assessors
Hawk
Dove
Complex area of assessor decision making in performance testing (Sadler, 2009; Govaerts, 2011; Kogan et al, 2011; Yorke, 2011). The literature reveals that the factors affecting assessor decision-making can be
highly individualised,
contextualised
influenced by characteristics such as assessor experience and seniority (Pell & Roberts, 2006; Kogan et al, 2011).
Halo Effect
Horns Effect
Central Tendency

13. Assessment – Students Perspective
Accept direct observational is a core course requirement
Desire for a fair assessment
Passing should not be down to “luck”
Transparent process
Feedback should be timely
Accept dual examiners superior but report that multiple examiners increases anxiety and stress

14. Novel Approaches to Technical Skills Assessment
Brain activity was estimated by means of functional Near Infrared Spectroscopy (fNIRS). fNIRS is a non-invasive technique for optical estimation of cortical activity. fNIRS measurements were taken over the prefrontal cortex since this region is involved in cognitive processing34
Functional near infrared spectroscopy (fNIRS), (Leff et al, 2008; Ohuchida et al, 2009; Cui et al, 2011).
Functional Magnetic Imaging (Morris et al, 2012 ; Bahrami et al, 2012).

15. Why use Technology in Assessment ?
Technology savvy millennial generation students
Promotes student engagement
Facilitates student centred learning
Furthers the objectives of the Trinity Education Project.
Promotion of non-traditional modes of assessment
Embeds 21st Century assessment modalities into the curriculum

16. R.I.T.A Undergraduate Setting Double RoboticsTM Telepresence Device
Remote Interpretation of Technical Ability
Double RoboticsTM
Telepresence Device
Five-foot tall tablet computer on wheels
via Wi-Fi

18. Logistics - Double RoboticsTM
Controlled by the absent examiner via their mobile phone
Remote assessor appears on the screen
Conducts a face-to-face interaction with the student
Student and patient can be seen up close
Examiner can check for levels of competence in the students’ performance
Remote assessor can quickly travel around the exam room

19. Feasibility Study
Mixed methods approach – quantitative and qualitative
An investigation into the levels of agreement (correlations) between two examiners assessments – one present, one remote
Remote assessment via a tele-presence device
Examiner, student and simulated patient evaluations

20. Alternative Clinical-Skills Examination
“ACE”  
Common cases (0.5 hour per station);
Observed by examiners with GRS raters, standardised patient and materials (referral letters, imaging, investigation results)

21. Standardisation
Content experts within the medical school who have experience with assessment at a local and national level developed the cases ensuring comparable complexity across cases.
Each case was reviewed further by content experts and pilot-tested.

22. Data preparation and analysis
Krippendorrf's Alpha - Inter-rater reliability tests - (Greene, 1999) to examine levels of agreement between independent examiners.
Low correlation = < 0.59
Medium correlation =
High correlation =
Free text comments were analysed using the “cut and paste” technique (Richie, 2000)

23. Methods
48 - Year 3 medical students volunteered to participate in November 2016.
The "ACE" format consisted of 4 sequential patient encounters observed by two independent examiners. One present and one remote (n=16).
Examiners and students evaluated “RITA” using free text comments.

24. Standard Setting Exceeds Expectation – IH (5) and 2H (4)
Marking Scale 1 – 5
Mark 3 out of 5 on GRS = readiness for Year 4 practice.
Examiners must mark the student independent of each other.
Professionalism and Patient Safety to be considered in overall grade
Expectation regarding performance was
Exceeds Expectation – IH (5) and 2H (4)
Meets Expectation- Pass (3)
Borderline- Safe but needs more practice (2)
Below Expectation – Unsafe – Fail (1)

25. Feedback to students
A copy of all examiners marking sheets across 5 encounters were supplied to all students within 3 days
Yellow Sticker – issues with professionalism
Red sticker – patient safety
Large group – PowerPoint on common areas requiring improvement – within one week of the assessment
Opportunity to meet for one to one feedback and discussion of gradings and examiners comments.

26. RESULTS – Quantitative
Skill
Result
History taking
0.487
Physical Examination
0.458
Communication Skills
0.0714
Privacy/Dignity
-0.182
Time Management
Procedural Skills - Phlebotomy
0.711
Data Interpretation
0.425
Response to Patient queries
0.593
Over all Performance
0.795
(Ordinal Method)
More generalisable and robust to imperfect data than Cronbach's
Krippendorrf's Alpha

27. RESULTS – Quantitative
100% agreement between both Examiners regarding Pass/Fail grades.
Where marks differed one examiner marked a 4 and the other 5.
The remote examiners reduced grades according to the amount of prompting required. This was reflected in the wide variation in grades for time management.
The remote assessor reduced grades where the patient was over exposed and exposed for longer periods than required for the physical examination. This was reflected in the wide variation in grades for privacy/dignity .

28. RESULTS – Qualitative Themes and Issues Identified by Remote Assessor
Good Wi-Fi Access required
Drainage of battery
Picture Good – screen freezing
Difficult to hear on occasion- student needs to talk to screen
Difficult to move around bed on occasion – practice required
Remote assessor observer rather than participant
Consider a mounted device

29. RESULTS – Qualitative Themes and Issues Identified by Present Assessor
Picture Good
Occasional screen freezing – short lived
Had little interaction with remote assessor as observing the student
Needed to be aware of where the device was so as not to obstruct view
Remote assessor observer rather than participant
Consider a mounted device

30. RESULTS – Qualitative Themes and Issues Identified by Students
Non- intrusive
Better than 2 Examiners in the room
Benefit of feedback from 2 Examiners
Unsure re talking to the remote assessor

31. RESULTS – Qualitative
Themes and Issues Identified by Simulated Patients
Didn’t take much notice of device or remote assessor
Remote assessor more about watching the student rather than interacting with us
Need to ensure enough room around bed for device to move quickly
Would a mounted device work better ?

32. Costings Purchase cost of double robotics device = 2,600 Euro per unit
Includes Double 2,
Charging Doc,
Audio and Camera Kit,
Driver Apps,
AC Adapter
1 Year Warranty.

33. Conclusion Technology enhanced assessment is feasible.
It is a non traditional assessment method - TEP
It addresses the issue of limited access to subject experts.
It reduces the cost of external examiners.
A larger study is needed with more examiner pairings.

34. Future Larger study week commencing September 2017
Correlation Study – inter-rater reliability between examiner pairings
4 Telepresence devices
8 Examiners – 4 Remote, 4 Present
Year 3 Students
Checklists and procedural skills

35. OSCE Format

36. Checklist PLACEMENT OF ELECTRODES FOR 12 LEAD ECG AND ECG RECOGNITION
Accurately identify Angle of Louis then 4th intercostal space
Place electrodes and state this is the position for V1 on left & V2 on right
State that they are identifying 5th intercostal space mid clavicular line
Accurately do this and state that this is the position for V
Accurately identify the midpoint between V2 and V
Place electrode and state that this is for V
Accurately identify 5th intercostal space mid axillary line
Place electrode and state that this is for V
Accurately identify anterior axillary line or midway between V4 and V
Place electrode and state that this is for V
Place arm leads correctly
Place leg leads correctly stating which is neutral
GLOBAL RATING
FAIL BORDERLINE SATISFACTORY GOOD EXCELLENT

37. Future Reach of the Project
Remote Ward Round - Professor PF Ridgway
Significantly increases the number of patients seen each day.
“As modern Consultants work in a number of institutions it is very useful to be able to dial in from wherever you are.
“Device has enabled us to be more efficient in delivering diagnosis after assessment, reaching a decision to admit and initiating therapy.
“Initial research has shown that patient acceptability has been very much higher than we expected. And this is not just with the younger age group where it might have been expected.
Re: communication skills - ability to extend to five feet in height or drop down to eye level if the patient is lying in bed, can interact with patients at eye level at all times.

38. Future Reach of the Project
Colleges & Universities
Double is being used in higher education in many ways, including building classrooms with both online and in-person students, to enable Tutors to continue teaching when they move out of the building.
House bound Students
Double enables the remote student to join the classroom in real-time, interact with other students one-on-one, and participate in group discussion. The telepresence robot is a revolutionary tool for housebound students.

39. Research Output
Morris MC, Gallagher TK, Ridgway PF. Tools used to assess medical students competence in procedural skills at the end of a primary medical degree: a systematic review. Med Educ Online. 2012;17. doi: /meo.v17i Epub 2012 Aug 23. Review
Morris MC, Frodl T, D'Souza A, Fagan AJ, Ridgway PF. Assessment of Competence in Surgical Skills Using Functional Magnetic Resonance Imaging: A Feasibility Study. J Surg Educ Oct 25. pii: S (14)00258-X. doi: /j.jsurg
M Morris, M Bennett, M Hennessy, KC Conlon, PF Ridgway. An Alternative Certification examination "ACE": Can post graduate methods be used to asses clinical skills in medical undergraduates? British Journal of Medicine and Medical Research. 3(4): , 2013
*Morris MC, Gillis AE, Smoothey CO, Hennessy M, Conlon KC, Ridgway PF. An alternative certification examination ("ACE") in surgery. J Surg Educ Nov-Dec;71(6): doi: /j.jsurg Epub 2014 Jul 11

40. Marie Morris, Amy Gillis, Nikita R. Bhatt, Paul F Ridgway
Marie Morris, Amy Gillis, Nikita R. Bhatt, Paul F Ridgway .A comparison of common Instruments for procedural skills assessment. In Press. IJME. May 2017
Research Output
A Gillis & M. Morris, O. Mothersill, AJ Fagan, T. Frodl, G. Donohoe, PF Ridgway. Using fMRI to distinguish levels of expertise among surgeons and surgical trainees in complex surgical tasks. Under Review. JSE . May 2017.
Bhatt NR, Morris M, O'Neil A, Gillis A, Ridgway PF. When should surgeons retire?
Br J Surg Jan; 103 (1): doi: /bjs Epub 2015 Nov 18. Review.
*Marie Morris , Amy Gillis , Donal O` Connor, Kevin C Conlon, Paul F Ridgway Technology Enhanced Remote Assessment . Conference Presentation. NASCE Dublin. Oct 2016.

41. Acknowledgements Dean of Health Sciences Award 2015 – “ACE” Project
Dean of Health Sciences Award 2016 – Remote Assessment Project
Collaborators
Ms Amy Gillis
Mr Donal O` Connor
Professor KC Conlon
Professor PF Ridgway
Clinical Skills Team
Ms Clare Whelan
Ms Triona Flavin
Ms Olive Killoury
Dr Erika Keane
Facilitators
Mr Damien O` Connor
Class of 2018 and 2019

42. THANK YOU