Establishing the reliability and validity of a Virtual Reality Upper Gastrointestinal simulator using a novel video-endoscopic assessment technique. Moorthy K, Munz Y,Jiwanji M, Bann S, Chang A, Darzi A Department of Surgical Oncology and Technology, Imperial College, London, UK INTRODUCTION: A few studies have established the construct validity and reliability of other virtual reality (VR) simulators as tools for assessment. Construct validity is the ability of the simulator to differentiate endoscopists with varying levels of expertise. Reliability is the ability of the simulator to assess skills reliably over 3-4 procedures. Face validity is the extent to which the simulation is representative of real procedures. Demonstration of construct validity is still a crude way of correlating performance on the simulator with performance in real life. The demonstration of this correlation is crucial to the acceptance of VR simulators as tools for assessment. It has been difficult to demonstrate transfer of skills from VR to real procedures primarily because it is difficult to assess performance in real procedure. An objective assessment tool could correlate performance in real procedures with performance in virtual reality. AIM: Establish the validity and reliability of the VR upper gastrointestinal (UGI) simulator Develop a global rating scale for the assessment of skills Correlate the global scale with the simulators metrics MATERIAL AND METHODS: The VR system (Simbionix, Israel): The Subjects: Group 3- experts (>200 diagnostic procedures) Group 2- intermediate (10-50 procedures) Group 1- novices (never performed) The modules: Diagnostic upper GI endoscopy (gastroscopy) module. The participants were permitted to perform case 1 on the simulator for an unlimited period of time to get acquainted with the system. Following which they all undertook 2 attempts each on cases 4 and 5 in a random manner. Case 4: Hiatus hernia with significant reflux esophagitis, ulcer in the posterior wall of D 1 Case 5: Esophageal diverticulum, a polyp in the fundus of the stomach, malignant lesion in the antrum. Study data: Construct validity- simulators metrics, Face validity- Questionnaire on a Likert scale, Video- endoscopic (global) score (Fig 1) Video-endoscopic score: 8 criteria- Examination of esophagus, body, antrum, fundus, duodenum, pyloric intubation, knowledge of the procedure and flow of the procedure- all rated on a 5 point Likert scale. One case of every participant played back to two independent blinded observers. Data analysis: Kruskal Wallis, Mann-Whitney for inter-group differences; Reliability- Cronbachs alpha; Correlation between simulators metrics and global score- Spearmans rho DISCUSSION: There were significant differences across the groups for all parameters. The percentage of mucosa visualized is an indicator of the thoroughness of examination. Even though the intermediate group carried out a more thorough procedure, they were only slightly faster than the novices, as a result of which the efficiency of performance remained unchanged. The fact that there was no difference in between Groups 1 and 2 in terms of the percentage of pathologies identified was of some concern. Playback of the procedure revealed that this was a result of both groups failing to identify an ulcer in the posterior wall of the first part of the duodenum. A majority of the trainees in the intermediate group failed to scan the first part of the duodenum in a systematic and thorough manner. In fact the stimulus for the development of the video-endoscopic assessment came from this finding. The systems reliability of assessment in terms of nearly all the parameters was greater than Such a high level of reliability between independent observers is considered to be suitable for even high stakes assessment in surgery. The video-endoscopic assessment is able to discriminate endoscopists with varying levels of experience. CONCLUSIONS: This study has demonstrated the construct validity and reliability of the simulators metrics. Significant correlation between the video-endoscopic score and the simulators metrics. Video-endoscopic score: novel method for the assessment of skills in UGI endoscopy FUTURE WORK: Validate the video-endoscopic score in real procedures Use the video-endoscopic to show transfer of skills from VR to real procedures. RESULTS Groups- performance parameters Face validity questionnaire (5 point Likert scale) The reliability coefficient between the averaged performance on case 4 and 5 was 0.88 for time taken for the procedure and for the percentage of mucosa visualized. The reliability of assessment across the 4 attempts (both cases 4 and 5) was 0.90 for time taken, 0.89 for percentage of mucosa visualized and 0.86 for efficiency of performance. Video-endoscopic score: Significant difference in the score across the groups (p<0.001). The level of agreement between the two observers was Correlation between metrics of the case rated by the observers and the simulators assessment of performance: Btween the global score and the percentage of mucosa visualized (Spearmans rho= 0.60, p<0.001), the percentage of pathologies identified (rho=0.34, p=0.05) and whether or not retroflexion was performed (rho=0.65, p<0.001). ParameterNovices (mean+/- SD) Intermediate (mean+/- SD) Experts (mean+/- SD) Time (sec)242.7 (107.6)236.4 (85.3)183.2 (49.5) % mucosa78.8 (9.1)86.1 (5.7)85 (5.5) Efficiency0.37 (0.17)0.40 (0.11)0.53 (0.17) %age pathology 82.8 (22.3)89.7 (19.5)98.7 (7.9) QuestionMedian Graphics4 Simulation of ease/ complexity4 Force feedback3 Training tool5 Assessment tool4 REFERENCES: Datta V, Mandalia M, Mackay S, Darzi A. The Pre-Op flexible sigmoidoscopy trainer. Validation and early evaluation of a virtual reality based system. Surg Endosc. 2002; 16: Ost D, DeRosiers A, Britt JE, Fein AM, Lesser ML, Mehta AC. Assessment of a bronchoscopy simulator. Am J Respir Crit Care Med. 2001; 164: Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skills via an innovative bench station examination. Am J Surg. 1996; 172: Neumann M, Firedl S, Meining A et al. A score card for Upper GI endoscopy: evaluation of Interobserver variability in examiners with various levels of experience. Z Gastroenterol. 2002; 40 (10): KW, p<0.001