Development Virtual Environments for fMRI Socially to Interact with a virtual avatar ; a pilot study Hyeongrae, Lee Dept. of Biomedical Engineering, Hanyang University, Seoul, Korea

Introduction

3 Introduction The social virtual environments (VEs) The social virtual environments (VEs) VEs Virtual avatar Virtual avatar Social VEs

4 Introduction Schizophrenia The research and therapy using the social VEs The research and therapy using the social VEs Alcoholism Social phobia Social VEs Neuroscience

5 Display device Tracking device Tactile device Conventional device … Introduction The functional MRI environments MR room High magnetic and electronic fields  For social VEs, the other method is needed!!!

6 Introduction In this study, we purpose thatIn this study, we purpose that 1.development the social VEs for social cognitive neuroscience study using MR compatible devices. 2.a pilot study performed on the developed social VEs.

Methods

8 Outside of magnet room Inside of magnet room Real-time video image Marker detection on the hand Computer Filter panel Calculate maker position Apply hand position to virtual environments MR compatible HMD MR compatible camera marker development the social VEs fMRI environment for social interaction fMRI environment for social interaction

9 development the social VEs Contents for the social VEs Contents for the social VEs (b) Acceptance (a) Rejection

10 development the social VEs

11 A pilot study Normal group ( n = 12 ) ( n = 12 ) Age (SD: 1.78) Sex (M:F) 5 : 7 Participants Participants

12 A pilot study fMRI experiment design

13 A pilot study Questionnaires of social realism about the other avatar (1) How real the other avatar’s action was? (2) Did you feel that you really shook hands with (or was rejected by) the virtual other ? Questionnaires of social cognition about the other avatar (1) Did you feel positive feeling about the other avatar’s response? (2) Did you feel negative feeling about the other avatar’s response? (3) Did you think that other avatar’s response was suitable in the situation? (4) Did you feel that the other avatar rejected your offer? (5) Did you feel that the other avatar hated you or had negative emotion about you? (6) Did you feel that the other avatar took threatening attitude to you? (7) Did you feel that the virtual other ignored or excluded you? (8) Did you feel that the virtual other was aware of you? Questionnaire (5 point response scale ) Questionnaire (5 point response scale )

14 A pilot study fMRI data acquisition fMRI data acquisition - The functional MRI conducted with 1.5T machine (GE Medical System). BOLD (Blood Oxygenate Level Dependent) EPI sequence, Gradient Echo 64x64x30 matrix with 3.75x3.75x5-mm spatial resolution, FOV: 240mm, Slice thickness: 5mm, TR: 2s, TE: 14.3, FA=90, # of slices: 30 High resolution anatomical images Fast Spoiled Gradient Echo sequence 256x256x116 matrix with 0.94x0.94x1.50-mm spatial resolution, FOV: 240mm, Slice Thickness: 1.5mm, TR: 8.5s, TE: 1.8s, FA: 12, # of slices: 116).

15 Coregistration Temporal Normalization 3D construction Preprocessing Temporal Smoothing (3-point Low pass filter ) Temporal Smoothing (3-point Low pass filter ) Spatial Normalization (MNI N27 templete, 2x2x2mm 3 ) Spatial Normalization (MNI N27 templete, 2x2x2mm 3 ) Spatial Smoothing (Gaussian filter, 8 mm FWHM) Spatial Smoothing (Gaussian filter, 8 mm FWHM) - Analysis of Functional NeuroImages (AFNI, Ver. 2006_06_30_1332) - subjects whose haed movements are more than 1mm are excluded A pilot study Statistical Analysis General Linear Model Analysis in each subject General Linear Model Analysis in each subject Group Analysis Paired t-test within group (uncorrected P < 0.001) Paired t-test within group (uncorrected P < 0.001) fMRI analysis fMRI analysis

Results & Discussions

17 Results & Discussions Questionnaire (5 point response scale)Questionnaire (5 point response scale) - Social realism about other avatars Results & Discussions (1)How real the other avatar’s action (2)really shake hands with (or was rejected by) the virtual other ? The questionnaire results. The questionnaire results. - 1 point : not at all 5 point : very much

18 Results & Discussions Questionnaire (5 point response scale)Questionnaire (5 point response scale) - social cognition about other avatars Results & Discussions The questionnaire results. The questionnaire results. (1) positive feeling (2) negative feeling (3) other avatar’s response was suitable (4) the other avatar rejected participant offer (5) hated participant or had negative emotion (6) threatening attitude (7) ignored or excluded participant (8) awareness of participant - 1 point : not at all 5 point : very much

19 Results & Discussions uncorrected P 80 microliter Acceptance > Rejection L thalamus (-11, -17, 8)R insula lobe (37, -3, 0) Rejection > Acceptance L prefrontal gyrus (-17, 53, 20)R rolandic operculum (37, -3, 20) L superior temporal sulcus (-37, -31, 6)L cerebelum (VI) (-25, -45, -22)

20 Results & Discussions Acceptance > Rejection L thalamus R insula lobe mirroring other’s emotion related to empathy Social cognition related friendship and teamwork

21 Results & Discussions Rejection > Acceptance L prefrontal gyrusL superior temporal sulcus understanding other’s mental states Social emotion (ex. embarrassment) Social emotion (ex. embarrassment)

22  In this study, we develop the VEs for social cognitive neuroscience study that it is possible to interact with the virtual avatars.  This social VEs would provide social environment to patients with deficit in social cognition, such as patients with schizophrenia, social phobia and etc., even though just only handshake with other.  The social interaction methods using maker would be practicable in various social cognitive neuroscience studies. Summary

23 Prof. Ph.Dcandidate M.S candidate Hyeongrae Lee Neuro-mechanism On Virtual Environment Lab. Dept of Biomedical Engineering In Hanyang University Homepage: FAX: Tel: