Immersive Virtual Characters for Educating Medical Communication Skills J. Hernendez, A. Stevens, D. S. Lind Department of Surgery (College of Medicine) Department of Surgery (College of Medicine) M. Duerson Department of Community Health and Family Medicine (College of Medicine) Department of Community Health and Family Medicine (College of Medicine) K. Johnsen, R. Dickerson, A. Raij, B. Lok Department of Computer and Information Science and Engineering (College of Engineering) The University of Florida, Gainesville, FL The University of Florida, Gainesville, FL J. Jackson, M. Shin Department of Computer Science The University of North Carolina at Charlotte, Charlotte, NC

What is a Virtual Character? Virtual character - a character who represents the state of a system Virtual character - a character who represents the state of a system In TRON (1982), humans and humans that represents software interacted within a world that represented the hardware. In TRON (1982), humans and humans that represents software interacted within a world that represented the hardware.

What is a Virtual Character We aim to to have humans and human the represent software interact in the real world. We aim to to have humans and human the represent software interact in the real world. r

Virtual Character Research Mostly Mostly –Rendering –Animation –Artificial Intelligence Newer approaches Newer approaches –Affective power –Human response to VCs Clinical Psychology Clinical Psychology –Fear of public speaking

Motivation “Doctor, I have a pain in my side!” “Doctor, I have a pain in my side!” What you do next depends on: What you do next depends on: –Age –Gender –Ethnicity –Visual Cues –Audio Cues

Motivation Current methodologies Current methodologies –Books –Standardized Patients (actors) Restricted in: Restricted in: –Training frequency –Scenario variety –Immediate feedback Affects medical student preparedness Affects medical student preparedness Provide additional training opportunities Provide additional training opportunities

Solution Interactive Virtual Characters Interactive Virtual Characters –Life-Sized –Computer Generated –Natural Interaction –Responds to User –Responses based on accepted medical procedure COTS Equipment: COTS Equipment: –Projectors –Laptops –Web cameras –Tablet PC

Why Do We Want Virtual Characters? Propose: Digital Characters as a new (meta-) medium to interact with information Propose: Digital Characters as a new (meta-) medium to interact with information Effective Interaction Effective Interaction –Better than keyboard and mouse for certain tasks –Dynamic –3D –Natural interaction –Low Bandwidth Effective Collaboration Effective Collaboration –Controlled conveyed visual information –Non-verbal communication (60%)

Let’s look at interaction Each participant in a communication has three stages: perception, cognition, and response Each participant in a communication has three stages: perception, cognition, and response Define interaction as both the input and output Define interaction as both the input and output Thinking Participant Thinking Digital Character Thinking PerceivingResponding Digital Character Responding PerceivingResponding Perceiving Interaction

Combines Speech Speech Computer Vision Computer Vision –Eye Gaze –Gesture Recognition –Repeat your gestures High Quality 3D models High Quality 3D models –Animation –Rendering –Visualization

Integrates Computer Science Computer Science –Computer Graphics –Image Processing –Natural Language Processing Medicine Medicine –Training –Education –Standardization Education Education –Multimedia Learning –Technology based Learning –Training What other areas could this be used for? What other areas could this be used for? –Psychology –Social Science –Education

Current and Future Work Current Status: Current Status: –Initial scenario created with gesture, speech, and visualization components integrated –Evaluating with a group of Medical Students Year 2 Future Work: Future Work: –Formal evaluation studies –Increase and improve scenarios –Enhance interactivity Show video Show video

Play Video