1. Relative influence of sensory cues in a multi-modal virtual environment Claudia Hendrix Pi-Ming Cheng William Durfee Department of Mechanical Engineering SIGHT (visual) TOUCH (haptic) SOUND (aural)

2. Take-home messages  Virtual environments can do a good job emulating soft materials  Haptic cues are important for identifying materials  Adding visual or sound cues improves ability to discriminate among materials  Of haptic, visual and sound, sound least important  Virtual prototyping using sight, sound and touch is feasible, but effects of sensory interaction are complex and vary from person to person

3. Some prior multi-modal VR work  Marks [1978]: Reviews psychophysics of sensory interaction  Richard and Coiffet [1995]: Adding substitute sensory haptic feedback improves performance on grasping and place tasks  Hendrix and Barfield [1995]: Adding synthesized sound sources to a virtual world increases sense of “presence”…..but not sense of “realism”  Srinivasan et al [1996] and Durfee et al [1997]: Visual displays influence perception of haptic stiffness  DiFranco et al [1997]: Audio cues influence perception of haptic stiffness

4. Virtual product prototyping  Apply virtual reality technology to create product prototypes  Move beyond CAD-based visual rendering SeeHearTouch ?? ??

5. Panel controls: simplified paradigm for research Fixed in space, single d-o-f, low-force, simple graphics Sufficiently complex to enable exploration of research questions

6. With panel controls paradigm... VisualAural Haptic “Head-related transfer function” system

7. Experiment system

9. Haptic display

10. Haptic display control Materials modeled as spring-damper, PD impedance controller, 60 u-sec update rate

11. Aural display  Record sound clips for different strike velocities  Synchronized playback  (Synthesized sound based on physical models too complex…for now)

12. Match virtual to one of 3 real  EXPERIMENT #1 “BEST” MATCH –V VR = V 1 –H VR = H 1 –A VR = A 1  EXPERIMENT #2 SENSORY CONFLICT –V VR = V 2 –H VR = H 1 –A VR = A 3 V VR H VR A VR V1H1A1V1H1A1 V2H2A2V2H2A2 V3H3A3V3H3A3 VIRTUAL REAL metal hard foam soft foam Material properties: V = visual H = haptic A = aural

13. Experiment setup

14. Stiffness (Nm/rad) 35 7.2 3.2 REALVIRTUAL Soft foam Hard foam Metal Soft foam Hard foam Metal Ability of haptic interface to mimic material stiffness

15. Protocol, Expt #1 (“best” VR) Test conditions MATERIAL TYPE MetalHardSoftfoam TESTVirtual XXX MATERIAL Real XXX PROCEDURE: Probe test and reference materials, then answer: 1. Which material best matches test? 2. What is the quality of the match? (1-5) 6 test conditions, 4 replications = 24 trials/session 12 subjects

16. Matching results Material, matches (%), quality rating (1-5) METAL HARD FOAM SOFT FOAM M100(4.6) HF0 SF0 M0 HF85(4.3) SF15(3.7) M0 HF2(3.0) SF98(4.5) When test was... Subjects matched to... Test material was real METAL HARD FOAM SOFT FOAM When test was... Subjects matched to... Test material was virtual M60(3.5) HF40(3.0) SF0 M0 HF83(3.9) SF17(2.7) M0 HF2(2.0) SF98(4.2)

17. Quality ratings

18. Protocol, Expt #2 (sensory conflict) V soft foam H metal A soft foam Looks and sounds like soft foam, feels like metal metal hard foam soft foam metal hard foam soft foam metal hard foam soft foam Visual Aural Haptic 27 test conditions (always virtual), 3 replications = 81 trials per session TEST MATERIAL

19. Matching with sensory conflict M HF SF TEST MATCH M11 HF79 SF10 M14 HF58 SF28 M13 HF11 SF76 HAPTIC M HF SF TEST MATCH M8 HF26 SF65 M14 HF50 SF36 M15 HF72 SF13 VISUAL M HF SF TEST MATCH M18 HF43 SF39 M10 HF40 SF50 M10 HF65 SF25 AURAL M HF SF TEST MATCH M31 HF62 SF7 M4 HF47 SF49 M4 HF13 SF83 HAPTIC + AURAL M HF SF TEST MATCH M38 HF61 SF1 M8 HF76 SF15 M7 HF7 SF86 HAPTIC + VISUAL M HF SF TEST MATCH M22 HF32 SF46 M11 HF50 SF39 M8 HF68 SF24 AURAL + VISUAL 1-WAY 2-WAY

20. 3-way match M HF SF TEST MATCH M47 HF44 SF8 M0 HF78 SF22 M0 HF3 SF97 HAPTIC + VISUAL + AURAL

21. Quality when matched

22. Cues used by the subjects “What strategy did you use to make your matches?”

23. What we learned Future work: 1. Use 2 out of 3 cues (V,H,A) to determine relative influence 2. Mechanical design task using virtual and real prototypes National Science Foundation (NSF/MIP-9420394, and a graduate training grant, Cognitive Sciences Center, University of Minnesota, DMC8857851) Work supported by:  Virtual environment does well when emulating soft materials  Haptic cues are important for matching  Adding visual or sound cues to haptics improves perceived quality of match  Subjects tended not to use sound cues for matching