Cross-modal perception of motion- based visual-haptic stimuli Ian Oakley & Sile OModhrain Palpable Machines Research Group Media Lab Europe Sugar House Lane Bellevue Dublin Ireland

Overview Background – Touch in Broadcast Media The Touch TV Project Role of cross modal perception in TTV Experimental Study - Pilot Conclusions and Future Work

Touch in Broadcast Media Adding haptic information to a broadcast may enhance viewing Motivation Consumer demand for richer media Led to advances in audio and video technologies Touch may provide greater richness Latest advances not enhanced viewing experience e.g. digital and interactive TV Evidence supporting this assertion Touch in virtual environments linked to increased ratings of presence and co-presence Adoption of touch in gaming and simulator systems

The Touch TV Project Ongoing work investigating integration of touch into broadcast content Currently, focused on a practical and theoretical exploration of the design space available for this work Involves a number of research threads ranging from authoring of media to psychological study

Cross-Modal Perception in TTV Touch TV inherently cross-modal Involves construction of immersive and involving media with graphics, audio and touch components Techniques for combining graphics and audio established in this context E.g. visual capture How to add touch to this format is an open question How do we add haptic information to an audio-visual stream ensuring that it will be interpreted as we intend?

Experimental Study Interested in combination of visual and haptic stimuli representing movement in complex graphical scenes Relatively complex stimuli Initial questions: Can haptic cues be associated with visual movement? What is the effect of visual distractors? What is the effect of visual target position? What kind of haptic cues should we use? Attempted to address these questions in a pilot study

Experimental Study In each trial, the visual display consisted of a number of bouncing balls controlled by a simple mathematical model Each ball had a different spring constant – different bounciness and was subject to gravity Force displayed on a PHANToM haptic interface either represented the velocity of one of the displayed balls, or of an unseen ball Participants task was to determine whether or not what they could feel related to one of the displayed balls

Experimental Study Varied: Number of distractors (5 possibilities) Target spring constant (6 possibilities) Target position (5 possibilities) Haptic-Visual match (2 possibilities) Total of 300 trials presented in random order Practice consisted of a random selection of 50 trials 10 Participants All employees at Media Lab Europe 6 male, 4 female, all right-handed, none reported tactile or visual impairment, range of familiarity with haptic devices (2.5 on a scale from 0-5) Participants wore noise cancelling headphones Were able to rest between each trial

Experimental Study Gathered trial time, error rate Trial time for visual presence/absence of haptic stimuli: T-test revealed no significant differences

Experimental Study Further analysis restricted to trials in which haptic ball was visually present Used ANOVA, post-hoc t-tests with Bonferroni CI adjustments Did not include trials in which subject made errors in temporal analysis However, relatively large error rate Collapsed position factor for temporal analysis Used a 5 (distractors) x 6 (spring values) No interactions found (F=0.399, p=0.991) Error data 5 (distractors) x 6 (spring values) x 5 (positions) One interaction: Distractors by Spring Value (F=1.974, p<0.01)

Results Trial time for differing numbers of visual distractors ANOVA: F=8.235, p<0.001 No of Distractors XNot Sig P<0.01 1XxNot SigP<0.05P<0.01 2XXXNot SigP<0.05 3XXXXNot Sig 4XXxXX Results of post-hoc t-tests

Results Trial time for differing spring values ANOVA: F=83.881, p<0.001 Spring Values XNot SigP<= XXP<= XXXP<= XXXXP<= XXXXXNot Sig 0.4XXXXXX Results of post-hoc t-tests

Results Error rate for differing numbers of distractors ANOVA: F=20.614, p<0.001 Results of post-hoc t-tests No of Distractors XP<0.05P<0.01 1XxNot SigP<0.05P<0.01 2XXXNot SigP<0.01 3XXXXNot Sig 4XXxXX

Results Error rate for differing spring values ANOVA: F=4.861, p<0.001 Post-hoc t-tests revealed 0.15 spring value led to significantly higher error rate than 0.4 spring value

Results Error rate for target positions ANOVA: F=0.399, p<0.05 Post-hoc t-tests did not attain significance

Interpretation Subjects merged visual-haptic motion stimuli Error rate always less than chance Increase in trial time and error rate with increased numbers of distractors indicates serial search Haptic target did not pop out as in parallel search Equality of present and absent results suggests subjects performed an exhaustive search Possible speed/accuracy trade off Errors decreased, time increased with larger spring values Eccentricity effect in position error data suggests subjects focused on the centre of the screen

Conclusions and Future Work Study successfully informed basic questions relevant for our TTV domain Relevant to our authoring process Many questions remain Contrast same scenario with visual/audio and visual/audio/haptic stimuli Role of visually distinct objects Is mapping force to motion the best solution? Basic scaling research on force presentation in this domain …

Cross-modal perception of motion based visual-haptic stimuli Ian Oakley & Sile OModhrain Palpable Machines Research Group Media Lab Europe Sugar House Lane Bellevue Dublin Ireland EuroHaptics th -9 th July