Cross-modal Prediction in Speech Perception Carolina Sánchez, Agnès Alsius, James T. Enns & Salvador Soto-Faraco Multisensory Research Group Universitat Pompeu Fabra Barcelona

Auditory + visual performance MSI enhancement Background Visual + Auditory Improve Speech Perception Multisensory Integration

Background Prediction within one sensory modality Many levels of information processing –Phonological prediction “ This morning I went to the library and borrowed a … book” (De Long, 2005; Pickering, 20707) –Visual prediction: Visual search (Enns, 2008; Dambacher, 2009) –Sensorimotor prediction: forward model (Wolpert, 1997)

Predictive coding Pickering, 2007

Hypothesis If there exists prediction within the same modality, and if predictive coding models can account for prediction at a phonological level, then … Predictive Coding could occur across different sensory modalities too.

Indirect evidences of cross-modal transfer in speech van Wassenhove’s, 2005 time ERPs Amplitud reduction Shortening latency /pa/ high visual saliency /ka/ short visual saliency

Our study Visual prediction Auditory prediction Visual-to-auditory cross-modal prediction Auditory-to-visual cross-modal prediction

Visual prediction Visual stream Auditory stream V A With visual informative visual context Without informative context Task : AV Match vs. AV Mismatch Target fragment Context fragment speech non speech

Results * Reaction time msec matchmismatch With visual informative context Without informative context * With previous context participants respond faster than without it. VISUAL PREDICTION

Auditory prediction Visual stream Auditory stream V A With auditory informative auditory context Without informative context speech non speech Task : AV Match vs. AV Mismatch Target fragment Context fragment

Results * With auditory informative context Without informative context Reaction time msec match mismatch * With previous context participants respond faster than without it. AUDITORY PREDICTION

Visual vs. Auditory Visual prediction Auditory prediction Rts msec congruentincongruent With visual informative context Without informative context * With auditory informative context Without informative context Rts msec congruent incongruent *

Conclusions Visual prediction Auditory prediction Is this prediction cross-modal?

Predictability of Vision-to-Audition Design of the experiment V A Mismatch Unimodal continued Auditory stream Visual stream Match Unimodal continued V A Discontinued Match V A Discontinued Mismatch V A Cross-modal continued Mismatch

Predictability of Vision-to-Audition Stimuli V A Mismatch V A V A Unimodal continued Discontinued Cross-modal continued

Results Participants were faster in the cross-modal condition than in the completely incongruent one. VISUAL –TO-AUDITORY PREDICTION Reaction time msec * Visual Auditory Unimodal continued Discontinued Cross-modal continued

Predictability of Audition-to-Vision Design of the experiment Auditory stream Visual stream Match Unimodal continued V A Mismatch Unimodal continued V A Match Discontinued V A Mismatch Discontinued V A Mismatch Cross-modal continued

Reaction time msec Visual Auditory Unimodal continued Discontinued Cross-modal continued Results We didn’t find any difference between the mismatch condicions NO AUDITORY-TO-VISUAL PREDICTION

Conclusions There is some kind of prediction from vision-to- auditory modality There is not any prediction from auditory-to-vision modality Does this prediction depend on the language?

Canadian participants with english sentences VISUAL –TO-AUDITORY PREDICTION IN NATIVE LANGUAGE Reaction time msec * Visual Auditory Unimodal continued Discontinued Cross-modal continued Reaction time msec * Visual Auditory Unimodal continued Discontinued Cross-modal continued Spanish participants with spanish sentences Results (L1)

Canadian participants with english sentences Reaction time msec No differences between the mismatch conditions No prediction from auditory-to- visual modality in native language Spanish participants with spanish sentences Reaction time msec Visual Auditory Unimodal continued Discontinued Cross-modal continued Visual Auditory Unimodal continued Discontinued Cross-modal continued

Conclusions There is some kind of prediction from vision-to- auditory modality in L1 There is not any prediction from auditory-to-vision modality L1 What happens with an unknown language?

Unknown language : visual to auditory Canadian participants with spanish sentences NO VISUAL-TO-AUDITORY IN OTHER LANGUAGE Reaction time msec Visual Auditory Unimodal continued Discontinued Cross-modal continued

Unknown language: auditory to visual Spanish participants with english sentences Canadian participants with spanish sentences Reaction time msec Reaction time msec No differences between the mismatch conditions No prediction from auditory-to- visual modality in other language Visual Auditory Unimodal continued Discontinued Cross-modal continued Visual Auditory Unimodal continued Discontinued Cross-modal continued

Conclusions No visual-to-auditory cross-modal prediction in an unknown language… it seems that some level of knowledge about the articulatory phonetics of the language is required to obtain the advantage of the predictive coding No auditory-to-visual cross-modal prediction

General Conclusions Unimodal prediction from visual to visual modality from auditory to auditory L1: ASYMMETRY –Cross-modal prediction from visual-to-auditory modality –No cross-modal prediction from auditory-to-visual modality Unknown language: previous knowledge of the language is neccesary to make the prediction –No cross-modal prediction from visual-to-auditory modality –No cross-modal prediction from auditory-to-visual modality

-Agnès Alsius, Postdoc Queen’s University -Antonia Najas, MA/ Research Assistant Universitat Pompeu Fabra -Phil Jaekl, Postdoc Universitat Pompeu Fabra - All the people of the Vision Lab, UBC, Vancouver Thanks to… Thanks for your attention!!