1. Каналы мультимодальной коммуникации: относительный вклад в понимание дискурса А.А. Кибрик (ИЯз РАН и МГУ) Н.Б. Молчанова (BearingPoint) aakibrik@gmail.com «Мультимодальная коммуникация» 15 ноября 2013

2. 2 What is the contribution of different communication channels?  Traditional approach of mainstream linguistics: the verbal channel is so central that prosody and the visual channel are at best downgraded as “paralinguistics”  Applied psychology It is often stated that (figures go back to Mehrabian 1971): body language conveys 55% of information prosody conveys 38% of information the verbal component conveys 7% of information  Who is right?

3. 3 Relative contribution of three communication channels? DISCOURSE Vocal channelsVisual channel Verbal channel Prosodic channel

4. 4 Experimental design  Isolate the three communication channels  Present a sample discourse in all possible variants (2 3 =8)  Present each of the eight variants to a group of subjects  Assess the degree of understanding in each case  Such assessment may lead to estimates of the contributions of communication channels

5. 5 Studies in this line of research  Èl’bert 2006, year paper  Èl’bert 2007, diploma thesis  Reinterpreted and refined in Kibrik and Èl’bert 2008  Molchanova 2008, year paper  Molchanova 2009, year paper  Molchanova 2010, diploma thesis  Reinterpreted and refined in Kibrik and Molchanova 2013

6. 6 Èl’bert 2007, Kibrik and Èl’bert 2008  Russian TV serial “Tajny sledstvija” – “Mysteries of the investigation”  Context excerpt: 8 minutes  Experimental excerpt: 3 min. 20 sec.  consisting of conversation alone, to ensure that we are testing the understanding of discourse rather than of the film in general  Two vocal channels have been separated:  Verbal: running subtitles  Prosodic: superimposed filter creating the “behind a wall” effect  Participants:  Native speakers of Russian  Eight groups of 10 to 17 participants

7. 7 Eight experimental groups  Group 0: only the context excerpt  Groups 1 (one communication channel)  Verbal: subtitles, temporally aligned  Prosodic: filtered sound  Visual: video  Groups 2 (two communication channels):  Verbal + prosodic = original sound  Verbal + visual: subtitles and video  Prosodic + visual: filtered sound and video  Group 3: original material

8. 8 Verbal + visual

9. 9 Procedure  The context and the experimental excerpts were shown to a group of subjects on a large screen  Each subject answered 23 multiple-choice questions concerned with the experimental excerpt alone  What Tamara Stepanovna offers Masha before the beginning of the conversation:  a. to take off her coat  b. to have a cup of tea  c. to have a seat  d. to have a drink  Percentage of correct answers is used as an assessment of a subject’s degree of understanding

10. 10 Results  All three channels are substantially informative  Verbal > visual > prosodic  Integration of visual and prosodic channels is difficult

11. 11 Molchanova 2010 Kibrik and Molchanova 2013  Methodological issues  The following aspects of the prior study have been changed (improved)  Stimulus material  Methods of isolating the channels  Questionnaire  Participants and interviewing procedure

12. 12 Stimulus material: discourse type  Shortcomings of movies  Plot facilitates guessing  Possible familiarity with the movie  Quasi-natural behavior of actors  Solution: natural dialogue  Guessing game original.avi, 0:19 – 0:57

13. 13 Stimulus material: speakers  Shortcomings of the prior studies  Same-sex speakers  indistinguishable in the prosody-only version  Solution:  Different sexes: F0 range is different

14. 14 Methods of isolating the channels: Verbal channel  Shortcomings of subtitles  Subtitles belong to the visual mode  Hard to read without punctuation Especially at the rate of speech And especially in the “verbal + visual” condition  Solution: spoken prosody-free signal  Each word in transcript is recorded individually from the corresponding person  All thus elicited words are glued together in the right order

15. 15 Visual + verbal (the robot condition)

16. 16 Verbal channel  Remaining problem  Unnatural input No reduction No intonation etc.

17. 17 Methods of isolating the channels: Prosodic channel  Shortcomings of the prosodic material as used in previous studies  Excessive noise  Solution:  Loudness is decreased radically at all frequencies except for the speaker’s average F0 frequency  This has led to a more satisfactory “behind the wall” (or “behind the glass”) effect

18. 18 Visual + prosodic (the mermaid condition)

19. 19 Questionnaire  Shortcomings of prior studies  Èl’bert 2007: gap between Group 0 (38.3%) and Group 3 (87.4%) is insufficient  Solution  Testing stage Identify trivial questions (high Group 0)–5 Identify unfortunate questions (low Group 3) –2 30  23  Group 0: 34.5% correct answers  Group 3: 88.0% correct answers

20. 20 Participants and interviewing procedure  Shortcomings of prior studies  Uncontrolled social status and geographical origin of participants  Multiple participants in one room may affect each other’s performance  Need for a big screen  Solutions  Control for social status and geographical origin; homogeneous group  Comparable, independent, and comfortable conditions Detailed guidelines  Remote implementation Stimulus materials at Youtube.com Questionnaire at Googledocs

21. 21 Kibrik and Molchanova 2013: Results  Each individual channel is substantially informative and prevails over the null condition (34.5%) F-test: verbal and visual: p<0.05, prosodic: p=0.127  Verbal (58.8%) > visual (52.2%) > prosodic (40.2%) F-test: verbal > prosodic, visual > prosodic: p visual: p=0.071

22. 22 Kibrik and Molchanova 2013: Results  Two-channel conditions prevail over the one-channel conditions much more clearly than in the previous experiment (verbal+prosodic – 73.5%, verbal+visual – 88.2%) F-test: all pairwise comparisons but “visual+prosodic > visual”: p all one- channel conditions: p<0.0001  A dramatic dip in the visual+prosodic condition is even clearer F-test: significant difference from the two other two-channel conditions, p<0.0001

23. 23 Kibrik and Èl’bert 2008 vs. Kibrik and Molchanova 2013  General picture is remarkably similar  In the new study all effects are clearer

24. 24 Normalized contribution of three channels  Suppose the three channels are independent  Sum up all percentages of individual channel contributions and normalize to 100%  Identify normalized contribution

25. 25 Normalized contribution of three channels Kibrik and Èl’bert 2008Kibrik and Molchanova 2013 Summed percentages72+51+62=18559+52+40=151 Normalized contributions Verbal 72%:1.85≈39%59%:1.51≈39% Prosodic 51%:1.85≈28%46%:1.51≈30% Visual 62%:1.85≈33%49%:1.51≈32%

26. 26 Gender differences  Molchanova 2010: gender advantages  Percentages of correct answers ConditionMenWomenAdvantage Verbal only59.169.9Women: +10.7 Visual + prosodic 66.151.6Men: +14.5

27. 27 Conclusions  All communicatioin channels are highly significant  the traditional linguistic viewpoint is incorrect  The verbal channel is the leading one  the viewpoint popular in applied psychology is incorrect  Information from the prosodic and the visual channels is primarily used through integration with the verbal channel  Very similar results have been attained in different studies, in spite of very different methodological details

28. 28 Further questions  Auditory or graphic presentation of the “verbal alone” channel?  Explore different discourse types, such as monologic discourse  …and: Other suggestions on this approach?

29. 29 Acknowledgements  Olga Fedorova  Anna Laurinavičiute  Andriy Myachykov  RGNF #11-04-00153

30. 30 Thanks for your attention verbal channel visual channel prosodic channel language