UNDERSTANDING SPOKEN DISCOURSE: THE CONTRIBUTION OF THREE INFORMATION CHANNELS Poster presented at the Cogsci ’2008, Moscow, Russia, June BackgroundExperiment Discussion Method: The three channels – verbal, prosodic, and visual – have been isolated from each other and presented in all possible combinations (8 altogether).  The visual channel by itself is video alone (without sound)  The verbal channel is subtitles running in temporal alignment with the original film  The prosodic channel is the original audio component with a superimposed filter creating the effect of a conversation behind the wall. Subjects: 99 participants, divided into 8 groups. ¾ - women, ¼ - men. Native speakers of Russian. Each group comprised 10 to 17 subjects Material: Russian TV serial “Tajny sledstvija” – “Mysteries of the investigation” (3 min. 20 sec. long), preceded by a 8 minutes context (that starts from the beginning of the series). The excerpt fully consists of a conversation, to ensure that we are testing the understanding of discourse rather than of the film in general. Procedure: Every subject was instructed to watch the context and the experimental excerpt and then answer a set of questions concerned with the experimental excerpt alone. A subject was supposed to choose only one answer out of four listed variants.  29 questions were originally included in questionnaire  But six were later discarded as either trivial or prone to guessing  23 questions kept after the testing phase Example of a question: Results - the mean percentage of correct answers in each of the eight experimental groups: 1.Each of the three information channels, taken in isolation, is quite informative. The percentages in groups 5 through 7 are significantly higher than the percentage in group 8. 2.The hierarchy of informativeness can be represented as follows: verbal > visual > prosodic. 3.Combining the verbal channel with one additional channel does not increase the percentage of correct answers (compare group 5 with groups 2 and 3). Only inclusion of all three channels (group 1) yields significantly better results than the verbal channel by itself (group 5). 4.Adding the visual channel to the prosodic channel does not result in increase in correct answers (compare group 6 with group 4). 5.The combination ‘prosodic plus visual’ (group 4) displays significantly lower result than in other pairs of channels (groups 2 and 3). Evidently, this combination is not customary for subjects, and they have trouble integrating information from prosody and video. The relative contribution of the three channels Assuming, for the sake of simplicity, that all three channels are independent, ( =185)/100 Results:  Verbal channel 39% (72:1.85≈39),  Prosodic channel 28% (51,1:1.85≈28),  Visual channel 33% (61,7:1.85≈33), 39 Group number Experimental material OriginalSoundSubtitles+ video Prosody+ video SubtitlesProsodyVideoNothing (context only) Information channels verbal prosodic visual verbal prosodic verbal visual prosodic visual verbalprosodicvisual[none] Mean % of correct answers 87,4%70,4%73,9%51,2%72,0%51,1%61,7%38,3% Examples of experimental material: Full variant Verbal+Visual Verbal Ну, конечно, может быть, Вам педагога порекомендовать Conclusions  All information channels are highly significant  the traditional linguistic viewpoint is erroneous.  The verbal channel is the leading one  the viewpoint popular in applied psychology is erroneous.  Information from the prosodic and the visual channels is primarily used through integration with the verbal channel. Traditional point of view in linguistics (as well as in general psychology)  Language is ultimately segmental, or verbal.  Hierarchy of segmental units: phonemes – morphemes – words – phrases – clauses – sentences. However:  There are other kinds of encoding, or information channels, relevant for linguistic communication  Prosody, including (i)-(vii), conveys much information – imagine listening to a conversation behind the wall i.Pausing ii.Accents iii.Tone iv.Tempo v.Register vi. Reduction vii. Phonations viii. Loudness  Body language, including (i)-(iv), also constitutes an important visual channel i.Gestures ii.Mimics iii.Postures iv.Proxemics Gestures are now viewed by many (see e.g. McNeill 1992) as a system not separate from language but forming a single complex with it Three information channels Discourse Vocal Channels Visual Channel Verbal Channel Prosodic Channel  Prosody and body language are generally neglected in mainstream linguistics  However, in applied psychology it is often stated that (figures go back to Mehrabian 1971):  body language conveys 55% of information  prosody conveys 38% of information  the verbal component conveys 7% of information  Research question: who is right – mainstream linguistics, or applied psychology, or none? Andrej A. Kibrik (Institute of Linguistics RAN) Ekaterina M. El’bert (CJSC GLITNIR Securities ) What Tamara Stepanovna offers Masha before the beginning of the conversation: a. to take off her coat b. to have a cup of tea  c. to have a seat d. to have a drink Directions for subjects:  Please watch an extract of a film (11 minutes) and answer a set questions related to the last portion of the extract  When answering questions, you need to choose only one of the four answers  Questions are printed on separate sheets of paper. After answering a question, please turn to the next sheet Correcting previous answers is not allowed  Please consider one question at a time, beginning from the first question  More natural discourse material, such as usual conversation  Further improvement of the questionnaire Main criterion: increase the delta of correct answers between groups 1 and 8; the current difference between 38% and 87% is insufficient  Construction of a statistical model, assessing the relative contribution of channels, including possible relationships between channels  Search for correlation between type of question and information channel Acknowledgements: O.V.Fedorova, S.A.Krejchi, O.F.Krivnova, A.V.Proxorov, E.A.Iljushina, S. Lando Further Directions